AI Evaluation Specialist

Location: Remote (Continental U.S.)

Company: JR Software Solutions Inc. (JRSS)

Employment Type: Full-time

Position Overview: The JRSS AI/ML Team is seeking an AI Evaluation Specialist to determine the effectiveness, safety, usability, and readiness of enterprise AI tools for adoption in a federal public health environment. You will partner with product managers, data scientists, AI engineers, and security and privacy teams to run structured assessments of generative AI products and platforms, including OpenAI tools and life sciences models such as Rosalind. Your findings will guide which AI solutions are adopted, comparing them across quality, accuracy, reliability, safety, privacy, and security.

Key Responsibilities: Plan and execute hands-on evaluations of AI tools, including OpenAI products and life sciences models such as Rosalind, using representative business scenarios, workflows, prompts, datasets, and user personas. Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness. Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications. Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders. Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost. Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities. Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders. Stay current with advancements in generative AI, model evaluation, AI safety, and enterprise AI products, and incorporate relevant methods into the evaluation program. Contribute to knowledge sharing and a strong, collaborative team environment.

Required Qualifications: Bachelor's degree.5+ years of experience in model evaluation or model quality assurance, with hands-on experience evaluating AI-enabled products or services. Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria. Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes. Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost. Familiarity with common generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, enterprise search, and tool or connector integrations. Must be a U.S. Citizen or Lawful Permanent Resident (Green Card holder).Ability to obtain and maintain a Public Trust clearance.

Preferred Qualifications: Master's degree. Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind, or comparable generative AI solutions. Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing. Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows. Experience building dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results. Proficiency with Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics and summarize findings. Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.

What We Offer: 401(k) plan with a 4% company match. Two weeks of paid time off. Comprehensive medical, dental, and vision insurance. Ongoing investment in professional growth and development. Equal Opportunity StatementJR Software Solutions Inc. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.