GenAI/LLM Test Engineer

Required Skills: LLM/GenAI Product Testing Python & SQLAI Evaluation Frameworks – Ragas, Deep Eval, Lang Smith, Promptfoo, OpenAI Evals, Tru Lens Test Automation, Regression & API Testing Performance & Adversarial Testing Generative AI, LLMs & RAG

Nice to Have: Prompt Engineering / Prompt TestingCI/CD Automation Azure & Snowflake Cloud Data Platform experience

Key Responsibilities: Design and execute test plans for GenAI/LLM features, prompts, APIs, and applications. Build automated functional, regression, integration, and API test suites. Evaluate model outputs for accuracy, relevance, hallucination, bias, toxicity, and robustness. Develop evaluation frameworks, golden datasets, scoring rubrics, and regression test suites. Perform adversarial/red-team, performance, scalability, and reliability testing. Validate AI data pipelines for data quality, schema, and drift. Integrate automated and AI evaluation testing into CI/CD pipelines. Collaborate with Data Scientists, ML Engineers, and Dev Ops teams to define quality metrics and acceptance criteria. Environment: Primarily Microsoft Azure and Snowflake, with GenAI initiatives leveraging OpenAI and Claude models. ABOUT BRICKRED SYSTEMSBrick Red Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, Brick Red Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. Brick Red Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers. Through its focus on delivering specialized talent and technology expertise, Brick Red Systems supports organizations in building scalable teams and executing critical business initiatives across diverse industries and functional areas.