Lead Engineer, AI Quality
virtual vocationsDenver (CO)
Lead Engineer, AI Quality
Posted yesterday
virtual vocationsDenver (CO)
SENIORITY
Lead
About the role
To support the AI Quality engineering team, the full-time Lead Engineer, AI Quality will build evaluation frameworks, diagnostic infrastructure, and observability tooling for AI systems while managing the team in a remote environment.
Key responsibilities:
Build and own evaluation infrastructure, including CI/CD pipelines and datasets for AI agents
Diagnose quality breakdowns in AI systems and prototype solutions or identify priorities for improvement
Lead the AI Quality engineering team, setting technical direction and managing day-to-day priorities while remaining hands-on in coding
Required qualifications:
7+ years of experience in building and shipping production software, particularly with LLM-powered agents
Proficiency in Ruby on Rails and familiarity with Python for data evaluation tasks
Experience in developing evaluation or observability infrastructure for ML/AI systems
Knowledge of designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation
Ability to thrive in a fast-paced, dynamic environment with a strong alignment to company values
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.