Evaluation Engineer

Posted yesterday

reinforced learningTonawanda (NY)

SENIORITY

Lead

About the role

The role:

Research designs evaluations; you make them run reliably. You'll build the harnesses, graders and reporting that let a model be evaluated on hundreds of tasks across our environments, with results anyone can reproduce.
What you'll own
The evaluation harness: task loading, environment orchestration, model adapters, scoring.
Graders and verifiers implemented with the environment and verifier engineers.
Runs that are deterministic, resumable and cheap to repeat.
Dashboards and reports that researchers and customers read.
Discrepancies between runs, chased until they're explained.
What you bring
Strong software engineering in Python with an eye for reliability.
Model APIs, batch systems and asynchronous processing.
You care about reproducibility and know how to achieve it.
Comfort reading research code and making it production-grade.
Helpful, not required
Open evaluation frameworks.
Containers and cloud infrastructure.

Applying takes about a minute. These four things decide how fast it moves after that.

Your profile is current

It's what we read first. Occupations, seniority and locations matter more than a long history.

Two examples you can talk through

Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.

A number in mind

What you're on now and what would make you move. We negotiate better when we know both.

Your notice period

Employers plan around it, and it's the question that stalls offers most often.

Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.