Tech Lead Machine Learning Engineer - LLM Evaluation & Agent Systems

Responsibilities: Design, build, and scale robustLLM evaluation architecture, including automated offline datasets, metric suites, and benchmarking pipelines to assess prompt and model performance. DevelopLLM-driven synthetic data generationapproaches to address data scarcity bottlenecks and enable systematic evaluation of long-tail and edge-case risk scenarios. Move beyond criteria-based prompt engineering by extracting risk patterns from complex case data to create native, ML-driven decision models and reasoning abstractions. Architect, evaluate, and continuously optimizemodular agent skills and toolstailored for Data Science, Analytics, Data Engineering, and SRE workflows, improving tool-use accuracy, execution efficiency, and task success rates. Apply advanced evaluation methodologies (includingLLM-as-a-judgeand multi-agent benchmarking) across both risk intelligence and agentic ecosystems. Provide technical leadership by defining core ML system design patterns, benchmarking standards, and engineering best practices, and by translating complex operational requirements into scalable ML solutions with cross-functional partners. Requirements5+ yearsof software or ML engineering experience, including a strong track record designing, building, and deploying ML/LLM systems in production. Hands‑on expertise withLLM architectures, fine‑tuning, RAG, prompt tuning, and agentic frameworks. Experience building robustLLM evaluation frameworks(LLM-as-a-judge, automated benchmark pipelines) and working on data scarcity or imbalance problems. Strong coding skills with production experience supporting scalable data processing pipelines. Prior domain experience inRisk & Compliance Intelligenceand Automated Decision Systems. Hands‑on experience with synthetic dataset generation or bootstrapping models for cold‑start scenarios. Experience building AI tools or agents for technical personas including Data Analysts, Data Scientists, Data Engineers, and Data SRE. Demonstrated ability to serve as afounding technical leadon ambiguous, high-impact initiative pods. Location and CompensationLocation: Seattle, WA (onsite)

Salary: USD 198,360 - 416,100 per year

Benefits: Day one access to medical, dental, and vision insurance 401(k) savings plan with company match Paid parental leave Short-term and long-term disability coverage Life insurance Wellbeing benefits, among others 10 paid holidays per year 10 paid sick days per year 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure)

About US: DSTikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. On-site presence across teams is intended to improve speed, alignment, and agility, particularly for real-time decision-making, team development, and integrated execution. The company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week. USDS Reasonable AccommodationUSDS is committed to providing reasonable accommodations in its recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs, or other reasons protected by applicable laws.