Annotation Data Engineer

Key Responsibilities: Define, implement, and maintain annotation quality standards, guidelines, defect taxonomies, and acceptance criteria. Develop and maintain quality metrics and reporting frameworks to measure accuracy, consistency, completeness, rework rates, throughput, and reviewer agreement. Design and execute sampling strategies, annotation audits, escalation workflows, and root-cause analysis processes for large-scale annotation programs. Evaluate generated captions, reasoning outputs, and foundation model training datasets against established quality benchmarks. Analyze annotation quality trends, identify recurring defects, and coordinate corrective and preventive actions with cross-functional teams. Validate dataset releases to ensure compliance with defined quality requirements and readiness for model training. Maintain traceable quality records, audit documentation, issue logs, and continuous improvement plans. Conduct calibration sessions and support reviewer alignment to improve annotation consistency and inter-annotator agreement. Collaborate with annotation operations, data engineering, quality engineering, and AI/ML teams to optimize annotation workflows. Leverage SQL, Python, and reporting tools to analyze quality data, automate measurements, and generate actionable insights. Required Technical and Functional Skills2–5 years of relevant experience in annotation quality, machine learning data operations, quality engineering, or data quality management. Experience supporting physical AI, robotics, computer vision, or machine learning programs. Demonstrated experience developing quality metrics, KPIs, and performance measurement systems. Hands-on experience working with large-scale annotation programs and dataset quality assurance processes. Experience evaluating generated captions, reasoning outputs, or foundation model training datasets. Strong understanding of annotation guidelines, sampling plans, acceptance criteria, defect classification, and audit methodologies. Ability to perform quality trend analysis, identify root causes, and drive corrective actions. Working knowledge of SQL and Python for data analysis, quality measurement, and reporting. Experience creating quality dashboards, reports, and performance tracking mechanisms. Strong analytical, problem-solving, documentation, and cross-functional communication skills.