Overview:
In this role you design, build, and optimize scalable data pipelines and autonomous agents using Databricks Mosaic AI Agent Framework and related tools. You collaborate with analytics, data science, and cloud teams to ensure reliable, high-performance data delivery for clients. You work within a global, collaborative ATC network delivering advanced data and AI solutions. You will engage with modern lakehouse architectures, vector search, and agent design to enable impactful AI-enabled analytics. Compensation / Benefits medical dental vision life long-term disability coverage 401(k) plan","bonus opportunities","paid holidays","paid time off"),
Responsibilities:
Architect, develop, and maintain scalable, low-latency data pipelines on Azure Databricks and Spark Design and optimize Lakehouse architecture using Bronze/Silver/Gold with Delta Lake Partner with data scientists and business analysts to deliver clean, validated datasets Design and implement autonomous agents using Mosaic AI Agent Framework and LangChain/AutoGen Build and optimize vector search indexes for accurate context retrieval Process large-scale distributed datasets on the Databricks Intelligence Platform with Apache Spark Design data models to meet business requirements and enhance usability Continuously improve data architecture and processes for efficiency Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and observability Monitor pipeline health with automated alerts and observability frameworks for proactive optimization
Key requirements:
3+ years as Data & AI Engineer or similar 3+ years of experience with Microsoft Azure cloud data services 2+ years experience with Databricks (PySpark, SQL, Delta Lake, workflows) Bachelor's degree or equivalent (12 years of work experience) problem-solving collaboration/teamwork communication Databricks (PySpark, SQL) Azure Databricks; Delta Lake; Auto Loader; Lakehouse Medallion Architecture (Bronze/Silver/Gold) LangChain; AutoGen; Mosaic AI Agent Framework