Skip to main content

Sr. AI & Data Engineer - Databricks

Posted today

accentureHouston (TX)

SENIORITY

Senior

Apply (opens in a new tab)

About the role

Overview: In this role you design, build, and optimize scalable data pipelines and autonomous agents using Databricks Mosaic AI Agent Framework and related tools. You collaborate with analytics, data science, and cloud teams to ensure reliable, high-performance data delivery for clients. You work within a global, collaborative ATC network delivering advanced data and AI solutions. You will engage with modern lakehouse architectures, vector search, and agent design to enable impactful AI-enabled analytics. Compensation / Benefits medical dental vision life long-term disability coverage 401(k) plan","bonus opportunities","paid holidays","paid time off"),
Responsibilities: Architect, develop, and maintain scalable, low-latency data pipelines on Azure Databricks and Spark Design and optimize Lakehouse architecture using Bronze/Silver/Gold with Delta Lake Partner with data scientists and business analysts to deliver clean, validated datasets Design and implement autonomous agents using Mosaic AI Agent Framework and LangChain/AutoGen Build and optimize vector search indexes for accurate context retrieval Process large-scale distributed datasets on the Databricks Intelligence Platform with Apache Spark Design data models to meet business requirements and enhance usability Continuously improve data architecture and processes for efficiency Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and observability Monitor pipeline health with automated alerts and observability frameworks for proactive optimization
Key requirements: 3+ years as Data & AI Engineer or similar 3+ years of experience with Microsoft Azure cloud data services 2+ years experience with Databricks (PySpark, SQL, Delta Lake, workflows) Bachelor's degree or equivalent (12 years of work experience) problem-solving collaboration/teamwork communication Databricks (PySpark, SQL) Azure Databricks; Delta Lake; Auto Loader; Lakehouse Medallion Architecture (Bronze/Silver/Gold) LangChain; AutoGen; Mosaic AI Agent Framework

Before you apply

Applying takes about a minute. These four things decide how fast it moves after that.

Your profile is current

It's what we read first. Occupations, seniority and locations matter more than a long history.

Two examples you can talk through

Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.

A number in mind

What you're on now and what would make you move. We negotiate better when we know both.

Your notice period

Employers plan around it, and it's the question that stalls offers most often.

Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.

More like this