Lead Data Platform and ML Manager
Lead Data Platform and ML Manager
Posted yesterday
SENIORITY
Manager
About the role
- Operate and improve the pipelines that bring data from our operational systems and document sources into the lakehouse· Extend the lakehouse to new internal and third-party data sources as the business grows, with quality checks and governance built in from the start· Enforce data ownership, sharing, and ingestion boundaries across every pipeline, data product, and model, and be able to demonstrate that they hold· Own the dbt transformation layers and the curated data products built on top of them· Govern the lakehouse in Unity Catalog: access controls, lineage, and the handling of sensitive data as new tables and consumers arrive· Own the platform as code, with infrastructure and workloads deployed through automated, reviewed CI/CD pipelines· Keep the data quality framework running: automated tests, freshness checks, and anomaly detection, held to an agreed reliability target· Own document ingestion and extraction using Databricks AI functions and Amazon Textract, and extend it to new document types over time· Extend our existing machine learning models and design, build, and ship new ones, from problem framing and feature design through validation, governance review, and production· Run the model lifecycle in MLflow: feature pipelines, retraining, evaluation, registry, versioning, and approval, kept auditable· Operate batch and real-time inference through Databricks Model Serving against the latency targets the business requires· Monitor models in production, respond to drift, and maintain the explainability, fairness testing, and documentation our model governance requires· Build and maintain the semantic layer and Databricks AI/BI dashboards our business teams use, reconciled to authoritative sources so one question does not produce two answers· Own cloud cost for the data and machine learning estate against an agreed budget, and keep runbooks and documentation current
- Hard Skills·
- Data engineering. 5 or more years building production data pipelines. Strong SQL and Python, production experience with Spark or PySpark, and dbt for transformation and testing.·
- Databricks. Hands-on production experience with Unity Catalog, Delta Lake, Lakeflow, Databricks SQL, and Databricks AI functions.·
- AWS and infrastructure as code. Running Databricks on AWS, with infrastructure as code in Terraform or OpenTofu and workloads as Databricks Asset Bundles.·
- CI/CD. Pipeline ownership using GitHub Actions, with automated testing and controlled promotion across environments.·
- Machine learning engineering. Building and shipping models end to end: feature pipelines, experiment tracking, MLflow, and both batch and real-time serving.·
- Model operations. Drift detection and response, retraining strategy, champion and challenger evaluation, and explainability methods such as SHAP.·
- Regulated data handling. HIPAA or equivalent. PHI and PII protection, encryption, least-privilege access, and audit logging.·
- AI-assisted development. We build with AI every day, and we expect our data and AI engineer to do the same. You use tools like Claude Code, Databricks Assistant, or GitHub Copilot as part of your normal workflow, and you review their output as carefully as a teammate’s.·
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.
