Airflow Architect – Big Data
programmersAustin (TX)
Airflow Architect – Big Data
Posted yesterday
programmersAustin (TX)
SENIORITY
Lead
About the role
Need an experienced Airflow Architectto design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise inApache Airflow , Big Data technologies , cloud-native architectures, anddata engineering best practicesto build scalable, reliable, and high-performance data pipelines.
Key Responsibilities:
Architect and implement large-scale data orchestration frameworks using Apache Airflow.
Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
Lead cloud-based data platform implementations on AWSOptimize pipeline performance, scalability, reliability, and cost efficiency.
Establish CI/CD processes and DevOps practices for Airflow environments.
Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.
Required Skills
Airflow & Data Engineering:
- Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
- Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
- Hands-on expertise in ETL/ELT development and data pipeline automation.
- Hadoop ecosystem
- Data Warehousing concepts
- Data Lakes and Lakehouse architectures
Preferred Qualifications:
- 10+ years of experience in data engineering and big data platforms.3+ years of architecture experience designing enterprise data solutions.
- Experience with Kubernetes, Docker, and cloud-native deployments.
- Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
- Experience leading architecture reviews and technology strategy discussions.
- Success Metrics
- Pipeline reliability and SLA adherence
- Workflow performance optimization
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.