Data Engineer - Lead
robert halfHouston (TX)
About the role
We are looking for an experienced Lead Data Engineer to oversee the design, implementation, and management of advanced data infrastructure in Houston, Texas. This role requires expertise in architecting scalable solutions, optimizing data pipelines, and ensuring data quality to support analytics, machine learning, and real-time processing. The ideal candidate will have a deep understanding of Lakehouse architecture and Medallion design principles to deliver robust and governed data solutions.
Responsibilities:
Develop and implement scalable data pipelines to ingest, process, and store large datasets using tools such as Apache Spark, Hadoop, and Kafka.
Utilize cloud platforms like AWS or Azure to manage data storage and processing, leveraging services such as S3, Lambda, and Azure Data Lake.
Design and operationalize data architecture following Medallion patterns to ensure data usability and quality across Bronze, Silver, and Gold layers.
Build and optimize data models and storage solutions, including Databricks Lakehouses, to support analytical and operational needs.
Automate data workflows using tools like Apache Airflow and Fivetran to streamline integration and improve efficiency.
Lead initiatives to establish best practices in data management, facilitating knowledge sharing and collaboration across technical and business teams.
Collaborate with data scientists to provide infrastructure and tools for complex analytical models, using programming languages like Python or R.Implement and enforce data governance policies, including encryption, masking, and access controls, within cloud environments.
Monitor and troubleshoot data pipelines for performance issues, applying tuning techniques to enhance throughput and reliability.
Stay updated with emerging technologies in data engineering and advocate for improvements to the organization's data systems.
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.
More like this