We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Python, PySpark, Apache Spark, and large-scale data engineering. The ideal candidate will be responsible for designing and developing scalable ETL pipelines, implementing data processing solutions, managing data ingestion frameworks, and optimizing cloud-based data platforms. The role also involves technical leadership, data governance, and collaboration with analytics and business intelligence teams.
Required Technical Skills
Strong hands-on experience with Databricks, Delta Lake, Delta Live Tables, and Unity Catalog.
Proficiency in Python, PySpark, Spark SQL, and Apache Spark.
Strong experience in ETL development, data pipeline design, and data transformation.
Knowledge of Hadoop, Kafka, Hive, HDFS, and Sqoop.
Expertise in SQL, data modeling, and data warehousing.
Experience with batch and real-time data processing using Kafka and Spark Streaming.
Cloud experience with AWS and/or GCP.
Strong understanding of data quality, validation, cleansing, and governance.
Experience working in Agile environments (Scrum, Kanban, SAFe).
Preferred Skills
Scala / Scala-Spark, IBM DataStage, TIBCO.
AWS S3, Glue, Lambda, Kinesis/MSK.
Tableau, Power BI, Spotfire, OAC.
Teradata, DB2, BigQuery, Mainframe.
Git, data governance, and retention modeling.
Roles &
Responsibilities: Design, develop, and maintain scalable data pipelines and ETL solutions.
Build data ingestion and transformation frameworks for structured and unstructured data.
Develop and optimize batch and streaming data processing solutions.
Implement data quality, validation, and governance standards.
Integrate heterogeneous data sources into enterprise data platforms.
Support analytics, reporting, BI, and machine learning initiatives.
Collaborate with Data Scientists, BI teams, and business stakeholders.
Lead technical discussions, mentor junior engineers, and drive data engineering best practices.
