Job Title: Big Data Engineer

Responsibilities: Design, develop, and maintain big data pipelines using Spark (PySpark/Scala), Hadoop, Kafka, and distributed computing frameworks. Build and optimize ETL/ELT pipelines for structured and unstructured data across cloud and on-prem data platforms. Work with cloud technologies (AWS, Azure, or GCP) including Data Lake, Databricks, EMR, Glue, Dataflow, Synapse, or Snowflake. Develop Delta Lake / Lakehouse architecture for high-performance ingestion and processing. Build real-time streaming solutions using Kafka, Spark Streaming, Kinesis, or Event Hub. Collaborate with data architects, analysts, and application teams to gather data requirements and deliver scalable solutions. Implement and maintain CI/CD pipelines, automated jobs, and orchestration using Airflow/Azure Data Factory/Glue Workflows. Optimize data pipelines for performance, cost efficiency, and reliability. Ensure data quality, validation, governance, and lineage best practices across the ecosystem. Troubleshoot and resolve production issues in a high-availability environment. Required Skills & Qualifications5–8+ years of experience as a Big Data Engineer or Data Engineer. Strong hands-on skills with Spark (PySpark or Scala), Hadoop, Hive, HDFS, MapReduce. Experience working with Databricks, EMR, Glue, Synapse, DataProc, or equivalent big-data compute engines. Proficiency in Python or Scala for data engineering. Experience with Kafka or other event-streaming technologies. Strong understanding of cloud data architectures (AWS S3/Glue/EMR | Azure ADLS/ADF/Databricks | GCP BigQuery/DataProc).Solid SQL skills and experience with relational + NoSQL databases. Experience with version control (Git) and CI/CD tools (Jenkins, Azure DevOps, GitHub Actions).Hands-on experience with Airflow, ADF, or other orchestration and scheduling tools. Familiarity with data modeling, data governance, and best practices for data quality.