Overview:
In this Data Engineer role, you will design and maintain scalable data pipelines to support large datasets and critical business operations in a brokerage context. You will work with cross-functional teams to translate data needs into robust technical solutions, improving accessibility and processing efficiency. You’ll focus on data quality, reliability, and performance to enable near-real-time data delivery and analytics. This is a fast-paced role that offers impact through shaping the data platform and engineering initiatives.
Responsibilities:
Design, develop, and optimize data pipelines for ingestion, transformation, and delivery of large-scale datasets Build and maintain ETL workflows ensuring timely and accurate data movement across platforms Use Python and Apache Spark to process data and enhance pipeline performance Work with Apache Hadoop technologies to manage distributed storage and computation for high-volume workloads Implement streaming and messaging solutions with Apache Kafka for near-real-time data integration Collaborate with cross-functional teams to translate business needs into technical solutions Monitor data processes, troubleshoot issues, and resolve bottlenecks to ensure reliability and data integrity Contribute to data platform enhancements and internal process improvements Key requirements strong technical expertise with modern data engineering tools Python Apache Spark Apache Hadoop Apache Kafka experience designing data pipelines and ETL focus on data quality, performance, and consistency ability to collaborate with cross-functional teams troubleshooting and optimization of data processes cross-functional collaboration problem-solving attention to detail Python Apache Spark Apache Hadoop