Lead Databricks Engineer
united itIselin (NJ)
Lead Databricks Engineer
Posted 2 days ago
united itIselin (NJ)
SENIORITY
Lead
About the role
Finance Data Analyst Major Responsibilities:
Work on finance data related to collaterals, ETD, OTD, settlement market, cash product, repo, duos repo. Design, develop, and deploy Databricks jobs to process and analyze large volumes of data. Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines. Optimize Databricks jobs for performance and scalability to handle big data workloads. Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks. Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing enterprise data warehouse solutions. Demonstrated proficiency with data analytics, data insights. Proficient writing SQL queries and programming including stored procedures and reverse engineering existing processes. Azure Synapse/Bigquery/Redshift is good to have. Perform code reviews to ensure fit to requirements, optimal execution patterns, and adherence to established standards.
Skills:
5+ years' – strong experience in finance/banking industry – capital markets, investment banking - collaterals, ETD, OTD, settlement market, cash product, repo, duos repo. 10+ years - enterprise data management. 10+ years - SQL server based development of large datasets. 5+ years with data warehouse architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding. Azure Synapse/Bigquery/Redshift experience is good to have. 3+ years Python(numpy, pandas) coding experience. Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have. Experience in data warehousing - OLTP, OLAP, dimensions, facts, and data modeling. Previous experience leading an enterprise-wide cloud data platform migration with strong architectural and design skills. Capable of discussing enterprise level services independent of technology stack. Experience with cloud based data architectures, messaging, and analytics. Superior communication skills. Cloud certification(s). Any experience with regulatory reporting is a plus.
Education: Minimally a BA degree within an engineering and/or computer science discipline.
Before you apply
Applying takes about a minute. These four things decide how fast it moves after that.
Your profile is current
It's what we read first. Occupations, seniority and locations matter more than a long history.
Two examples you can talk through
Not a portfolio — just two pieces of work where you can explain the decisions and what you'd change.
A number in mind
What you're on now and what would make you move. We negotiate better when we know both.
Your notice period
Employers plan around it, and it's the question that stalls offers most often.
Once you apply, someone reads it and calls you before anything reaches the employer — usually within two working days.