Azure DataBricks Developer

Location: Louisville, KY/ Chicago, IL/ Dallas, TX / Arlington, VAFulltime OnlyMust Have Technical/Functional SkillsDesign, develop, and maintain cloud native data engineering solutions using Azure Databricks. Build and manage PySpark notebooks to process large scale structured and semi structured datasets. Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement. Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns. Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution. Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs. Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks. Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices. Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines. Collaborate with data architects, analysts, and downstream consumers to understand data requirements. Perform debugging and root cause analysis for data quality, performance, or pipeline failures. Support testing, validation, and reconciliation of data during development, migration, and production phases. Follow security, governance, and compliance standards applicable to cloud data platforms. Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment. Maintain documentation for notebooks, workflows, data models, and migration approaches. Roles &Responsibilities Develop and maintain data engineering solutions using Azure Databricks and PySpark. Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation. Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases. Support cloud data migrations, including data validation and performance benchmarking. Optimize Spark jobs for performance, scalability, and cost efficiency. Collaborate with platform, DevOps, and data governance teams to ensure environment stability. Perform data pipeline testing and validation, ensuring correctness and completeness. Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution. Participate in code reviews and enforce data engineering best practices. Support production deployments and post deployment stabilization. Provide inputs to data architecture and platform improvement initiatives. Mentor junior data engineers when required.