Key Responsibilities:
- Design, build, and optimize scalable data pipelines using Azure Databricks, PySpark, and Delta Lake for high-volume data processing and transformation.
- Develop and maintain complex SQL and Python solutions for data transformation, validation, and quality assurance across large enterprise datasets.
- Implement robust data engineering workflows on cloud platforms, ensuring performance, scalability, and reliability.
Required Skills & Experience:
- Strong hands-on experience with Azure Databricks, including notebooks, workflows, and cluster configuration.
- Solid expertise in Delta Lake and Azure data services such as ADLS (Azure Data Lake Storage) and ADF (Azure Data Factory).
- Proficiency in Python, PySpark, and SQL for building production-grade data engineering solutions.
- Experience with CI/CD pipelines and best practices for automated deployment and version control.
- Strong understanding of data migration, data transformation, and cloud-based data architecture.
Pay: From $90,000.00 per year
Work Location: Hybrid remote in Sydney NSW
