Role Overview
We are looking for an experienced Data Engineer with strong expertise in Databricks, Azure Data Factory (ADF), Python, SQL, and Apache Spark. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and data processing solutions.
Key Responsibilities
- Design and develop scalable data pipelines using Databricks and Apache Spark.
- Build and maintain ETL/ELT workflows using Azure Data Factory (ADF).
- Develop efficient data processing solutions using Python and PySpark.
- Write complex SQL queries for data transformation, validation, and analysis.
- Work with large datasets and optimize Spark jobs and data pipelines for performance.
- Implement data quality, validation, monitoring, and error-handling processes.
- Collaborate with Data Architects, Analysts, Developers, and business stakeholders.
- Troubleshoot data pipeline failures and resolve performance issues.
- Follow best practices for data security, governance, and documentation.
Required Skills
- Strong hands-on experience with Databricks.
- Proficiency in Apache Spark / PySpark.
- Strong programming skills in Python.
- Advanced knowledge of SQL.
- Hands-on experience with Azure Data Factory (ADF).
- Good understanding of ETL/ELT concepts and data warehousing.
- Experience working with cloud-based data platforms, preferably Microsoft Azure.
- Strong problem-solving and communication skills.
Good to Have
- Experience with Delta Lake / Delta Tables.
- Knowledge of Azure Data Lake Storage (ADLS).
- Experience with CI/CD and version control tools such as Git/Azure DevOps.
- Understanding of data governance and security practices.
Pay: $800.00 – $900.00 per day
Work Location: Hybrid remote in Melbourne VIC
