Senior Data Engineer
Full-Time | Permanent | Australia
Salary: AUD $110,000 – $120,000 plus Superannuation (commensurate with qualifications and experience)
About the Role
We are seeking an experienced Senior Data Engineer to design, develop and support enterprise-scale cloud data platforms and modern data integration solutions. The successful candidate will play a key role in delivering scalable, secure and high-performing data engineering solutions that enable business intelligence, advanced analytics, Artificial Intelligence (AI) and Machine Learning initiatives.
The role requires strong expertise in cloud data platforms, ETL/ELT development, API integration, enterprise data governance, solution design, and modern data engineering technologies.
Key Responsibilities
- Design, develop and maintain enterprise-scale ETL and ELT data pipelines for structured, semi-structured and unstructured data.
- Design and implement cloud-based data integration solutions using Informatica Intelligent Data Management Cloud (IDMC), Azure Data Factory (ADF) and other enterprise integration technologies.
- Build scalable data platforms using Snowflake and Databricks.
- Develop robust integrations with enterprise applications and third-party systems using REST APIs and API integration frameworks.
- Design and implement scalable batch, near real-time and streaming data ingestion solutions.
- Develop automation scripts using Python, Shell scripting, SQL and other scripting technologies.
- Develop and maintain enterprise orchestration workflows using tools such as Apache Airflow, Azure Data Factory Pipelines, Control-M or equivalent workload automation platforms.
- Design enterprise data models supporting reporting, analytics, AI and Machine Learning workloads.
- Produce high-quality solution designs, technical specifications and architecture documentation.
- Implement enterprise Data Governance capabilities including metadata management, data lineage, data quality, data cataloguing and master data management.
- Ensure compliance with enterprise security, privacy and regulatory requirements.
- Optimise data platform performance, scalability, reliability and operational efficiency.
- Design and implement CI/CD pipelines and DevOps practices for data engineering solutions.
- Collaborate with Solution Architects, Business Analysts, Developers and business stakeholders to translate business requirements into scalable technical solutions.
- Support AI-enabled initiatives including Generative AI, predictive analytics, Retrieval-Augmented Generation (RAG) and machine learning data pipelines.
- Evaluate emerging technologies and recommend improvements to enterprise data architecture.
- Mentor junior team members and promote engineering best practices.
Essential Skills and Experience
Applicants must possess:
- Bachelor's degree or higher in Computer Science, Information Technology, Engineering or a related discipline.
- Minimum five (5) years of professional experience in Data Engineering, Cloud Data Engineering or Enterprise Data Integration.
- Demonstrated experience designing and developing enterprise ETL and ELT solutions.
- Strong hands-on experience with:
- Informatica Intelligent Data Management Cloud (IDMC)
- Azure Data Factory (ADF)
- Snowflake
- Databricks
- Python
- Shell scripting
- Advanced SQL
- REST API integration
- Data modelling and data warehousing
- Enterprise orchestration tools such as Apache Airflow, Azure Data Factory Pipelines, Control-M or Autosys
- Experience designing cloud-native data platforms on Microsoft Azure.
- Strong understanding of enterprise data architecture principles.
- Experience implementing Data Governance, Metadata Management, Data Lineage, Data Quality and Master Data Management frameworks.
- Experience building secure, scalable and high-performance cloud data solutions.
- Experience implementing CI/CD pipelines, Git version control and DevOps practices.
- Strong analytical, problem-solving and troubleshooting skills.
- Excellent written, verbal and stakeholder communication skills.
Desirable Skills
The following experience will be highly regarded:
- Microsoft Fabric
- Azure Synapse Analytics
- Apache Spark and Spark SQL
- Delta Lake
- Kafka or other streaming technologies
- dbt
- Azure Event Hub
- Azure Functions
- Docker and Kubernetes
- Terraform or Infrastructure as Code
- Azure OpenAI Service, AWS Bedrock or Google Vertex AI
- Vector databases and Retrieval-Augmented Generation (RAG)
- AI and Machine Learning pipeline development
- Agile delivery methodologies including Scrum or SAFe
- Enterprise Solution Architecture frameworks
Personal Attributes
The successful candidate will demonstrate:
- Strong solution design and architecture capabilities.
- Ability to lead technical discussions and mentor engineering teams.
- Excellent stakeholder engagement and communication skills.
- Strong analytical and problem-solving abilities.
- Commitment to delivering high-quality, scalable and secure solutions.
- Ability to manage multiple priorities in a fast-paced environment.
- Continuous learning mindset with a strong interest in emerging cloud and AI technologies.
Salary
AUD $110,000 – $120,000 plus Superannuation, depending on qualifications and relevant experience.
How to Apply
Interested applicants should submit:
- A current resume.
- A cover letter addressing the essential skills and experience.
- Details of relevant qualifications and professional certifications.
Only shortlisted candidates will be contacted.
Our organisation is an equal opportunity employer and welcomes applications from suitably qualified candidates from diverse backgrounds.
Pay: $110,000.00 – $120,000.00 per year
Work Location: Hybrid remote in Sydney NSW
