Our client is looking for an experienced Senior Data Engineer to join the Channel & Data Insights team of a leading organization in South Africa.
The successful candidate will be responsible for designing, building and optimizing scalable data platforms and pipelines that support critical business and channel insights.
Working primarily across the Microsoft Azure data estate, including Microsoft Fabric, you will develop robust ETL/ELT solutions, modern data models and reliable data products that enable data-driven decision-making across the organization.
Key Responsibilities
- Design, develop and maintain scalable data platforms and data pipelines using Microsoft Fabric, Azure Data Factory, Databricks, ADLS Gen2, Azure Synapse Analytics, Event Hubs and Stream Analytics.
- Build and optimise solutions across Fabric Lakehouse, Warehouse, Data Pipelines, notebooks and semantic models.
- Develop robust ETL/ELT processes and implement modern data modelling approaches, including Kimball and/or Data Vault 2.0.
- Develop and optimize data transformations and processing using T-SQL, Python and Apache Spark.
- Build and maintain CI/CD pipelines and Infrastructure-as-Code solutions using Bicep, ARM, PowerShell/CLI and Azure DevOps.
- Apply data governance, security, quality and lineage practices using tools such as Databricks Unity Catalog and/or Microsoft Purview.
- Work closely with data analysts, product owners and business stakeholders to translate business requirements into effective technical solutions.
- Monitor, troubleshoot and optimize platform performance, reliability and cost efficiency.
- Contribute to data engineering standards, documentation, best practices and knowledge sharing within an Agile environment.
Desired Skills:
- 7-8 years’ hands-on experience in platform and data engineering
- at intermediate to senior level. Strong experience across the Azure data platform
- particularly: Microsoft Fabric Azure Data Factory Databricks ADLS Gen2 Azure Synapse Analytics Event Hubs Stream Analytics Strong experience in ETL/ELT
- data warehousing and data modelling
- including Kimball and/or Data Vault 2.0. Strong proficiency in T-SQL
- Python and Apache Spark. Practical experience with Azure DevOps
- CI/CD and Infrastructure-as-Code
- including Bicep
- ARM and PowerShell/CLI. Exposure to Databricks Unity Catalog and/or Microsoft Purview. Experience working in Agile environments. Strong communication
- problem-solving and stakeholder engagement skills.
Desired Work Experience:
- 5 to 10 years