A technology focussed individual, data engineers will need to design, configure, develop and deploy data transformations
- Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed
- Combine and curate data in a central data lake
- Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, CosmosDB and TSI
- Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory
- Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies.
- Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail.
Minimum Requirements
- Phyton – Proficient
- PySpark – Proficient
- SQL – Competent
- Solution Architecture – Competent
- API Design – Competent
- Containers – Competent
- CI/CD – Competent
- Azure Cloud – Competent
- Data Stream patterns and technology – Proficient
- Data stream patterns and technology – Proficient
- Data engineering design patterns – Competent
- Mining data – Beneficialy