Our client is recruiting for a Senior Data Engineer to join them on a contracting opportunity. Profile Outline
- A technology focussed individual, data engineers will need to design, configure, develop and deploy data transformations.
Key Responsibilities
- Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed using ARM templates
- Combine and curate data in a central data lake
- Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, CosmosDB and TSI
- Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory
- Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies
- Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail.
Minimum RequirementsKey Skills:
- Python
- PySpark
- SQL
- Solution Architecture
- API Design
- Containers
- CI/CD
- Azure Cloud
- Data stream patterns and technology
- Data engineering design patterns
Qualifications:
- Bachelor Degree in Computer Science, Software Engineering or Engineering
Specialist Areas:Unstructured Data
- Applies to /wiki/spaces/DAGDG/pages/[Phone Number Removed]; and any products that handles large scale data such as images.
- Strong experience of building large scale file shipping and pipelines, ideally using Azure services such as AzCopy and Azure Data Lake.
- Experience of managing unstructured file meta-data, conversions, standardisation and related workflows.
- Experience of building analysis jobs that scale on technologies such as Databricks or Azure Batch.