Our client is recruiting for a Senior Data Engineer to join them on a contracting opportunity. Profile Outline

  • A technology focussed individual, data engineers will need to design, configure, develop and deploy data transformations.

Key Responsibilities

  • Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed using ARM templates
  • Combine and curate data in a central data lake
  • Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, CosmosDB and TSI
  • Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory
  • Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies
  • Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail.

Minimum RequirementsKey Skills:

  • Python
  • PySpark
  • SQL
  • Solution Architecture
  • API Design
  • Containers
  • CI/CD
  • Azure Cloud
  • Data stream patterns and technology
  • Data engineering design patterns


  • Bachelor Degree in Computer Science, Software Engineering or Engineering

Specialist Areas:Unstructured Data

  • Applies to /wiki/spaces/DAGDG/pages/[Phone Number Removed]; and any products that handles large scale data such as images.
  • Strong experience of building large scale file shipping and pipelines, ideally using Azure services such as AzCopy and Azure Data Lake.
  • Experience of managing unstructured file meta-data, conversions, standardisation and related workflows.
  • Experience of building analysis jobs that scale on technologies such as Databricks or Azure Batch.

Learn more/Apply for this position