About the Role:
- Your role as a Data Engineer, is to develop ETL/ELT pipelines, to unify these data sources, using a variety of technologies.
- Since you will be working with big data sources, you will be expected to be able to work with both real time data streaming sources as well as batch data sources.
- You will be expected to use Spark for big data workloads as well as SQL/T-SQL for structured data workloads.
- Once the data has been unified, you will be responsible for the transformation, cleansing and validation of the data to generate rich data assets for Data Science and Business Intelligence workloads.
- This will include the modelling and engineering of a data warehouse, data models and data marts for various downstream processes and consumers.
Qualifications:
- A bachelor’s degree in Computer Science or another IT related field of study is preferred.
- Technology Certifications in Databricks, Azure Data Engineering and other technologies is highly advantageous.
Experience:
- 1-2 years as a data engineer.
- Experience with Delta Lake and Spark is highly advantageous.
- Experience with cloud technologies specifically Azure is preferred.
TECHNOLOGIES:
Languages, Frameworks:
- Python
- Spark
- Scala
- T-SQL
- Delta Lake
Cloud Technology
- Azure Synapse Analytics
- Databricks
- Azure Data Lake
- Azure Event Hub
- Azure Functions
Desired Skills:
- Data Engineer
- Python
- Spark
- Scala
- T-SQL
- Delta Lake
- Azure Synapse Analytics
- Databricks
- Azure Data Lake
- Azure Event Hub
- Azure Functions
About The Employer:
IT Services – Consulting