- Data Engineers build and support data pipelines and DataMart’s built off those pipelines.
- Both must be scalable, repeatable and secure.
- The Data Engineer helps to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously.
- This role functions as a core member of an agile team.
- These professionals are responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly.
- They enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.
- Additional responsibilities include developing proof of concepts and implements complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets.
- They know how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions.
- Data Engineering is a technical job that requires substantial expertise in a broad range of software development and programming fields.
- These professionals have a knowledge of data analysis, end user requirements and business requirements analysis to develop a clear understanding of the business need and to incorporate these needs into a technical solution.
- They have a solid understanding of physical database design and the systems development lifecycle.
- This role must work well in a team environment.
Qualifications:
- 4 years Bachelor’s degree in computer science, computer engineering, or equivalent work experience AWS Certification at least to associate level
Experience:Essential:
- 5+ years Data engineering or software engineering
- 3-5 years demonstrated experience leading teams of engineers
- 2+ years Big Data experience
- 5+ years’ experience with Extract Transform and Load (ETL) processes
- 2+ years Could AWS experience
- At least 2 years demonstrated experience with agile or other rapid application development methods – Agile exposure, Kanban or Scrum
- 5 years demonstrated experience with object oriented design, coding and testing patterns as well as experience in engineering (commercial or open source) software platforms and large scale data infrastructures.
Desirable:
5+ years Retail Operations experience