Data Engineers build and support data pipelines and datamarts built off those pipelines. Both must be scalable repeatable and secure. They help facilitating getting data from a variety of different sources, in the correct format, assuring that it conform to data quality standards and assuring that downstream users can get to that data timeously. This role functions as a core member of an agile team.
These professionals are responsible for the infrastructure that provide insights from raw data, handling and integrating diverse sources of data seamlessly. They enable solution handling large volumes of data in batch and realtime leveraging emerging technologies from both the big data and cloud spaces. Additional responsibilities include developing proof of concepts and implement complex big data solution with a focus on collecting, parsing, managing,
analysing and visualising large datasets. They know how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions.
JOB RELATED EXPERIENCE
– Retail (Desirable)
– Business Intelligence (Essential)
– Big Data (Desirable)
– Extract Transform & Load (ETL) Processes (Essential)
– Cloud AWS (Essential)
– Agile exposure, Kanban or Scrum (Essential)