Responsible to support data science projects and solutions by leveraging data and software engineering experience to solve a variety of use casesacross the client and for its customers. Expected to be highly skilled in batch and real-time data ingestion and data transformation withthe ability to design, build and scale data pipelines for efficient data integration and processing.
Duties:
- Build data integration pipelines used to move data between databases, file systems & other locations while tracking data quality according to measuresset by data owner.-
- Provide data support for specialized data centric services we provide to grow our data maturity:- This includes but are not limited to: Data modelling, database design, data quality assessments, data format conversion, data transformation, data toolselection.-
- Enable big data engineering by creating and managing code and tools created to efficiently obtain and move vast amounts of data between file systemsand/or databases while ensuring data integrity.
- Enable live/real-time data engineering by creating and managing code and tools built to efficiently obtain and move real-time data while ensuring dataintegrity.
- Engage with stakeholders to support the design and delivery of data science projects and solutions.
- Use data engineering techniques to solve business problems.
- Responsible for working with a team of data engineers to develop and maintain our cloud-based development and production platforms.
- Support ETL/ELT processes and data ingestion for exploratory data analysis and solution development.- Lead and develop a team of junior data engineers.
- Contribute to our agile way of work and our innovation culture.
- Up to date knowledge of data platforms and related technologies.
- Translate business requirements into system requirements.
- Consistent documentation of all implemented data integration pipelines and processes.- Support tools, data integration applications and infrastructure lifecycles via standard service management principles and processes.
Functional Knowledge:
Relational and non-relational database foundational knowledge; Data engineering knowledge base; Data integration concepts and terms ; Softwareand data engineering knowledge (Python, Scala, Shell scripting, JavaScript, Firebase); Cloud data streaming knowledge (Kafka, Sqoop, Pub-Subbased models/ services, Redis, RabbitMQ, Airflow, Beam, Cloud DataFlow, Apache Arrow, Apache Arrow Flight, Apache Iceberg); Data ops CI/CD andcloud deployment best practises knowledge; Cloud computing and platform management (GCP, Azure, AWS, etc.)
REQUIRED CERTIFICATION/PROFESSIONAL REGISTRATION
Data and cloud certifications will be advantageous (GCP, Azure, AWS) as well as certifications for other products in our stack (Airflow, Beam, CloudDataFlow, Apache Arrow, Apache Arrow Flight, Apache Iceberg, Tableau, Alteryx)
QUALIFICATIONS3-year degree/ diploma (NQF level 6) preferably in Computer Science, Mathematics, Statistics, Data Engineering or a related field. A relevant postgraduate degree will be an added advantage.
EXPERIENCE3-5 years relevant experience, of which at least 2 years must have been in a data engineering environment. Experience in ICT/ Telecommunications willbe an advantage. Experience with system and process analysis and design.
SPECIAL REQUIREMENTS
Experience with Google Cloud [URL Removed] to stay abreast of new data engineering frameworks and developments and to put them into practice.
KEY STAKEHOLDERS (INTERNAL/EXTERNAL)
Internal stakeholdersData engineer team
KEY DECISION-MAKING AUTHORITY
Operational
Desired Skills:
- Problem solving
- Communication (written and verbal)
- Stakeholder management
Desired Work Experience:
- 5 to 10 years
About The Employer:
Our client is in the ICT industry.