- Design, develop, and maintain scalable data pipelines to ingest, process, and store structured and unstructured data from multiple sources.
- Develop ETL/ELT processes to transform raw data into clean, analysable formats for reporting, analytics and dashboards.
- Work with cloud data platforms
- Ensure data security and compliance
- Integrate and consolidate data from multiple sources, including SSE platforms, enterprise systems, and external APIs.
- Optimise data pipelines for performance, reliability, and scalability.
- Collaborate with other teams to ensure timely and accurate data delivery.
- Perform data validation and quality checks, identifying anomalies, gaps or inconsistencies and resolving them proactively.
- Automate data ingestion, transformation, and integration.
- Support analytics, machine learning, and business initiatives where necessary.
- Continuously evaluate and recommend improvements in data architecture, cloud integrations, and security practices.
- Troubleshooting and resolving data-related issues across the environments, ensuring minimal downtime or disruption.
- Own the development, maintenance, and operational support of a cloud-based AI platform, ensuring scalability, reliability, and performance for AI workloads.
- Translate business requirements into data-driven solutions.
- DevOps
Minimum Requirements:
- Qualification in Computer Science, Information Technology, Data/Software Engineering, Data Science or equivalent.
- Cloud Computing or Analytics is a plus.
- Data Science / Machine Learning
- Data Engineering
- MLOps
- Python
- CI/CD
- IaC
- Terraform
- APIs
- Any additional responsibilities assigned in the Agile Working Model (AWM) Charter
- Knowledge of cloud data platforms
- Experience with big data frameworks
- Competency in streaming technologies
- Understanding data warehousing and Lake house concepts
- Experience with Infrastructure-as-code and DevOps practices
- Understanding of data security, governance, and compliance.
- Experience in implementing and monitoring solutions
Desired Skills:
- Data Engineering
- ETL
- Python