You’ll be working with the team towards building and maintaining a world class data platform where numerous data pipelines make data products available for consumption.
Our data pipelines gather data from numerous disparate data sources and bring everything together in a central place – the data platform – where it is made available for downstream consumption. Our data platform serves various types of workloads – from raw data streams, to reporting and aggregation, to advanced analytics.
You will work with a variety of technologies on a day-to-day basis, and we expect you to be hands on. The nature of the job requires constant upskilling and personal development, as our data platform relies on state-of-the-art technologies. Our data platform is built natively on Amazon Web Services (AWS) and integrates to a variety of data sources that are both on-prem and in the cloud.
Duties:
Your primary duty is to build & maintain our data platform in AWS.
- Write data engineering code and tests using Python and SQL, using a variety of libraries and runtime environments.
- Write infrastructure as code using Terraform and build our deployment pipelines in GitLab.
- Perform code reviews, work with teammates on merge requests, participate in design sessions, etc.
- Participate in the team’s Scrum processes and ceremonies.
- Engage with stakeholders to elicit requirements, demonstrate functionality, and drive a data-driven mindset in the organization.
- Operate the data platform and monitor its performance.
Competencies:
- Presenting and communicating Information (written and verbal).
- Analysing.
- Planning and organising.
- Ability to priorities and sequence.
- Delivering results and meeting customer expectations.
- Following instructions and procedures.
- Coping with pressures and setbacks.
- Can work in a team.
- Able to multi-task.
- Sharing knowledge.
- Creating and innovating.
- Persuading and influencing.
- Learning & researching.
- Adapting & responding to change.
Technical skills
- Good programming skills in both Python and SQL..
- Good database design skills and an understanding of various data modelling techniques and approaches.
- An ability to engineer cloud-first data solutions in the Amazon Web Services cloud.
- An understanding of the software development process, with proficiency in Git for version control and build & deployment pipelines in GitLab (or similar) using containerization (Docker).
- Ability to engineer metadata-driven approaches for sustainability and scalability.
Qualifications, Experience and Knowledge:
- Experience with infrastructure as code, ideally Terraform and Terragrunt.
- An understanding of and exposure to different database technologies.
- Bachelor of Science in Informatics or applicable IT qualification.
- 4-5 years’ experience for Intermediate Data Engineer.
Desired Skills:
- Data engineering
- terraform
- python
- sql