Our client, a leading SA bank, is looking for both intermediate & senior Data Engineers to join their teams in the Western Cape! You would contribute to the design and development of the evolving data platforms and maintain the existing cloud data environments, through knowledge sharing, building new data features and enhancing existing processes.
Responsibilities
- Develop and architect cloud solutions for the data platforms, according to best practices, to develop enhancements to the existing data platforms.
- Gather context, requirements and define scope for new development requests.
- Work autonomously and take a high level of ownership in delivering software components.
- Document developed solutions thoroughly and in a way that facilitates ease of use.
- Include relevant automated tests in developed solutions, such as unit or integration tests.
- Maintain the data platforms, by investigating and fixing reported issues.
- Provide support to the data platforms and the platform users, including standby duties, and responding to and resolving issues.
- Provide support to specific value stream projects using the data platforms, through the design and development of new cloud workloads, and providing technical guidance to these projects
- Eagerly learn new relevant skills through just-in-time learning, by researching and deep-diving into the problem or feature that is currently under development. Have a willingness to tackle new work requiring knowledge and skills that are unfamiliar.
- Stay informed about developments in areas of technology that are relevant to the data platforms (e.g. AWS services, distributed data processing, source control tools, testing libraries, infrastructure as code, programming languages)
- Ensure platform contributions of fellow team members follow standards and best practices during code reviews.
- Seek to understand and learn from solutions contributed by fellow team members, and in turn be willing to knowledge share.
- Take ownership to improve own technical knowledge about the internal workings of the data platforms.
- Onboard and mentor new starters in the team.
- Collaborate and share experiences and knowledge with relevant communities by being an active participant with who is motivated and encourages others to participate
- Senior level individuals will act as technical lead on projects
Requirements
- At least 3 years’ proven experience in data engineering, preferably in a Cloud environment (AWS beneficial), together with a relevant 3 year tertiary qualification
- Application development with scripting languages: Python, Pyspark or Scala
- Relational database management systems
- Provisioning cloud resources using Infrastructure as Code (Terraform)
- Core AWS services (S3, EC2, VPC, IAM) & AWS serverless services (Step Functions, Lambda, EventBridge, API Gateway)
- Cloud or AWS data lake and warehousing services (Glue, LakeFormation, EMR)
- Software Version Control systems (git) and deployment tools (CI/CD)
- AWS Well-Architected Framework.
- Collaboration tools (JIRA, Confluence, [URL Removed] Batch and/or Realtime data pipelines.
- Trusted insights into Data Governance, Data Management, Data Quality, Data Security and Master Data Management.
Beneficial:
- Banking systems environment
- Banking business model
- AWS data stack (AWS Glue, AWS Redshift, AWS S3, AWS LakeFormation)
- Apache Spark, Hudi, Presto.
- Distributed Systems (Apache Hadoop, Amazon EMR)
- Advanced shell scripting.
- Infrastructure as Code (Terraform)
Desired Skills:
- Data
- Engineer
- Python