ENVIRONMENT:
EMPOWER data consumers by making a critical contribution to the design and development of new cloud workloads for Platform and Product teams of a dynamic Financial Institution seeking a highly technical Data Engineer. You will maintain and manage the existing cloud data environments and enable data producers to easily contribute to these environments, contribute new data features & and enhance/streamline existing processes e.g., improved re-use of code. Applications MUST HAVE AWS including its stack Glue, Redshift, S3, LakeFormation, EC2, VPC, IAM. You will also need at least 3 years’ proven experience in Computer Programming and Data Engineering, Python, Terraform, CI/CD, git and experience operationalizing Batch and/or Realtime data pipelines.
DUTIES:
Development and Design –
- Develop and architect cloud solutions for the data platforms, according to best practices, the Data Product Life Cycle (DPLC), and Way of Work (WoW), to develop enhancements to the existing data platforms.
- Gather context, requirements and define scope for new development requests.
- Work autonomously and take a high level of ownership in delivering software components.
- Document developed solutions thoroughly and in a way that facilitates ease of use.
- Include relevant automated tests in developed solutions, such as Unit or Integration Tests.
Support and Maintenance –
- Maintain the data platforms, by investigating and fixing reported issues.
- Provide support to the data platforms and the platform users, including standby duties, and responding to and resolving issues.
- Provide support to specific value stream projects using the data platforms, through the design and development of new cloud workloads, and providing technical guidance to these projects.
Research and Continuous improvement –
- Eagerly learn new relevant skills through just-in-time learning, by researching and deep-diving into the problem or feature that is currently under development. Have a willingness to tackle new work requiring knowledge and
- skills that are unfamiliar.
- Stay informed about developments in areas of technology that are relevant to the data platforms (e.g., AWS services, distributed data processing, source control tools, testing libraries, infrastructure as code, programming languages).
- Ensure platform contributions of fellow team members follow standards and best practices during code reviews.
- Seek to understand and learn from solutions contributed by fellow team members, and in turn be willing to knowledge share.
- Take ownership to improve own technical knowledge about the internal workings of the data platforms.
- Onboard and mentor new starters in the team.
- Collaborate and share experiences and knowledge with relevant communities by being an active participant with who is motivated and encourages others to participate.
REQUIREMENTS:
- At least 3 years’ proven experience in Computer Programming and Data Engineering, together with a relevant 3-year tertiary qualification OR At least 4 – 5 years’ proven experience in Computer Programming and Data Engineering.
Must Haves –
- AWS data stack (Glue, Redshift, S3, LakeFormation, EC2, VPC, IAM).
- Operationalizing Batch and/or Realtime data pipelines.
- Python, PySpark, or Scala.
- Version control in git, and CI/CD deployment.
- Relational database management systems.
- Provisioning cloud resources using Infrastructure as Code (Terraform).
- Cloud data lake and warehouse concepts.
- Software testing practices.
- Basic Terminal/Bash usage.
Solid understanding of –
- Banking systems environment.
- Banking business model.
Ideal to have –
At least 3 years’ proven experience in Cloud Data Engineering, particularly in AWS, together with a relevant 3-year tertiary qualification OR At least 4-5 years’ proven experience in cloud data engineering, particularly in AWS.
Proven experience in:
- Apache Spark, Hudi, Presto.
- Distributed Systems (Apache Hadoop, Amazon EMR).
- Advanced shell scripting.
- AWS serverless services (Step Functions, Lambda, EventBridge, API Gateway).
- Collaboration tools (JIRA, Confluence, [URL Removed] insights into Data Governance, Data Management, Data Quality, Data Security and Master Data Management.
While we would really like to respond to every application, should you not be contacted for this position within 10 working days please consider your application unsuccessful.
COMMENTS:
When applying for jobs, ensure that you have the minimum job requirements. OnlySA Citizens will be considered for this role. If you are not in the mentioned location of any of the jobs, please note your relocation plans in all applications for jobs and correspondence. Apply here [URL Removed] OR e-mail a Word copy of your CV to [Email Address Removed] and mention the reference number of the job.
Desired Skills:
- Data
- Engineer
- AWS