- Data Engineers are responsible for building and maintaining Big Data Pipelines using Data Platforms.
- Data Engineers are custodians of data and must ensure that data is shared in line with the information classification requirements on a need-to-know basis.
- Data Engineers are also expected to: Stay up to date with the latest data engineering tools, technologies and industry trends. Identify opportunities for process improvements and automation to enhance the efficiency and reliability of data pipelines.
- Explore and evaluate new data engineering approaches and technologies to drive innovation within the organisation.
- Mentor, train and upskill members in the team.
Minimum Requirements:
Education
- Relevant IT / Business / Engineering Degree
- Candidates with one or more of the certifications are preferred: AWS Certified Cloud Practitioner, AWS Certified SysOps Associate, AWS Certified Developer Associate, AWS Certified Architect Associate, AWS Certified Architect Professional, HashiCorp Certified Terraform Associate
- Experience in working with Enterprise Collaboration tools such as Confluence, JIRA, etc.
- Experience developing technical documentation and artefacts.
- Knowledge of data formats such as Parquet, AVRO, JSON, XML, CSV, etc.
- Experience working with Data Quality Tools such as Great Expectations.
- Experience developing and working with REST API’s is a bonus.
- Basic experience in Networking and troubleshooting network issues.
- Knowledge of the Agile Working Model.
Techstack:
Cloud & Infrastructure
- Terraform, AWS CloudFormation
- AWS services: S3, Glue, Athena, CloudWatch, SNS, Lambda, Step Functions, DynamoDB
- AWS Parameter Store, Secrets Manager
- CodeBuild, Code Pipeline
Programming & Scripting
- Python 3.x, PySpark, Boto3
- SQL (Oracle, PostgreSQL)
- PowerShell, Bash
- Linux / Unix environments
Data Engineering & Big Data
- ETL pipeline development
- Big Data processing and analytics
- Kafka, Kinesis Streams, Kinesis Firehose
- AWS EMR, Amazon Redshift
DevOps & Containerisation
- Docker
Data Modelling & BI
- Technical data modelling and schema design (engineered, non-drag-and-drop)
- Business Intelligence (BI) experience
Enterprise Data Platforms
- Cloud Data Hub (CDH)
- CDEC Blueprint
Desired Skills:
- Data
- Data Engineering
- AWS