Experienced Data Engineer required for an international software development company who creates, implements and maintains leading technology within online classifieds platforms in the marketplaces industry.
Fully remote working role. Candidates must be located in South Africa and either a SA citizen or permanent resident.
Required:
- BSc degree in Computer Science/information systems/Engineering or related technical field or equivalent work experience
- 3+ years related work experience
- Minimum of 2 years experience building and optimizing ‘big data’ data pipelines, architectures and maintaining data sets.
- Python + SQL (PostgreSQL, MS SQL) – Cloud services: AWS
- Proficiency in version control, CI/CD and Github
- Understanding/experience in Glue and PySpark
What will you do?
- Suggest efficiencies and execute on implementation of internal process improvements in automating manual processes, optimising data delivery, scabilibility of technology
- Implement enhancements and new features across data systems
- Improve streamline processes within data systems with support from Senior Data Engineer
- Test CI/CD process for optimal data pipelines
- Assemble large, complex data sets that meet functional / non functional business requirements
- Highly efficient in ETL processes
- Develop and conduct unit tests on data pipelines as well as ensuring data consistency
- Develop and maintain automated monitoring solutions
- Support reporting and analytics infrastructure
- Maintain data quality and data governance as well as upkeep of overall maintenance of data infrastructure systems i.e. AWS/database security
- Maintain data warehouse and data lake metadata, data catalogue, and user documentation for internal business users
- Ensure best practice is implemented and maintained on our database in terms of collations, database engines, indices etc.
Desired Skills:
- Python
- SQL
- PySpark
- AWS
- Airflow
- Docker
- Terraform
- Data engineering
- data pipelines
- data lakes
Desired Work Experience:
- 2 to 5 years
Desired Qualification Level:
- Degree