An award-winning leader in contact centre AI software is seeking a Senior Data Engineer to join their team on a remote working model. Open to SA citizens residing within South Africa.
In this role, you will have the chance to work on exciting projects, collaborate with experienced developers, and contribute to a positive and dynamic company culture. If the prospect of solving unique and challenging problems appeals to you we encourage you to apply for this opportunity.
As a Data Engineer specialising in Kafka pipelines and Elasticsearch, you will play a pivotal role in advancing our data infrastructure and analytics capabilities. Your responsibilities will include:
- Designing, implementing, and maintaining robust data pipelines using Kafka
- Leveraging your expertise in stream processing to optimise real-time data processing and streaming
- Developing and maintaining Elasticsearch clusters, fine-tuning them for high performance and scalability
- Extract, transform, and load (ETL) data into Elasticsearch for advanced analytics and search capabilities.
- Troubleshooting data pipeline and Elasticsearch issues
- Participating in the design and development of data models and schemas
- Continuously monitoring and optimising data pipeline and Elastic performance to meet growing data demands.
To excel in this role, you should possess the following qualifications and skills:
- Proven experience in designing and implementing data pipelines using stream processing technologies.
- Experience with end-to-end testing of analytics pipelines.
- In-depth expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability.
- Strong proficiency with data extraction, transformation, and loading (ETL) processes.
- Familiarity with data modeling and schema design for efficient data storage and retrieval.
- Proficiency in troubleshooting data pipeline and Elastic-related issues to ensure data integrity and availability.
- Solid programming and scripting skills, particularly in languages like Python, Scala, or Java.
- Knowledge of DevOps and automation practices related to data engineering.
Data Pipelines:
Kafka / stream processing
Python
Redis
Data Storage and Analysis:
Elasticsearch
Elasticsearch clusters management and optimisation
PostgreSQL
Desired Skills:
- Elasticsearch
- kafka
- data model
- data pipelines
- data stream
- data cluster
Desired Work Experience:
- 5 to 10 years
Desired Qualification Level:
- Diploma