ENVIRONMENT:
AN award-winning leader in contact centre AI software seeks a passionate Data Engineer with expertise in Kafka pipelines and a thorough understanding of Elastic, looking to contribute to cutting-edge technology and make a difference in the Financial Services industry. You will design, implement, and maintain robust data pipelines; troubleshoot data pipeline and Elasticsearch issues while ensuring data infrastructure aligns with business needs. The ideal candidate will have proven experience in designing and implementing data pipelines including End-to-End Testing of analytics pipelines & managing and optimising Elasticsearch clusters, including performance tuning and scalability. You will also be proficient with Python, Scala or Java and DevOps.
DUTIES:
- Design, implement, and maintain robust data pipelines, ensuring the efficient and reliable flow of data across systems.
- Develop and maintain Elasticsearch clusters, fine-tuning them for high performance and scalability.
- Collaborate with cross-functional teams to Extract, Transform, and Load (ETL) data into Elasticsearch for advanced analytics and search capabilities.
- Troubleshoot data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting.
- Participate in the design and development of data models and schemas to support business requirements.
- Continuously monitor and optimise data pipeline and Elastic performance to meet growing data demands.
- Collaborate with Data Scientists and Analysts to enable efficient data access and query performance.
- Contribute to the evaluation and implementation of new technologies and tools that enhance Data Engineering capabilities.
- Demonstrate strong analytical, problem-solving, and troubleshooting skills to address data-related challenges.
- Collaborate effectively with team members and stakeholders to ensure data infrastructure aligns with business needs.
- Embody the company values of playing to win, putting people over everything, driving results, pursuing knowledge, and working together.
- Implement standards, conventions and best practices.
REQUIREMENTS:
- Proven experience in designing and implementing data pipelines.
- Experience with End-to-End Testing of analytics pipelines.
- Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability.
- Strong proficiency with data extraction, transformation, and loading (ETL) processes.
- Familiarity with data modeling and schema design for efficient data storage and retrieval.
- Good programming and scripting skills using languages like Python, Scala, or Java.
- Knowledge of DevOps and automation practices related to Data Engineering.
As a Data Engineer with a focus on Kafka pipelines and Elastic, you will work with the following technologies:
Data Pipelines:
- Kafka / ksqlDB
- Python
- Redis
Data Storage and Analysis:
- Elasticsearch, cluster management and optimisation
- AWS S3
- PostgreSQL
DevOps:
- AWS
Advantageous –
- Experience with Data Engineering in an Agile / Scrum environment.
- Familiarity with ksqlDB / Kafka or other stream processing frameworks.
- Familiarity of Data Lakes and the querying thereof.
- Experience with integrating Machine Learning models into data pipelines.
- Familiarity with other data-related technologies and tools.
ATTRIBUTES:
- Strong analytical and problem-solving abilities, with a keen attention to detail.
- Excellent communication and collaboration skills to work effectively with cross-functional teams.
- A commitment to staying up to date with the latest developments in Data Engineering and technology.
- Alignment with company values and a dedication to driving positive change through data.
While we would really like to respond to every application, should you not be contacted for this position within 10 working days please consider your application unsuccessful.
COMMENTS:
When applying for jobs, ensure that you have the minimum job requirements. OnlySA Citizens will be considered for this role. If you are not in the mentioned location of any of the jobs, please note your relocation plans in all applications for jobs and correspondence. Apply here [URL Removed] OR e-mail a Word copy of your CV to [Email Address Removed] and mention the reference number of the job.
Desired Skills:
- Senior
- Data
- Engineer