Psybergate is an IT company that builds bespoke software solutions and provides highly skilled resources to its clients.
We are looking for a Data Engineer to join our financial services client based in Sandton for a 12-months contract role.
What you will be doing:
We are looking for an experienced Data Engineer with strong Python and SQL skills, as well as hands-on experience with Kafka, Docker, and Kubernetes. The ideal candidate will have a proven track record in building bespoke data processing pipelines, developing microservices, and designing data table structures and schemas. You will collaborate with cross-functional teams to integrate and optimize data systems, while working with both structured and unstructured data.
Key Responsibilities:
- Design and Build Data Pipelines: Develop and maintain custom, scalable data pipelines for processing large datasets, including unstructured text and document data.
- Integrate Data Systems: Collaborate with diverse teams to understand data sources and build robust integrations with various internal and external systems.
- Build Microservices: Develop microservices to handle data processing tasks, ensuring they are scalable, secure, and efficient.
- Optimize Performance: Design and optimize table structures, schemas, and database queries for improved data retrieval and storage performance.
- Kafka and Real-time Data: Leverage Kafka for real-time data streaming and processing, ensuring high availability and low-latency data delivery.
- Containerization & Orchestration: Use Docker and Kubernetes for deploying and managing data applications in production environments.
- Collaborate with Teams: Work closely with data scientists, analysts, and other engineers to define requirements and deliver high-quality, reliable data solutions.
- Ensure Data Quality: Maintain data integrity and implement best practices to ensure the consistency and quality of the data being processed
What we are looking for:
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
- 5 years experience as Date Engineer
- Strong proficiency in Python: Experience in developing scalable data applications, automating processes, and implementing algorithms.
- Strong SQL skills: Ability to design efficient queries and optimize database performance.
- Hands-on experience with Kafka, including setting up and managing streams, and integrating with data pipelines.
- Proficiency with Docker and Kubernetes for containerization and orchestration in production environments.
- Experience building bespoke data processing pipelines: Expertise in handling large volumes of data from multiple sources and optimizing processing workflows.
- Experience building microservices to deliver modular, scalable, and maintainable data solutions.
- Experience designing table structures and schemas that meet both performance and business requirements.
- Ability to collaborate with diverse teams, understanding their needs and building integrations that allow for efficient data flow.
- Java experience is advantageous, particularly for implementing additional data processing logic or integration points.
- Experience with unstructured text and document data is a plus, especially in processing and transforming such data for analytical purposes.
Please note that if you do not hear from us within 3 weeks, consider your application unsuccessful.
Please note that most of our positions are remote however candidates should be residing within the traveling distance as circumstance of the opportunity can change.
Desired Skills:
- SQL
- Python
- Kafka
- Docker
- Kubernetes