- Design, develop, and deploy data pipelines that meet the Client’s data needs, while ensuring scalability, reliability, and efficiency.
- Collaborate with the DevOps team to build and manage application pipelines using AWS CodePipeline and other relevant tools.
- Utilize Docker and Kubernetes for containerization and orchestration of data processing applications and services.
- Manage Apache Spark and Kafka deployments, optimizing performance and ensuring reliability in data processing tasks.
- Monitor and troubleshoot data pipeline performance, providing insights and solutions to improve efficiency and reduce errors.
- Implement security best practices and maintain compliance with relevant regulations for data engineering tasks.
- Continuously evaluate and implement new technologies and tools to enhance the team’s capabilities and efficiency in data engineering.
- Provide technical support to the Data Engineering team, assisting in the resolution of complex issues and challenges related to data processing and DevOps.
Non-Negotiables
- Strong knowledge of AWS services, including CodePipeline, CodeBuild, CodeDeploy, and CodeStar.
- Proficiency in Docker and Kubernetes for containerization and orchestration.
- Experience with Apache Spark and Kafka deployments, including configuration, monitoring, and troubleshooting.
- Familiarity with data pipeline tools and technologies such as Hadoop, Apache NiFi, or Apache Beam.
- Excellent problem-solving skills and the ability to work both independently and as part of a team.
- Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.
Desired Skills:
- Docker
- Kubernetes
- Apache Spark
- Kafka
Desired Qualification Level:
- Degree