We are looking for a Data Engineer who is skilled at creating data pipelines for ingest, transformation and storage of data usually at scale, and familiar with standing-up analytics clusters.
Our engineers use modern agile technical practices including Continuous Integration and Deployment, TDD, XP etc. to deliver timely and pragmatic solutions as well as helping others to do their jobs in a more effective way.
- We look for people with many of the following skills and experience:
- Significant experience working on Data pipelines, platforms and projects.
- Experience working in one or more of the main cloud providers (AWS, GCP, Azure).
- Significant experience of developing solutions in some of SQL (+ variants), Python, Scala, Java.
Experience in
- Working with a variety of RDBMS databases, ideally in production environments
- Creation and support of highly available data pipelines:
- Using ETL platforms and/or coded equivalents
- Using scheduling tools and/or familiarity with working in streaming environments
- Ensuring reliability and availability
- working with other data storage and processing technologies including:
- Spark
- Document databases
- Graph databases
- Columnar databases
- Working with configuration control (Git/Github, Terraform, ansible etc.) and CI/CD tools.
- Understands and can support data scientists or business intelligence teams- data prep,
- Mentoring and support functions.
Desired Skills:
- Python
- Scala
- Java
- AWS
- Azure
- CI/CD
- RDBMS
- ETL
- Data Pipelines
- Data Storage
- Git
- Github
- Spark
- Data engineering
Desired Work Experience:
- 5 to 10 years
Desired Qualification Level:
- Degree
About The Employer:
Market leading Global Consulting group are seeking a Lead Data Engineer to join their growing team. Must be skilled in creating data pipelines. Must be experience in CI/CD and TDD