The purpose of the Data Engineer role is to design, implement, test and maintain the architecture and infrastructure of a data platform and related components or systems. This includes, but is not limited to, data pipelines, customising and managing integration tools, databases, data lakes or warehouses, APIs and making data available for any related analytics systems such as reporting tools or Machine Learning pipelines. The Data Engineer responsibilities include working in a larger team to assist Analysts, Data Scientists or Machine Learning Engineers with their data needs and requirements. To do this job successfully, you need exceptional skills in programming with technical proficiencies in the latest data-related technologies. Just as important as the technical proficiencies are the solution design patterns, common application frameworks, and testing abilities that make up a fully developed and well-rounded Data Engineer.
Knowledge:
– Excellent knowledge of the latest required languages, development technologies and frameworks. o Examples of required back-end languages and technologies – Python Additional languages that are beneficial: Java, C# (.NET) o Examples of required data related technologies – Apache Druid, Apache Beam, Apache Kafka, Structured and unstructured databases, Apache Spark or Hadoop, Apache Airflow, ELK.
– Good working knowledge of designing, developing & maintaining end-to-end data-pipelines.
Good working knowledge of API design and development.
– Good working knowledge of data lake, database or data warehouse architecture and design.
– Good working knowledge of cloud technologies GCP/AWS/Azure.
– Good working knowledge of Agile development practises.
– Good working knowledge of Docker containers and Kubernetes.
– Knowledge of machine learning and analytics concepts are beneficial, but not required.
Desired Skills:
- GCP/AWS/Azure.
- Java C# (.NET)
- Python
- Docker containers and Kubernetes.
- Apache Spark or Hadoop Apache Airflow
Desired Qualification Level:
- Degree
About The Employer:
IoT.nxt is a global Internet of Things company offering dynamic software and hardware IoT solutions. A small start-up founded in 2015, the company has increased its headcount by a staggering 30x in 6 years. In 2019, telecoms giant Vodacom acquired a 51% stake in [URL Removed] a move indicative of the reputation and value the IoT solutions platform has established.