Our client, specialising in the mining space has an amazing opportunity for a Senior Data Engineer to join their team and will be assisting the data science team in South Africa as well as in the United States.
The Senior Data Engineer will assist the data science division in collecting, structuring and enabling access to data within a cloud and on-premises environment, assisting the business to take actionable steps to improve its performance and meet its strategic goals. Duties include implementing data strategies, implementing automated data pipelines, finding ways to improve a company’s efficiencies with regards to movement of data between environments and providing support to data analysts and data scientists in their efforts to analyze the data and extract insights. The Senior Data Engineer will form part of the data science division on the outset with a view of growing the data engineering function as the business grows.
KPA’s will be (but not limited to):
- -Designing and implementing data flows from the following sources into Azure data lake: IoT devices, API’s and traditional data sources, e.g. SQL databases, flat files & web scraping
- Design and implement a data lake in Microsoft Azure. The data lake should enable streams for both data science solutions as well as reporting solutions
- Implement processes to manage data quality checks and implement automated routines to ensure data accuracy, consistency and validity
- Implement a reporting suite to track important key performance indicators
- Maintain and support the above solutions
- Monitor and optimize data processing performance and scalability using Azure technologies, such as Azure Monitor
- Develop self-maintaining documentation with regards to data dictionaries, data exploration, data governance and technical specifications
- Maintain data security and data tenancy requirement across national boundaries
- Collaborate with data scientists to understand their data requirements and assist in implementing technical solutions to meet those needs
Minimum Requirements:
- A tertiary qualification demonstrating proficiency in programming and/or data design and manipulation
- Strong Azure development skills: 5-10 years of experience
- Strong MS SQL development skills: 5-10 years of experience
- Intermediate Python development skills 5-10 years of experience
- Programming, database design, and best practises for data security
- Experience working with unstructured datasets and building supporting data transformation, data structures, metadata, dependencies and workload management
- Working knowledge of building and optimizing high volume pipelines, architectures and data sets
- Some exposure to big data tools like Hadoop, Spark, Kafka, Azure Data Factory etc.
- Python, T-SQL, Azure technologies for streaming data, e.g. Azure Stream Analytics, Azure Databricks, Azure Data Explorer or Azure Synapse
- Experience in transforming and manipulating large datasets
- Some knowledge of object-oriented/object function scripting languages
- Team player with the ability to support the team
General Skill Requirements:
- Excellent verbal and written communication skills
- Data preparation, cleaning and quality assessment
- Research on various cloud platform topics
- Data analysis
- Curious, motivated and highly driven
- Interest in developing new skills
- Ability to work well in a pressured/high pace environment
Please Note: Only shortlisted candidates will be contacted
Desired Skills:
- Data Engineer
- Senior Data Engineer
- Azure
- MS SQL
- Python
- Data Science