An International Mining Company in the GAUTENG has a CONTRACT vacancy for a Data Engineer in the IM Department
The purpose of Global Information Management (IM) is to enable the delivery of business processes, communication, collaboration and knowledge management through the deployment, support and maintenance of enterprise IT/OT technology, infrastructure, applications and Mine Technical systems.
PLEASE NOTE; TRAVELING MAY BE REQUIRED
PLEASE NOTE THIS IS A CONTRACT POSITION. ONLY APPLY IF YOU QUALIFY IN FULL: TRAVELING MAY BE REQUIRED.
An undergraduate qualification (Bachelor’s degree or equivalent) in the relevant IM discipline and/or Technical competencies and certification with relevant years of experience in a similar role.
A technology focused individual, data engineers will need to design, configure, develop and deploy data transformations.
- Develop ETL pipelines. The data transformations will be developed in Azure Databricks using Python and on Azure SQL using T-SQL and deployed using ARM templates
- Combine and curate data in a central data lake
- Serve data for application and analytics through a variety of technologies such as SQL, Server Synapse, CosmosDB and TSI
- Build transformation pipelines into dimensions and facts and therefore a strong knowledge of standard BI concepts is mandatory
- Build stream pipelines leverage IoT Hub, Event Hub, Databricks streaming and other Azure stream technologies.
Mental Processing Ability
Evidence of ability to complete work of 2 complexity.
This work involves analysis, reflection and the use of data over a more extended period, to anticipate problems. It requires knowledge of the overall system and focuses on the direct application and improvement of systems
Application/Energy
Consistently displays a positive and engaging manner
Motivated, continuously work to best of ability to meet performance criteria
Strives to do things significantly better
Knowledge
Formal qualifications:
An undergraduate qualification (Bachelor’s degree or equivalent) in the relevant IM discipline and/or Technical competencies and certification with relevant years of experience in a similar role.
Role-specific knowledge:
· Data Lake
· Data Modeling
· Data Architecture
· Azure Data Environment
Specialist Areas:
Unstructured Data – Applies to /wiki/spaces/DAGDG/pages/[Phone Number Removed]; and any products that handles large scale data such as images.
Strong experience of building large scale file shipping and pipelines, ideally using Azure services such as AzCopy and Azure Data Lake. Experience of managing unstructured file meta-data, conversions, standardisation and related workflows. Experience of building analysis jobs that scale on technologies such as Databricks or Azure Batch.
Safety Knowledge:
Provides a consistent outstanding role model concerning safety practices with a deep understanding of the importance of safety
Technical Skills
Key Skills:
· Phyton – Proficient
· PySpark – Proficient
· SQL – Competent
· Solution Architecture – Competent
· API Design – Competent
· Containers – Competent
· CI/CD – Competent
· Azure Cloud – Competent
· Data Stream patterns and technology – Proficient
· Data stream patterns and technology – Proficient
· Data engineering design patterns – Competent
· Mining data – Beneficialy
Social Process Skills
Ability to:
– manage self
– develop self
– engage with team members and stakeholders
– appreciate different thoughts and opinions
– understand different contexts and adjust behaviours appropriately based on that understanding
– Work in a fluid environment with changing requirements whilst maintaining absolute attention to detail
Desired Skills:
- Data Lake
- Data Modeling
- Azure Data Environment
- Data Architecture