RESPONSIBILITES
- Designing and implementing data flows from the following sources into Azure data lake: IoT devices, API’s and traditional data sources, e.g. SQL databases, flat files & web scraping
- Design and implement a data lake in Microsoft Azure. The data lake should enable streams for both data science solutions as well as reporting solutions
- Implement processes to manage data quality checks and implement automated routines to ensure dataaccuracy, consistency and validity
- Implement a reporting suite to track important key performance indicators
- Maintain and support the above solutions
- Monitor and optimize data processing performance and scalability using Azure technologies, such as Azure Monitor
- Develop self-maintaining documentation with regards to data dictionaries, data exploration, data governanceand technical specifications
- Maintain data security and data tenancy requirement across national boundaries
- Collaborate with data scientists to understand their data requirements and assist in implementing technical solutions to meet those needs
GENERAL SKILLS
– Excellent verbal and written communication skills
– Data preparation, cleaning and quality assessment
– Research on various cloud platform topics
– Data analysis
– Curious, motivated and highly driven
– Interest in developing new skills
– Ability to work well in a pressured/high pace environment
INDUSTRY SKILLS
– Programming, database design, and best practises for data securityJob title Senior Data EngineerReports to Head of Data Science
– Experience working with unstructured datasets and building supporting data transformation, data structures, metadata, dependencies and workload management
– Working knowledge of building and optimizing high volume pipelines, architectures and data sets
– Some exposure to big data tools like Hadoop, Spark, Kafka, Azure Data Factory etc.
– Python, T-SQL, Azure technologies for streaming data, e.g. Azure Stream Analytics, Azure Databricks, Azure Data Explorer or Azure Synapse
– Experience in transforming and manipulating large datasets
– Some knowledge of object-oriented/object function scripting languages
– Team player with the ability to support the team
QUALIFICATIONS
– A tertiary qualification demonstrating proficiency in programming and/or data design and manipulation
– Strong Azure development skills: 5-10 years of experience
– Strong MS SQL development skills: 5-10 years of experience
– Intermediate Python development skills 5-10 years of experienc
Desired Skills:
- Data
- IOT
- Azure
- Python
Desired Work Experience:
- 5 to 10 years