Primary Duties and Responsibilities:
The Data Engineer will have a knack for data analysis, data manipulation and data modelling. The consultant will be responsible for understanding and driving the overall technical vision and planning of a client’s organisation and translating business needs into technical strategy.
Required Qualifications:
- Tertiary degree, diploma or certificate in a related field (BSc Computer Science, B.IT or Informatics related degrees).
- DP-203: Data Engineering on Microsoft Azure Certification.
Experience and Knowledge:
- 8-10 years’ working experience as a Data Engineer / Database Developer.
- Experience in data mining, large scale data modelling and business requirements gathering/analysis.
- Understanding and working experience in data integration and transformation.
- Experience implementing data modelling methodologies like Dimensional Modeling and / or Data Vault.
- Working knowledge of data quality processes and master data management.
- Experience implementing design support systems using Database Management Systems (DBMS) such as SQL Server or Oracle.
- Proficiency in designing and implementing data integration and ETL solutions using SSIS, Azure Data Factory and / or SQL Server stored procedures.
- Understanding of several Big Data technologies like Hadoop, MapReduce and Spark as well as event processing or message ingestion services like Kafka, Event Hub and Stream Analytics.
- Experience in database query languages such as T-SQL, ANSI SQL, PL/SQL.
- Some experience developing software solutions using Visual Basic, C++, C#, Java or Python.
- Experience using SQL Server management Studio and Visual Studio.
- Experience implementing solutions using Azure SQL databases, Azure Synapse (Previously SQL Data Warehouse), Azure Storage Accounts (Data Lake) and / or Databricks.
- Analytical mind and business acumen
- Additional skills in the following will be taken into consideration: Tableau, Power BI, strong math skills (e.g. statistics, algebra), Scala, Python or R.
Key Responsibilities:
- Identify valuable data sources and automate collection processes.
- Undertake preprocessing of structured and unstructured data.
- Analyze large amounts of information to discover trends and patterns.
- Data Modelling (Relational and Star Schema).
- Database design.
- Database development.
- Data Warehouse Design – Build and Development.
- Database Administration.
- Database Performance Tuning and Optimisation.
- Present information using data visualization techniques.
- Propose solutions and strategies to business challenges.
- Collaborate with engineering and product development teams.
- An understanding and hands on experience on Hadoop/Spark based distributed storage and computing frameworks.
- Real-Time analytics and batch processing.
- Strong experience in architecting analytical applications in cloud environment such as Amazon Web Services and Microsoft Azure.
Desired Skills:
- Database developer
- ETL
- data integration
- Power BI
- SQL
- SSIS
- Data engineering
- Big data