We are seeking a Data Engineer (Hadoop and NIFI essetial)
Job Purpose
To prepare data products and advance technical thought leadership for the Enterprise, to deliver fit for purpose data products for consumption by advanced analytics. Provide clean, usable data to whomever may require it.
Minimum Experience
8years+ experience across multiple date related disciplines
Able to gain a quick and in-depth understanding of multiple integration and related technologies and how they are deployed as part of a framework in a specific pattern.
Technologies currently include:
? Integration technologies: Apache NiFi (full stack) essential
? Storage technologies: SQL, Oracle, DB2WH, Hadoop
? Understanding of how Analytical technologies are deployed and work.
Good understanding of the various Security technology components and how they work in the Integration Landscape,
? Kerberos, TLS 1.2, JDBC type 4, AD, Identity IQ, etc.
? Beyond Trust (PAM) solution patterns for Linux
- Design, construct, test and maintain highly scalable data pipelines.
- Ensure pipelines meet business requirements and industry practices.
- Build high-performance prototypes, and proof of concepts.
- Research opportunities for data acquisition and new uses for existing data.
- Develop data set processes for data modelling, mining and production.
- Employ a variety of languages and tools (e.g. scripting languages) to marry systems together.
- Recommend ways to improve data reliability, efficiency and quality.
- Collaborate with data architects, modellers and IT team members on project goals.
- Determine in conjunction with the Data Architect what data management systems are appropriate.
- Obtain information from the business users to determine which data is needed for analysis to determine and provide data which is needed.
- Support the achievement of the business strategy, objectives and values
- To elaborate on understanding the high-level business needs, determine the architectural fit
- Understand the Current Architecture Landscape, Target State and Road Map
- To classify the project and to identify the governance that must be followed for the project
- Insure end-to-end implementation and productionalisation of designs.
- Canvas the client environment to discover details of the current technical environment, standards, and any other constraints that may be important for the solution to consider when making requirements and design decisions
- To create a high-level, shared vision and scope of the proposed architecture, allowing it to be communicated to the sponsor and other stakeholders
- Identify the business and technology context and forces that are driving the solution, develop a conceptual overview of the solution, define alternative technical solution approaches and develop a recommendation that balances business needs, strategic architecture requirements and technical viability
- Present the solution at the relevant governance forums to obtain approval for solution
8+ years working with open source Big Data technology stacks (Apache Nifi, Spark, Hadoop/HDFS, Hive, YARN, Map Reduce etc.) with at least 3 years as a senior software development engineer and/or technical lead with similar roles.
Experience with JSON, JOLT, SQL and NoSQL databases.
Successful track record delivering high quality products on time while working in teams and following Agile methodologies.
Skills (Technologies, systems or software knowledge etc.):
- Extensive Apache NiFi experience essential
- CDC DB2
- Kafka
- Hadoop Stack
- Data Warehousing
- Streaming
- Cloud experience advantageous
Desired Skills:
- Data
- Engineer
- Hadoop