My client, a well-known telecommunications company, is looking for a Cloud DataOps Engineer to be based within the Technology Business Unit. You will be required to be a cross-functional person responsible for data ingestion pipelines, monitoring and quality. This includes automating data monitoring, alerting, and checking across various stages of data transformation and projection. Ensure that data quality controls are inherent in our data ingestion patterns and data governance to serve quality and trusted data to our customers. In addition, you will support and maintain production data pipelines Duties:
- Engaging with stakeholders to understand data pipeline demands and support use cases
- Work as a DataOps Engineer across multiple data platforms to integrate data, be responsible for data quality control, research data issues, and formulate data integrity solutions
- Be hands-on and champion the implementation of proactive monitoring, alerting, trend analysis, and robust applications
- Implementation and compliance to the data governance policy and related controls across multiple data platforms
- Develop and advance data reporting and data QC applications/systems
- Triage data research requests and issues to prioritize and systemize for effective resolutions
- Serve as technical contributor in enhancing and improving ETL processes and data ingestions across multiple platforms
- Competently communicate and collaborate with multiple product lines, customers, and development teams for anything data related
- Work effectively on an Agile team and collaborate well with your other team members
- Providing support to other teams if any optimization or the troubleshooting on the performance of the application to avoid errors
Requirements:
- 3 Year degree in Computer Science, Data Analytics or Data Science
- Post graduate qualification within the Data field would be advantageous
- 5+ years of overall IT experience with Big Data, Advance Analytics, Data Warehousing and Business Intelligence
- Relevant cloud certification at professional or associate level would be advantageous
- Solid experience in building batch and stream data pipelines
- Agile exposure, Kanban, or Scrum
- In-depth knowledge of data as a product & Information best practices
- Expert level experience in designing, building, and managing data pipelines for batch and streaming applications
- Experience with performance tuning for batch-based applications like Hadoop, including working knowledge using Nifi, Yarn, Hive, Airflow and Spark
- Experience with performance tuning streaming-based applications for real-time data processing using Kafka, Confluent Kafka, AWS Kenesis, GCP pub/sub or similar
- Working experience within the ML and Analytics lifecycle capabilities such as Data Pipelines, Data Processing, Data Storing, Model Lifecycle, Data Operations, Data Management & Data Governance
- Experience in using a wide range for data tools such as Hadoop, Spark, Hive, Cassandra, Airflow, Kafka, Flink, AWS services, GCP Services, etc
- Working experience with Cloud platforms such as OpenShift, AWS and GCP
- Solid working experience with CI/CD
- Java and Python programming ability would be an advantage