Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.
Responsibilities
- Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
- Develop and manage RESTful APIs for data access and integration.
- Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
- Optimize Snowflake data warehouse performance and manage data models.
- Implement data quality, governance, and security best practices.
- Work with Databricks for data processing, transformation, and ML model support.
- Automate data validation and reconciliation processes.
- Document data architecture, pipelines, and integration processes.
Qualifications
- Bachelors or Masters degree in Computer Science, Engineering, or a related field.
- 8+ years of experience in a data modeling, data warehousing, and data integration role.
Required Skills
- Strong experience with Matillion ETL and Snowflake.
- Proficiency in Databricks
- Experience in data management requirements, integration platforms, and have expertise with APIs.
- Experience with API development using frameworks.
- Familiarity with Databricks and integrating Machine Learning workflows.
- Solid understanding of data warehousing, data modeling, and ELT best practices.
- Experience with CI/CD pipelines, version control (Git), and DevOps practices.
- Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
- Experience with data modeling, data warehousing, and data integration.
- Familiarity with cloud platforms and data services.
- Understanding of data governance, security, and compliance requirements
Job Description
Summary
Our client is seeking a highly experienced Senior Data Engineer to lead the design, development, and optimization of data pipelines, APIs, and data platforms. This role will focus on ETL/ELT processes using Matillion and Snowflake, API development, and integration with machine learning workflows and Databricks. The ideal candidate will have a strong background in data engineering, cloud platforms, and modern data architecture.
Responsibilities
- Design, build, and maintain ETL/ELT pipelines using Matillion and Snowflake.
- Develop and manage RESTful APIs for data access and integration.
- Collaborate with data scientists and ML engineers to integrate data pipelines with machine learning workflows.
- Optimize Snowflake data warehouse performance and manage data models.
- Implement data quality, governance, and security best practices.
- Work with Databricks for data processing, transformation, and ML model support.
- Automate data validation and reconciliation processes.
- Document data architecture, pipelines, and integration processes.
Qualifications
- Bachelors or Masters degree in Computer Science, Engineering, or a related field.
- 8+ years of experience in a data modeling, data warehousing, and data integration role.
Required Skills
- Strong experience with Matillion ETL and Snowflake.
- Proficiency in Databricks
- Experience in data management requirements, integration platforms, and have expertise with APIs.
- Experience with API development using frameworks.
- Familiarity with Databricks and integrating Machine Learning workflows.
- Solid understanding of data warehousing, data modeling, and ELT best practices.
- Experience with CI/CD pipelines, version control (Git), and DevOps practices.
- Strong proficiency in SQL and at least one programming language (e.g., Python, Java, or Scala).
- Experience with data modeling, data warehousing, and data integration.
- Familiarity with cloud platforms and data services.
- Understanding of data governance, security, and compliance requirements
Desired Skills:
- ETL and Snowflake
- Databricks
- APIs
- CI/CD pipelines
- Strong SQL
- Cloud platforms and data services