We are looking for a specialised Cloud Engineer to build, secure, and maintain the underlying infrastructure for modern data platforms. Unlike traditional cloud engineering roles focused on general application hosting, this position is dedicated to the unique requirements of data workloads — including large-scale storage, high-throughput networking, and specialised compute clusters for Spark and SQL.
You will bridge the gap between DevOps and Data Engineering, ensuring data scientists and analysts have a robust, secure, and automated platform for building pipelines and models. The role focuses on the core infrastructure of the data ecosystem, primarily within Databricks, AWS, and Azure environments.
Responsibilties
Platform Architecture:
- Design and deploy scalable cloud infrastructure for data lakes and analytics platforms using Infrastructure as Code (Terraform)
Security & Governance:
- Implement identity management (IAM / Entra ID), network security (Private Link / VPC), and governance controls (Unity Catalog)
Automation:
- Build CI/CD pipelines for infrastructure and data products.
- Automate provisioning of compute resources and workspaces
Observability:
- Monitor cost, performance, and reliability of data workloads
Enablement:
- Create self-service infrastructure patterns for Data Engineers
RequirementsRequirements
- Minimum of 5 years professional cloud engineering experience
- Experience in data engineering and Databricks is highly desirable
Skills and Experience
Core Skills, Tools & Frameworks
Infrastructure as Code:
- Terraform (modules, state, workspaces)
- CloudFormation, Bicep, or Crossplane advantageous
Containerisation & Orchestration:
- Docker and Kubernetes (EKS, AKS, or self-managed)
CI/CD & Version Control:
- Git, Azure DevOps, GitHub Actions
Scripting & Automation:
- Python and Bash
- DBX CLI, AWS CLI, Azure CLI, REST APIs
Cloud Security:
- Least Privilege, RBAC, encryption, secrets management
Cloud Networking:
- VNET/VPC design, Private Link, DNS, Firewalls
Data Platform Architecture:
- Medallion architecture, Delta Lake, Data Vault
Observability:
- Cost management, logging, alerting
Data Workload Understanding:
- PySpark, SQL, dbt, Spark Structured Streaming
AWS Platform Skills:
Storage: S3 configuration, lifecycle policies, intelligent tiering
Identity: IAM roles, cross-account access, federation
Networking: VPC, Transit Gateway, Route53
Serverless: Lambda, Step Functions
Data Services: Glue, Kinesis, EMR
Azure Platform Skills:
Storage: ADLS Gen2, Blob Storage
Identity: Entra ID, Service Principals, Managed Identities
Networking: VNets, Private Link, NSGs
Data Services: Azure Data Factory, Synapse
Databricks Platform & Infrastructure:
- Workspace automation via Terraform
- Unity Catalog configuration
- Cluster policies and instance profiles
- Private Link, VNet Injection, IP Access Lists
- Workflows, DLT, Airflow integrations
- MLflow, Mosaic AI, Vector Search infrastructure
Certifications (Nice to Have)
General Cloud & DevOps:
- AWS Solutions Architect (Associate/Professional)
- Azure Solutions Architect Expert
- Terraform Associate
- Certified Kubernetes Administrator
Data & Platform:
- Databricks Data Engineer Professional
- AWS Data Engineer Associate
- Azure Data Engineer Associate (DP-203)
Desired Skills:
- Platform Architecture
- Security & Governance
- Automation
- Observability
- Enablement
Desired Qualification Level:
- Degree
About The Employer: