ENVIRONMENT:
OUR client, a reputable Foundation with a mission to strengthen the entrepreneurial ecosystem, has an exciting 6-month renewable contract project-based role for an experienced Senior Data Engineer. The project entails building an enterprise data platform to transform AGGP_SA’s entrepreneurship ecosystem through automated data capture and real-time analytics, consolidating data from 5 disparate systems. The ideal candidate must have 5-8 years Data Engineering experience including utilsing Python/Java/Scala, Apache Spark, Kafka, or similar streaming technologies, AWS & preferably Azure, API integration and RESTful services, SQL and NoSQL databases. Any prior experience in the Non-profit/Social Impact Sector will prove beneficial.
DUTIES:
- Data Pipeline Development (40%): Build real-time ETL pipelines for Asana, SharePoint, AcceleratorApp, Fluxx, and Jibble integration.
- Data Architecture (30%): Design and implement data lake/warehouse architecture supporting 300,000+ beneficiaries.
- API Integration (20%): Develop robust API connections with error handling and retry logic.
- Performance Optimization (10%): Ensure <5-minute data synchronization and support 100,000 records/hour processing.
Deliverables –
- Fully functional data pipeline integrating all 5 systems.
- Data lake architecture documentation.
- Automated data quality monitoring system.
- Performance benchmarks achieved (<5 min sync).
REQUIREMENTS:
- 5-8 Years Data Engineering experience.
- Python/Java/Scala for Data Engineering (5+ Years).
- Apache Spark, Kafka, or similar streaming technologies.
- Cloud platforms (Azure preferred, AWS acceptable).
- API integration and RESTful services.
- SQL and NoSQL databases.
- Data lake/warehouse architecture (Delta Lake, Databricks).
- ETL/ELT pipeline development.
- Real-time data processing.
- Previous non-profit/social impact sector experience beneficial.
- Proven track record of multi-system integrations.
- Experience with data volumes of 1M+ records.
Desirable Skills –
- Microsoft Azure Data Factory.
- Experience with CRM data models.
- TolaData or similar impact measurement platforms.
- SharePoint API/Graph API experience.
- Predictive Analytics pipeline setup.
While we would really like to respond to every application, should you not be contacted for this position within 10 working days please consider your application unsuccessful.
COMMENTS:
When applying for jobs, ensure that you have the minimum job requirements. OnlySA Citizens will be considered for this role. If you are not in the mentioned location of any of the jobs, please note your relocation plans in all applications for jobs and correspondence. Apply here [URL Removed] OR e-mail a Word copy of your CV to [Email Address Removed] and mention the reference number of the job.
Desired Skills:
- Senior
- Data
- Engineer