About the position
Background:
- Wits Health Consortium (Pty) Limited ("WHC") is a wholly-owned Company of the University of the Witwatersrand, Johannesburg (South Africa) under its Faculty of Health Sciences
- WHC provides Faculty with a legal framework within which to operate the research and other activities necessary to support its academic objectives
- In addition, WHC offers a range of products and services to Academics
Division: Wits Planetary Health Research
Main purpose of the job:
- To design, implement, and manage a secure, scalable, and compliant Azure cloud environment underpinned by a Databricks Lakehouse Architecture
- Support federated data analysis approaches, enable real-time data processing, and facilitate collaboration across three countries while maintaining strict data governance and privacy protections
Location:
Key performance areas:
Cloud Architecture & Platform Engineering: -
- Scalable and secure Azure architecture designed, implemented, and documented
- Databricks workspaces configured and optimized across Development, Staging, and Production environments
- Secure network connectivity established and maintained
- Infrastructure as Code (IaC) implemented for repeatable and version-controlled deployments
- Azure subscriptions managed effectively with cost visibility and optimization
- Technical architecture documentation completed and continuously maintained
Data Engineering & Platform Enablement: -
- Scalable ETL/ELT frameworks implemented
- Medallion Architecture (Bronze, Silver, Gold) established using Delta Lake
- A data harmonisation framework developed to integrate health, climate, and socioeconomic datasets
- Streaming pipelines implemented for real-time ingestion (weather stations and early warning systems)
- Automated data quality checks and validation frameworks are operational
- Secure APIs and query endpoints developed for researcher access
- Analysis-ready datasets made available for epidemiological and climate research
Data Governance, Security & Compliance: -
- Unity Catalog implemented for centralized governance across all data assets
- Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC) established following least-privilege principles
- Encryption at rest and in transit implemented using Azure Key Vault
- Data masking and fine-grained access controls configured for sensitive health datasets
Federated Analytics & Cross-Institution Collaboration: -
- Existing federated analysis protocols (e.g., DataSHIELD) implemented
- Bespoke solutions developed for research-specific requirements
- Secure connectivity established between cloud platforms and partner institutions across all participating countries
- Distributed computation environments configured for cross-site analysis
- Federated analysis documentation developed for researcher enablement
- Ongoing technical support provided for federated analysis projects
DevOps, Automation & Release Management: -
- CI/CD pipelines implemented for automated infrastructure and code deployment
- Pipeline orchestration established for reliable data processing
- Databricks Asset Bundles implemented for version-controlled deployments
- Controlled promotion processes implemented across environments
- Manual deployment effort and deployment errors reduced
- Faster iteration cycles achieved for data pipeline development
Performance Optimization & Cost Management: -
- Spark jobs optimized to reduce processing time and resource consumption
- Quarterly cost monitoring and optimization reports produced
- Infrastructure resources right-sized based on workload requirements
- Databricks cluster policies implemented to prevent unnecessary spend
- Cost allocation and chargeback mechanisms established
- Annual cloud expenditure maintained within approved budget
Required minimum education and training:
- Bachelor’s Degree in Computer Science, Information Technology, Data Engineering, or related field
Required minimum work experience:
- Minimum of 5 years’ experience in cloud engineering, data engineering, or related roles, experience working with Azure tools and services
Desirable additional education, work experience and personal abilities:
- Experience working with health data, clinical research, or in regulated environments
- Familiarity with federated learning, privacy-preserving analytics, or trusted research environment (TRE) concepts
- Relevant certifications such as: Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Associate/Professional, HashiCorp Terraform Associate
- Experience with Delta Lake, Apache Spark, Databricks SQL, Azure Data Factory, Azure Machine Learning, MLflow, or Power BI
- Experience working across multiple countries or with distributed teams in Southern Africa
- Experience with Infrastructure as Code (Terraform, Bicep) for Azure resource provisioning and Databricks workspace deployment
- Foundational skills in Python, CLI tools, and Git
- Ability to work overtime and on weekends
- Regional travel may be required
TO APPLY:
- Only if you meet the minimum job requirements and experience as mentioned above, you may submit a detailed updated CV and a cover letter (maximum one page)
- Please Apply Online and complete your registration via our website to enable and protect you as a candidate, and to accept the new POPIA terms and conditions
- This will then create your permanent profile with which you can apply for all jobs as advertised by AJ PERSONNEL
- However, you may remove your profile from AJ PERSONNEL when you are no longer in the job market
- Please take note that the applicants who do not adhere to the above criteria will not be considered for the respective position
- Wits Health Consortium will only respond to shortlisted candidates
- Candidates who have not been contacted within two weeks of the closing date can consider their applications to be unsuccessful
- Closing date: 03 July 2026
- Note: No CV will be accepted after the closing date
Please note:
- WHC, in accordance with its Employment Equity goals and plan, will give preference to suitable applicants from designated groups as defined in the Employment Equity Act 55 of 1998 and subsequent amendments thereto
- AJ Personnel is only responsible for advertising and the response handling on behalf of their client Wits Health Consortium
- AJ Personnel does not have any salary or other information regarding the position
Desired Skills:
- Data Engineering
- Azure tools
- Clinical Research
- Databricks Certified
- Git
- Tshwane
