About the position
We are looking for a skilled Python / Spark / AI Developer to join a growing technology team.
You will be responsible for developing scalable data pipelines, replatforming legacy SQL workloads onto modern Spark and lakehouse technologies, and supporting the development of APIs and AI-driven features.
The ideal candidate is a strong Python developer with hands-on production experience in Apache Spark/PySpark, SQL, Delta Lake and REST APIs.
Key Responsibilities
- Build Spark/PySpark data pipelines using Delta Lake.
- Replatform legacy T-SQL logic into Spark SQL.
- Develop modern, type-safe and well-tested Python applications.
- Build REST APIs using FastAPI or equivalent frameworks.
- Implement authentication, idempotency and job-status functionality.
- Validate migrated pipelines against legacy systems and provide parity evidence.
- Work from technical specifications, design documents and ADRs.
- Develop automated tests using pytest.
- Contribute to CI/CD pipelines, code reviews and technical documentation.
Minimum Requirements
- 4+ years of professional Python development experience.
- 2+ years of production experience with Apache Spark / PySpark.
- Strong knowledge of Spark SQL and DataFrame APIs.
- Production experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
- Strong SQL skills and experience working with legacy T-SQL.
- Experience with pytest and automated testing.
- Experience developing REST APIs, preferably with FastAPI.
- Experience with Docker and containerised development environments.
- Strong Git and CI/CD experience.
- Experience with type-hinted Python and static analysis tools such as Pyright or Mypy.
- Experience with modern Python tooling such as uv or Poetry.
- Ability to work independently from written technical requirements and design documentation.
- Bachelor’s degree in Computer Science, Engineering or a related field, or equivalent experience.
- Proven track record of delivering production Python/Spark pipelines independently.
Advantageous Skills
- Experience with LLM / AI integration.
- Experience with Ollama, vLLM, [URL Removed] LM Studio or hosted AI APIs.
- Knowledge of data governance, privacy engineering and POPIA/GDPR principles.
- Experience with Hive Metastore, Trino or Apache Ranger.
- Azure experience, including ADLS Gen2, Synapse, ADF, Key Vault or Service Bus.
- Experience with structured logging, OpenTelemetry or OpenLineage.
- Data migration and pipeline parity testing experience.
- Databricks Certified Developer for Apache Spark or equivalent certification.
How To Apply:
Contact Hire Resolve today for your next career-changing move
Our client is offering a highly competitive salary for this role based on experience.
Send your CV to: [Email Address Removed].com or connect with Mischa Bornman via LinkedIn.
Alternatively, you can also contact me directly at Hire Resolve [Email Address Removed]
We will contact you telephonically in 3 days should you be suitable for this vacancy. If you are not suitable, we will put your CV on file and contact you regarding any future vacancies that arise.
Desired Skills:
- Python / Spark / AI Developer
- Python / Spark / AI Developer
- Python / Spark / AI Developer
