Senior Data Engineer (Databricks)

About the position

Purpose:


Develop and maintain our data storage platforms and specialised data pipelines to support the company's Technology Operations.


 


Operational Delivery:

• Development and maintenance of LakeHouse environments.


• Development of data engineering and data analysis pipelines and solutions in the cloud (Azure).


• Ensure DevOps compliance for all data pipelines and workloads.


• Ensure development and maintain PowerBI reports.


• Documentation of pipelines and databases.


• Troubleshooting and support to our Technical Operations team when dealing with production issues.


• Contribute to application and systems design in collaboration with the Software Architects.


• Provide mentorship and support to junior data engineers.

Qualifications:


B.Sc (Electronic and/or Computer Engineering) or similar qualification.


 


Experience:

• Minimum of 5 years of experience working within a data engineering environment.

Competencies:

• Strong SQL skills.


• Databricks experience (minimum 1 year), preferably with certifications.


• Python and PySpark experience.


• DataLake experience with practical experience developing and maintaining a LakeHouse architecture.


• Azure Data Factory (ADF) experience or similar.


• Experience in writing technical documentation (architectural diagrams, release notes, etc.).


• Good inter-personal and communication skills.

Additional Nice-to-have Skills:

• DataWarehouse experience – especially Azure Synapse


• Datascience and AI/ML experience would be beneficial


• NoSQL database experience (e.g. CosmosDB, CouchBase, MongoBD, etc.)

Desired Skills:

• Systems Analysis
• Complex Problem Solving
• Programming/configuration
• Critical Thinking
• Time Management