About the position
Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands-on role, combining the building and optimization of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them.
The role functions as a key interface between business stakeholders and department heads, the wider data and analytics team, and the owners of source systems and data in Finance and IT.
RequirementsKEY RESPONSIBILITIES:
- Design and build robust, production-grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP).
- Work “under the hood” to optimise the platform for performance and cost tuning compute, jobs and queries,managing storage and table layout, and keeping platform spend under control.
- Define and maintain consistent business metrics and a semantic layer (for example, using Metric Views) so that reporting is built on trusted, reusable definitions.
- Integrate data from core business systems, including SAP S/4HANA, and other sources, using tools such as Fivetran (both SaaS connectors and HVR).
- Set and uphold engineering standards across the team, including code quality, testing, documentation, CI/CD and data governance.
- Coach and mentor junior and mid-level engineers, review their work and help them develop.
- Partner with business stakeholders to understand their needs, shape practical solutions and ensure the platform delivers genuine value.
- Engage data owners in Finance and IT to ensure data is well understood, valid and fit for purpose, and initiate data quality improvements where required.
REQUIRED SKILLS AND EXPERIENCE:
- At least three years of hands-on, day-to-day Databricks experience, covering both pipeline development and under-the-hood performance and cost optimisation.
- Up to date with recent platform developments, such as Spark Declarative Pipelines (SDP) and Metric Views.
- Strong proficiency in programming languages commonly used in data engineering, such as Python, SQL and Spark.
- Advanced experience with data manipulation, data modelling, database design and query optimisation.
- Experience managing or coaching junior developers.
- A track record of managing and influencing business stakeholders.
KEY COMPETENCIES:
- Combining deep, hands-on engineering skill with sound judgement about cost, performance and long-term maintainability.
- Coaching, mentoring and raising the capability of less experienced engineers.
- Collaborating, communicating confidently and influencing business stakeholders.
- Breaking down complex technical concepts and explaining them simply to non-technical audiences.
- Staying current with a fast-moving platform and bringing new capabilities into everyday practice.
- Taking ownership and driving work independently, from concept through to production.
NICE TO HAVE:
- Experience working with SAP S/4HANA data sets.
- Experience with Fivetran, including both SaaS connectors and HVR.
- Power BI semantic modelling experience.
QUALIFICATIONS:
- A relevant tertiary qualification in Computer Science, Information Systems, Engineering or a related field, or equivalent practical experience.
Desired Skills:
- Databricks
- PySpark
- SQL
- Delta Lake
- Data Pipelines.
Desired Qualification Level:
About The Employer:
