About the position
Job Description
Architects, engineers, and drives optimisation across enterprise-state of the art petabyte scale data pipelines and infrastructure foundational to the banks data driven strategy. This role is crucial in translating high volume, heterogeneous raw data into governed low latency and actionable data products for advanced analytics, machine learning and real time data insights that enable the delivery of accurate, high-quality, and timely data. Own the end-to-end data lifecycle from source ingestion, batch, streaming via various modalities ensuring data quality and semantic consistency and metadata. Partners with cross-functional teams to align technical solutions with business needs, strengthen the enterprise data architecture, and advancing the Bank’s strategic goal of becoming a data-driven organisation.
Accountabilities
- Data Pipeline Development and Engineering
- Engineer and maintain robust scalable and fault tolerant data pipelines using technologies like Ab Initio, Python/Scala, Apache spark, Microsoft Fabric, Databricks for ingestion, transformation and delivery of structured, semi structured and unstructured data.
- Ensure efficient performance tuned extraction and loading of data into the enterprise Data Warehouse and Data Lake focusing on high availability, cost efficiency and reliability.
- Implement modern Data Ops practices including automation, testing and orchestration for both batch and low latency streaming data workflows.
- Collaborate with Software Engineers to design and implement consumption APIs and services that enable secure real time data access for business applications
- Data Infrastructure and Platform Management
- Manage the robust, hybrid cloud data platform utilising Azure Services (Microsoft Fabric, Ab initio, synapse, Databricks, SAS) and potentially on-premises technologies (DB2 warehouse, Netezza, Denodo, Ab Initio). Demonstrate proficiency in infrastructure as code.
- Establish and manage comprehensive observability for all data infrastructure components such as databases, data lakes, and data warehouses to meet strict service level objectives and aligned with architectural standards.
- Collaborate with Information Security, CISO, and Data Governance to enforce data security, policies and implement masking/anonomysation techniques to ensure compliance with data privacy regulations (POPIA)
- Lead the technical evaluation, proof of concept and the successful integration of cutting-edge data engineering technologies and tools to enhance platform maturity and competitive advantage.
- Data Quality, Validation, and Governance
- Embed data quality as code by implementing automated, unit and end to end data validation, reconciliation, and auditing frameworks within the CI/CD data pipelines.
- Design and maintain the automated capture and maintenance of technical and operational metadata ensuring complete automated data lineage within the enterprise metadata hub (Ab Initio).
- Act as a technical guardian of the data integrity partnering with Data Governance to define and enforce master data management and data quality standards across all data products.
- Proactively manage data quality through regular data profiling, advanced issue analysis, and swift documented root cause resolution.
- Collaboration, Delivery, and Support
- Serve as trusted technical partner, collaborating with business stakeholders, Data Scientists, Analysts, and Architects to translate requirements into scalable data solutions.
- Drive the full data lifecycle of data products and end-to-end ownership of data engineering initiatives, ensuring timely delivery and alignment with business objectives.
- Provide Level 2/3 support for complex data issues, coordinating with cross-functional teams to resolve incidents, ensuring a minimal mean time to resolution and clear communication
- Contribute to sprint planning, backlog refinement, and agile delivery processes.
- Continuous Improvement and Professional Development
- Drive continuous optimisation of data engineering practices, standards, and processes to improve efficiency and performance.
- Mentor and guide junior engineers, providing coaching and technical support to build capability.
- Stay abreast of emerging technologies and industry trends to enhance data engineering maturity.
- Contribute to innovation initiatives that align with the bank’s data-driven strategy.
Desired Skills:
- Systems Analysis
- Complex Problem Solving
- Programming/configuration
- Critical Thinking
- Time Management
