Data Engineer/ Scientist (1595) KG

About the position

WHAT WILL BE YOUR ROLE AND RESPONSIBILITIES?

• Design, develop, and maintain data ingestion pipelines for company connected fleet, focusing on Call Detail Records (CDRs).


• Utilise AWS Glue for ETL processes to transform and load data into the analytics platform.


• Implement efficient querying solutions using AWS Athena.


• Develop serverless applications and workflows using AWS Lambda.


• Monitor and optimise data pipelines using AWS CloudWatch.


• Manage and maintain PostgreSQL RDS databases for data storage and retrieval.


• Create interactive dashboards and reports using AWS QuickSight.


• Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.


• Use Git and GitHub for version control, collaboration, and maintaining code repositories.


• Implement and manage GitHub CI/CD pipelines to automate testing, deployment, and delivery processes.


• Ensure data security and compliance with company information classification requirements.


• Stay updated with the latest data engineering tools, technologies, and industry trends.


• Identify opportunities for process improvements and automation to enhance pipeline efficiency.


• Collaborate with cross-functional teams to understand data requirements and deliver solutions.


• Develop and maintain technical documentation for data engineering processes and solutions.

Minimum Requirements:

WHICH QUALIFICATIONS/EXPERIENCE DO WE NEED FOR THE ROLE?

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.


• Minimum of 3 years’ experience as a Data Engineer.


• At least 2 years of experience working with AWS services.


• Proven experience in building and maintaining data pipelines for large-scale datasets.


• Agile working experience

 



ESSENTIAL SKILLS REQUIREMENTS:

• Proficiency in Python 3.x for data processing and automation.


• Experience with AWS Glue for ETL processes.


• Strong knowledge of AWS Athena for querying large datasets.


• Hands-on experience with AWS Lambda for serverless computing.


• Familiarity with AWS EC2 for scalable computing resources.


• Expertise in AWS CloudWatch for monitoring and logging.


• Proficiency in working with PostgreSQL RDS for database management.


• Experience with AWS QuickSight for data visualisation and reporting.


• Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).


• Proficiency in Git and GitHub for version control and collaboration, including experience with GitHub CI/CD pipelines.


• Any additional responsibilities assigned in the Agile Working Model (AWM) Charter

 



ADVANTAGEOUS SKILLS REQUIREMENTS:

• Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimising AWS resources.


• Experience with other AWS services such as S3, Kinesis, and DynamoDB.


• Knowledge of data formats such as Parquet, AVRO, JSON, and CSV.


• Experience with Docker for containerisation.


• Understanding of Big Data technologies and frameworks.


• Familiarity with Agile working models and tools like JIRA and Confluence.


• Experience with data quality tools such as Great Expectations.


• Knowledge of REST API development and integration.


• Strong analytical skills for troubleshooting and optimising data pipelines.


• Experience in developing technical documentation and artefacts.

 

Desired Skills:

• Python
• AWS Glue
• AWS Athena
• AWS Lambda
• EC2
• AWS CloudWatch