About the position
Our client in the motor industry is looking to employ a Senior AWS Data Engineer. As a Big Data Engineer, you will be responsible for building and maintaining robust Big Data Pipelines using our client's Data Platforms.
RESPONSIBILITIES
- You will serve as a custodian of data, ensuring that it is shared securely and in compliance with information classification requirements on a need-to-know basis.
- Designing, implementing, and maintaining Big Data Pipelines using BMW Data Platforms.
- Collaborating with cross-functional teams to understand data requirements and develop scalable solutions.
- Ensuring data accuracy and integrity through thorough testing and data validation processes.
- Developing technical documentation and artifacts to support data engineering processes.
- Utilizing your expertise in Terraform, Python 3x, SQL (Oracle/PostgreSQL), PySpark, ETL, Docker, Linux/Unix, and other relevant technologies to optimize data workflows.
- Leveraging AWS services such as S3, RDS, DynamoDB, Glue, and Data Pipeline for building and managing data pipelines.
- Implementing software design patterns and best practices to ensure scalability, reliability, and maintainability of data solutions.
- Providing support and guidance in data modelling, SQL optimization, and performance tuning.
SKILLS
- Proficiency in Terraform, Python 3x, SQL (Oracle/PostgreSQL), PySpark, Boto3, ETL, Docker, Linux/Unix, and Big Data technologies.
- Experience with PowerShell/Bash scripting for automation tasks.
- Strong understanding of AWS Cloud services with relevant certifications such as AWS Certified Cloud Practitioner, AWS Certified SysOps Associate, AWS Certified Developer Associate, AWS Certified Architect Associate, or AWS Certified Architect Professional.
- Familiarity with HashiCorp Certified Terraform Associate certification is a plus.
- Ability to develop technical documentation and artifacts.
- Knowledge of data formats including Parquet, AVRO, JSON, XML, and CSV.
- Experience with Data Quality Tools like Great Expectations.
- Proficiency in working with REST APIs is desirable.
- Familiarity with AWS Glue, Data Pipeline, and other similar platforms for building data pipelines.
- Solid understanding of software design patterns and best practices.
- Exceptional analytical skills for analyzing large and complex datasets.
QUALIFICATIONS
- Bachelor's degree in IT, Business, Engineering, or related field.
- Preferred certifications: AWS Certified Cloud Practitioner, AWS Certified SysOps Associate, AWS Certified Developer Associate, AWS Certified Architect Associate, AWS Certified Architect Professional, or HashiCorp Certified Terraform Associate.
ADVANTAGEOUS
- Experience in data modelling and Oracle SQL.
- Hands-on experience with AWS components and services.
- Ability to prepare specifications for program development, testing, and debugging.
- Experience in building data pipelines using AWS Glue or Data Pipeline.
- Knowledge of data storage technologies such as AWS S3, RDS, or DynamoDB.
- Strong understanding of various software design patterns and best practices.
Only apply should you meet the above requirements.
Desired Skills:
- AWS
- SQL
- Docker
- REST APIs
- Big Data
- ETL
- Python
