About the position
ENVIRONMENT:
WORK on cutting-edge Data Engineering projects of a US-based growing premium Software Product and Consulting Services company seeking a Senior Data Engineer with strong Java or Scala skills. You will be focused on executing critical enterprise data retention, migration, and privacy enhancement projects. This role combines robust Software Engineering capabilities with technical leadership skills to deliver production-quality solutions at petabyte scale. The successful candidate must have a Masters/Bachelors Degree in Computer Science/Engineering or related field with 10 years Data Engineering/Software Engineering experience with 4-7 years technical leadership. You will also need advanced proficiency in Scala or Java programming language with modern build systems (Gradle or Maven) & proven expertise in large-scale data migration and ETL pipeline development.
DUTIES:
Data Engineering & Migration Leadership –
- Design and implement large-scale data migration strategies for identity transformation across multi-petabyte datasets.
- Build comprehensive data lineage mapping and validation systems to ensure migration accuracy and data integrity.
- Develop scalable data transformation solutions supporting both batch and streaming processing with parallel pipeline orchestration.
- Implement advanced monitoring and observability solutions for data pipeline performance and system health tracking.
- Create testing and validation frameworks to ensure data accuracy and consistency during complex migrations.
Technical Leadership & Engineering Excellence –
- Provide technical leadership for data migration initiatives, taking responsibility for critical system components and project phases.
- Mentor Engineers through pair programming, code reviews, and collaborative data pipeline development.
- Lead technical design discussions for complex data migrations and create solutions meeting enterprise requirements.
- Demonstrate deep expertise in Data Engineering lifecycle management, from migration planning through deployment and operational support.
- Drive technical work forward by identifying and removing blockers, enabling continuous delivery of value.
- Collaborate effectively with stakeholders across disciplines, ensuring alignment and successful project outcomes.
REQUIREMENTS:
Qualifications –
- Masters or Bachelors Degree or equivalent experience in Computer Science, Engineering, or related field.
Experience/Skills -
- 10 Years Data Engineering/Software Engineering experience with 4-7 years technical leadership.
- Proven expertise in large-scale data migration and ETL pipeline development.
- Advanced proficiency in Scala or Java programming language with modern build systems (Gradle or Maven).
- Expertise in distributed file systems (HDFS) or cloud object storage (S3, GCS).
- Experience with cloud data platforms (AWS, GCP, Azure, or Databricks).
- Hands-on experience with AWS EKS (Kubernetes) for containerized data processing environments.
- Understanding of data privacy, security, and compliance requirements (GDPR experience preferred).
- Strong knowledge of data structures, algorithms, and distributed systems design principles.
Preferred to haves –
- Experience with Scala or Python for data processing applications.
- Experience with Apache Spark or PySpark for large-scale data processing and analytics.
- Experience with Hadoop ecosystem including HDFS, MapReduce, and related tools.
- Experience working in Enterprise Engineering environments with large-scale tools and development processes.
- Experience with data lineage analysis, dependency mapping, and data validation frameworks.
- Identity Management experience in large-scale distributed systems.
- Experience with modern data formats like Apache Iceberg, Parquet, ORC, or Avro.
- Proficiency in streaming platforms like Apache Kafka or AWS Kinesis.
- Hands-on experience with observability tools (Prometheus, Grafana, Spark UI).
Desired Skills:
- AWS
- Data Engineering
- ETL (Extract
- Transform and Load)
- Java
- Scala
About The Employer:
A US-based growing premium Software Product and Consulting Services company
