About the position
Role Overview
Our client is looking for experienced (mid-level Senior) Confluent Kafka Migration Engineers to support the migration of a large-scale, on-premises Confluent Platform environment to Confluent Cloud. The resource will be responsible for migration planning, cloud environment configuration, secure network connectivity, Kafka data replication, authentication modernisation, production cutovers, and post-migration stabilisation.
The role requires strong hands-on expertise in Apache Kafka, Confluent Platform, Confluent Cloud, and enterprise-scale production migrations, with a focus on data integrity, high availability, security, and minimal service interruption.
Roles and Responsibilities
- Assess the existing on-premises Kafka and Confluent Platform environment and validate the target cloud architecture.
- Develop detailed migration strategies, wave-based execution plans, cutover runbooks, rollback procedures, and risk mitigation plans.
- Provision and configure Confluent Cloud environments for development, pre-production, and production.
- Configure Kafka topics, partitions, replication, Schema Registry subjects, ACLs, service accounts, and Kafka Connect connectors.
- Implement and manage Confluent Cluster Linking for replication and migration of Kafka workloads from on-premises to cloud.
- Validate data consistency, schema compatibility, consumer offsets, application connectivity, and message processing before and after migration.
- Configure secure private connectivity between on-premises data centres and cloud environments using AWS PrivateLink, GCP Private Service Connect, VPC peering, or equivalent technologies.
- Modernize Kafka client authentication by migrating from Kerberos-based authentication to supported mechanisms such as API keys, OAuth, or mutual TLS (mTLS), as applicable to the target architecture.
- Collaborate with application teams to update Kafka client configurations, libraries, and authentication settings.
- Implement Infrastructure as Code (IaC) using Terraform and automate platform provisioning and configuration through CI/CD pipelines.
- Configure monitoring, alerting, audit logging, consumer lag tracking, and performance dashboards using enterprise observability tools.
- Troubleshoot Kafka replication, connectivity, throughput, partitioning, consumer lag, and production performance issues.
- Execute production cutovers in planned waves, validate workload functionality, and maintain rollback readiness.
- Support hypercare, operational handover, knowledge transfer, and decommissioning of legacy Kafka infrastructure after successful migration.
Required skills:
- Strong hands-on experience with Apache Kafka and Confluent Platform administration.
- Proven experience in enterprise-scale Kafka migrations from on-premises environments to cloud-managed platforms.
- Practical experience with Confluent Cloud architecture, configuration, administration, and operations.
- Hands-on experience with Confluent Cluster Linking and Kafka replication strategies.
- Strong understanding of Kafka topics, partitions, replication factors, consumer groups, offsets, and high-availability configurations.
- Experience with Confluent Schema Registry, schema compatibility, Kafka Connect, connectors, ACLs, and service accounts.
- Knowledge of Kafka security and authentication mechanisms, including Kerberos, SASL, TLS, API keys, OAuth, or mTLS.
- Experience with AWS or GCP cloud services, private networking, and secure connectivity.
- Hands-on experience with Terraform or other Infrastructure as Code tools and CI/CD automation.
- Experience with production cutovers, rollback planning, disaster recovery, monitoring, and incident troubleshooting.
- Strong understanding of data integrity, performance optimisation, security controls, and operational readiness.
- Ability to coordinate with application development, cloud infrastructure, security, networking, and operations teams.
Good-to-Have Skills
- Experience with migration of Confluent Platform to Confluent Cloud Enterprise clusters.
- Knowledge of Apache Flink architecture, stream-processing concepts, and Kafka–Flink integration.
- Exposure to multi-cloud or cross-region Kafka migration strategies.
- Experience with AWS PrivateLink, GCP Private Service Connect, VPC peering, VPN, or Direct Connect.
- Knowledge of ksqlDB, Tableflow, and Confluent Cloud governance capabilities.
- Experience with Prometheus, Grafana, Instana, or similar observability tools.
- Knowledge of Kubernetes, Red Hat OpenShift, GitOps, and cloud-native deployment practices.
- Experience with managed streaming services and enterprise event-driven architectures.
- Confluent Certified Developer or Administrator for Apache Kafka certification.
- AWS or GCP cloud certification and HashiCorp Terraform Associate certification.
Desired Skills:
- Systems Analysis
- Complex Problem Solving
- Programming/configuration
- Critical Thinking
- Time Management
