Lead Data Engineer

Experience: 6–8 years
Location: Bengaluru
Work Mode: Hybrid
Employment Type: Full-Time

Role Overview

The role involves designing, developing and maintaining high-performance, production-grade real-time data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse and Apache Airflow.

The candidate will work across the complete data pipeline, from source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards and downstream integrations.

Key Responsibilities

Real-Time Data Engineering

• Design and maintain real-time data pipelines using Apache Flink and Kafka.

• Build high-volume, low-latency streaming applications.

• Implement data transformation, filtering, enrichment, aggregation and event processing.

• Develop reliable event-processing pipelines with error handling and recovery.

• Manage Kafka event consumption and publishing.

• Work with partitions, consumer groups, offsets and delivery mechanisms.

• Troubleshoot latency, throughput, failures and performance issues.

Apache Flink

• Develop and maintain production-grade Flink jobs.

• Implement transformations, joins, aggregations and windowing.

• Work with event time, watermarks and state management.

• Implement checkpoints, savepoints and recovery mechanisms.

• Optimize Flink jobs for scalability and resource utilization.

• Monitor backpressure, latency, throughput and resource consumption.

Apache Kafka

• Develop Kafka-based ingestion and streaming pipelines.

• Create and manage Kafka topics and event streams.

• Work with partitions, offsets, consumer groups, replication and retention.

• Build reliable producer and consumer applications.

• Handle ordering, retries, duplicate events and replay scenarios.

• Troubleshoot consumer lag and throughput issues.

• Work with Kafka schemas and serialization formats.

CDC & Debezium

• Build CDC pipelines using Debezium.

• Configure and maintain Debezium connectors.

• Capture source-system inserts, updates and deletes.

• Publish CDC events into Kafka.

• Manage initial snapshots and incremental CDC.

• Handle schema evolution and source-system changes.

• Implement reconciliation and consistency checks.

• Troubleshoot CDC and source-to-target data issues.

Data Orchestration

• Develop workflows using Apache Airflow or equivalent tools.

• Build reusable DAGs for ingestion, CDC, validation, Flink execution, transformation, ClickHouse loading and downstream integrations.

• Implement dependencies, scheduling, retries, backfills, SLAs and alerting.

• Integrate Airflow with Kafka, Flink, Debezium, ClickHouse, APIs and cloud services.

• Monitor workflow execution and troubleshoot failures.

ClickHouse & Analytical Data

• Integrate streaming pipelines with ClickHouse.

• Design analytical data models.

• Develop and optimize SQL queries.

• Implement partitioning, sorting, indexing and retention strategies.

• Optimize data ingestion and query performance.

• Support dashboards and analytical use cases.

Data Quality & Reliability

• Implement data validation and quality checks.

• Build source-to-target reconciliation mechanisms.

• Monitor data freshness, completeness, accuracy and consistency.

• Implement error handling, retry, replay and recovery mechanisms.

• Establish logging and observability.

• Support incident investigation and root-cause analysis.

Integration & APIs

• Integrate streaming and analytical data with APIs, dashboards and downstream applications.

• Develop data interfaces and integration components.

• Work with application teams on data contracts and integration requirements.

Engineering Practices

• Follow Git and version-control practices.

• Participate in code reviews.

• Develop unit and integration tests.

• Support CI/CD, logging, monitoring and documentation.

• Build reusable and maintainable data engineering components.

• Participate in technical design and architecture discussions.

• Mentor Data Engineers and contribute to engineering standards.

Required Skills

• 6–8 years of Data Engineering experience.

• Apache Flink — mandatory/core requirement.

• Strong Apache Kafka experience.

• Hands-on Debezium and CDC experience.

• Strong Java or Scala programming skills.

• Python experience is an advantage.

• Strong SQL skills.

• Experience with analytical databases; ClickHouse highly preferred.

• Apache Airflow or another data orchestration framework.

• Strong understanding of distributed systems and real-time data processing.

• Experience supporting production-grade streaming pipelines.

• Strong Kafka knowledge covering topics, partitions, offsets, consumer groups, replication and retention.

• Experience with transformation, enrichment, filtering, aggregation and event processing.

• Strong troubleshooting and performance optimization skills.

• Familiarity with cloud platforms and containerized environments.

Preferred Skills

• Advanced Flink: state management, checkpoints, savepoints, watermarks, event time, windows, backpressure and state backends.

• Airflow, Dagster, Prefect or Apache NiFi.

• Kafka Schema Registry.

• Avro, Protobuf or JSON.

• Kubernetes.

• AWS, Azure or GCP.

• CI/CD.

• Docker.

• Terraform or other Infrastructure-as-Code tools.

• Data observability and monitoring tools.

• High-volume, low-latency data platforms.

• REST APIs and system integrations.

Technical Stack

Apache Flink | Apache Kafka | Debezium | CDC | ClickHouse | Apache Airflow | Java/Scala | Python | SQL | Kubernetes | Docker | Cloud | CI/CD | REST APIs

How to Apply

Share your updated CV at Khushboo@Sourcebae.com or WhatsApp 8827565832.