Senior Data Engineer – PySpark, Cloud & Kafka - YoE - r

Position: Senior Data Engineer – PySpark, Cloud & Kafka - 5+ YoE - Immediate Joiner - Any UST Location
Location: Bengaluru

Candidates ready to join immediately can share their details via email for quick processing.

CCTC | ECTC | Notice Period | Location Preference

Act fast for immediate attention! ⏳

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Py Spark
- Build and manage batch and real-time data processing systems
- Develop and integrate Kafka-based streaming solutions
- Optimize Spark jobs for performance, cost, and scalability
- Work with cloud-native services to deploy and manage data solutions
- Ensure data quality, reliability, and security across platforms
- Collaborate with data scientists, analysts, and application teams
- Participate in code reviews, design discussions, and production support

Must-Have Skills

- Strong hands-on experience with PySpark / Apache Spark
- Solid understanding of distributed data processing concepts

- Experience with Apache Kafka (producers, consumers, topics, partitions)
- Hands-on experience with any one cloud platform:
- AWS (S3, EMR, Glue, EC2, IAM) or
- Azure (ADLS, Synapse, Databricks) or
- GCP (GCS, Dataproc, Big Query)
- Proficiency in Python
- Strong experience with SQL and data modeling
- Experience working with large-scale datasets
- Familiarity with Linux/Unix environments
- Understanding of ETL/ELT frameworks

- Experience with CI/CD pipelines for data applications

Good-to-Have Skills

- Experience with Spark Structured Streaming
- Knowledge of Kafka Connect and Kafka Streams
- Exposure to Databricks

- Experience with No

SQL databases (Cassandra, Mongo

DB, HBase)
- Familiarity with workflow orchestration tools (Airflow, Oozie)
- Knowledge of containerization (Docker, Kubernetes)

- Experience with data lake architectures
- Understanding of security, governance, and compliance in cloud environments
- Exposure to Scala or Java is a plus
- Prior experience in Agile/Scrum environments