Senior Data Engineer _ Python Spark

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Spark.
• Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
• Work closely with data analysts, data scientists, and business stakeholders to understand data requirements.
• Develop efficient and reusable data processing frameworks and components.
• Perform data validation, cleansing, and transformation to ensure quality and consistency.
• Optimize and tune Spark jobs and SQL queries for performance and scalability.
• Collaborate with Dev Ops and platform teams to deploy and manage data solutions in cloud environments.
• Ensure data security, governance, and compliance standards are met.
• Troubleshoot production issues, perform root cause analysis, and implement fixes.
• Participate in design discussions, contribute to data architecture decisions, and promote best practices.
• Work in an Agile environment, supporting sprint activities and continuous improvement initiatives.

• Strong hands-on experience in Python programming for data engineering and data processing.
• Extensive experience with Apache Spark (PySpark) for large-scale data processing and distributed computing.
• Strong knowledge of SQL and experience working with relational and non-relational databases.
• Experience in building and maintaining ETL/ELT pipelines for data ingestion and transformation.
• Good understanding of data warehousing concepts, data modeling, and data architecture.
• Experience working with big data technologies such as Hadoop ecosystem, Hive, or similar platforms.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and related data services.
• Hands-on experience with data pipeline orchestration tools such as Airflow or similar.
• Knowledge of data formats such as Parquet, Avro, JSON, and CSV.
• Experience with performance tuning and optimization of Spark jobs and data pipelines.
• Strong problem-solving skills and ability to work with cross-functional teams.
• Bachelor’s degree in Computer Science or related field.