Data Support Engineer

A Data Application Support role specializing in Databricks focuses on maintaining, troubleshooting, and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsibilities include resolving L2/L3 production incidents, performance tuning SQL/Python (PySpark) jobs, managing Delta Lake assets, and collaborating with data engineers to ensure data reliability.

• Incident & Problem Management: Provide L2/L3 support for data applications, resolving production issues and troubleshooting Databricks jobs, notebooks, and workflows.
• Performance Tuning: Optimize Spark applications, SQL queries, and Delta Lake tables to improve efficiency and reduce costs.
• Pipeline Maintenance: Monitor and troubleshoot ETL/ELT pipelines in Databricks (including Data Factory/Delta Live Tables), ensuring data quality and lineage (Unity Catalog).
• Collaboration: Act as a liaison between users, data engineering teams, and platform engineering, providing technical expertise and contributing to documentation.
• Automation: Create tools to automate routine support tasks and enhance support team productivity.

• Technical Expertise: Strong hands-on experience with Apache Spark, Python/PySpark, and SQL.
• Databricks Ecosystem: Proficiency with Databricks Unified Data Analytics Platform, Delta Lake, and ideally Azure Databricks.
• Cloud Data Storage: Experience with Azure Data Lake Storage (ADLS Gen2) or similar data lake technologies.
• Version Control & CI/CD: Experience with Git/Azure Dev Ops for code management.
• Problem-Solving: Strong analytical skills, with the ability to diagnose complex data processing bottlenecks.

• Data orchestration tools (e.g., Apache Airflow, Azure Data Factory).
• Data governance tools (e.g., Unity Catalog, Collibra).
• Streaming data knowledge (e.g., Spark Structured Streaming).