Big Data Engineer

Skill:
Big Data Engineer (Pyspark + SQL)
Exp: 5 to 8 years
Loc:
Pune & Chennai

JD
Lead end to end analysis of large scale multisource datasets using Big Data technologies driving insights for strategic and operational decision making
Design develop and optimize complex analytical queries using Starburst Trino Presto to enable fast federated access across enterprise data sources
Build manage and optimize analytics workflows in Databricks using SQL PySpark and Python for scalable data processing
Partner closely with Data Engineering Architecture and Platform teams to influence data models analytics architecture and performance optimization
Translate ambiguous business problems into structured analytical approaches and clearly communicate findings trends and recommendations to senior stakeholders
Define and enforce best practices for data quality validation reconciliation and governance within analytics solutions
Mentor junior and midlevel analysts providing technical guidance code reviews and analytical best practices
Support executive reporting dashboards and advanced adhoc analysis as required
Contribute to continuous improvement of analytics processes standards and reusable frameworks
710 years of experience as a Data Analyst Senior Data Analyst working in large scale enterprise data environments
Strong handson experience with Big Data ecosystems Hadoop Hive Spark distributed data platforms
Proven experience using Starburst Trino Presto for complex high performance analytical queries
Extensive experience with Databricks including Databricks SQL Spark based analytics and notebook development
Expert level SQL skills query optimization performance tuning complex joins
Strong proficiency in Python and or PySpark for data analysis and transformation
Solid understanding of data warehousing data modeling and analytics architecture concepts
Excellent analytical problem solving and stakeholder communication skills Experience working on cloud platforms such as AWS Azure or GCP
Exposure to BI and visualization tools Power BI Tableau or equivalent
Knowledge of enterprise data governance metadata management and data quality frameworks
Experience working in Agile Dev Ops delivery models Strong ownership and ability to lead quality outcomes end to end
Clear communication and stakeholder management
Mentoring mindset and collaboration across QA Dev and Dev Ops teams