Senior Data Engineer: Python PySpark Databricks; Remote

Job Title :
Senior Data Engineer Python/PySpark & Databricks

Work Location:
Remote

Experience : 8

Years

Job Summary: We are looking for an experienced Backend/Data Developer with hands-on experience in Databricks, PySpark, Python/Java, Web Services, and SQL to support Dealer Services (Data Edge). The ideal candidate will have strong backend and data engineering skills, experience designing and troubleshooting data-driven solutions, and the ability to work closely with architects, technical leads, business analysts, and operations teams.

Essential Job Functions

• Understand the overall system architecture and support operations with monitoring, dashboards, and system health activities.
• Work closely with architects and technical leads on solution design and implementation.
• Continuously monitor and optimize system performance, scalability, and cost.
• Partner with Business Analysts to understand and implement integration requirements.
• Troubleshoot and resolve performance, backend, data, and functional issues.
• Analyze application and data-processing issues and identify root causes.
• Maintain strong attention to detail and document key technical decisions, solutions, and project artifacts.
• Support fast-paced projects and participate in troubleshooting and resolving production issues when required.
• Quickly learn and adopt new technologies, tools, and development methodologies.
• Communicate promptly and effectively with technical and business stakeholders.
• Participate in code reviews and provide technical guidance and mentoring to junior engineers.
• Collaborate with cross-functional teams to deliver reliable, scalable, and high-quality solutions.

Minimum Qualifications & Requirements

• Minimum 7 years of IT development experience.
• Strong experience developing data-driven backend solutions.
• Strong hands-on experience with:
• Python / Py Spark
• Databricks
• SQL / PL/SQL
• Java or C#
• Backend development and Web Services/API integration
• Strong SQL skills with experience working with complex queries and data processing.
• Experience designing technical solutions and participating in architecture/design discussions.
• Experience performing code reviews and mentoring junior engineers.
• Strong analytical, troubleshooting, and problem-solving skills.
• Ability to investigate performance issues and identify root causes.
• Strong understanding of software development lifecycle, coding standards, and best practices.
• Ability to work independently as well as effectively within a team.
• Strong verbal and written communication skills.
• Must be a collaborative team player with a customer-focused mindset.

Preferred Qualifications

Experience with one or more of the following is highly desirable:

• Microsoft Azure
• Apache Kafka
• Node.js
• Azure Data Factory
• Cloud-based data engineering and integration solutions
• Distributed data processing and large-scale data platforms
• Monitoring, logging, and performance optimization tools