Lead Software Engineer - Python, Databricks, AWS

:

Category:
Software Engineering

Job Schedule:

Full time

Posted Date: T18:12:45+00:00

Job Shift: Day

:

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

Job responsibilities

* Execute creative, data-driven software solutions, including design, development, and technical troubleshooting, with the ability to think beyond routine approaches to solve technical problems.

* Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, and Databricks on AWS, processing and transforming large-scale financial datasets for analytics and reporting.

* Develop fact and dimension data models for reporting and analytics.

* Write secure, high-quality production code, and review and debug code written by others.

* Ensure data quality, consistency, security, and lineage throughout all stages of data processing and transformation, implementing monitoring and alerting mechanisms to maintain pipeline reliability.

* Support data migration and modernization initiatives, transitioning legacy systems to cloud-based data warehouses.

* Collaborate with business stakeholders to develop data management strategies, transforming data into insights that drive strategic decisions.

* Document data flows, logic, and transformation rules to maintain transparency and facilitate knowledge sharing.

* Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.

* Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

* Utilize AI tools to accelerate development and testing of data pipelines (e.g. Git Hub CoPilot, Claude Code).

Required qualifications, capabilities, and skills

* Formal training or certification on software engineering concepts and 5+ years applied experience

* Proven experience in data management, ETL/ELT pipeline development, and large-scale data processing.

* Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.

* Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).

* Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).

* Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.

* Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices

* Strong understanding of data quality, security, and lineage best practices.

* Experience with cloud-based data warehouse migration and modernization.

* Proficient in CI/CD, continuous delivery methods (Jules/Jenkins, Spinnaker, Sonar), the full Software Development Life Cycle, and Agile methodologies.

* Excellent problem-solving, troubleshooting, and analytical skills with ability to investigate data issues, identify root causes, and implement solutions.

Preferred qualifications, capabilities, and skills

* Knowledge of data pipeline tools such as PySpark, Snowflake, or Databricks.

* Experience with data orchestration tools (Airflow, Step Functions, etc.).

* Databricks or AWS certifications.

* In-depth knowledge of the financial services industry and their IT systems.