Senior Manager of Software Engineering - Databricks, AWS

This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.

As a Manager of Software Engineering at JPMorgan

Chase within Corporate Sector, Enterprise Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

*
* Job Responsibilities:

*
* +  Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).

+  Establish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.

+  
** Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.*
* +  
** Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.*
* +  Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, unit scripts, cluster policies, pools, and instance profiles.

+  Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.

+  Design secure data ingestion and transformation frameworks leveraging Databricks services:
Design delta or unmanaged tables, Create tasks for data, ingestion process, Create DAGs using Airflow to orchestrate creation of trusted and refined data.

+  Enforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.

+  Drive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.

+  Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.

+  Implement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.

** Required qualifications, capabilities, and skills:*
* +  
** Formal training or certification on software engineering concepts and 5+ years applied experience.*
* +  10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.

+  
** Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.*
* +  
** Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices*
* +  Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.

+  
** Hands-on Databricks expertise**  (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).

+  Proven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.

+  Deep skills in Spark performance tuning and Databricks cluster setup/optimization.

+  Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).

+  CI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).

+  Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.

** Preferred qualifications, capabilities, and skills:*
* +  

Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.

+  AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.

+  

Experience with Terraform for Infra deployments

+  Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.

+  Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).

+  Drive…