Senior Data Engineer

US citizens, Green Card Holders, and those authorized to work in the US are encouraged to apply. We are unable to sponsor H1b candidates at this time

Job Title: Senior Data Engineer

Location: Charlotte, NC or Hartford, CT, Short Hills, NJ (Hybrid role, 2 days in office)

Employment Type: Full-Time

Preferred Industry: Financial Services

Experience

Required:

5+ Years

We are seeking a highly skilled Senior Data Engineer with strong expertise in Azure, Databricks, PySpark, Python, and SQL to design, develop, and optimize modern data platforms and pipelines. This role requires more than just moving data from one system to another—the ideal candidate must be able to understand business requirements, analyze complex datasets, and build scalable solutions that generate actionable insights.

The successful candidate will have deep experience building enterprise-grade data ingestion and transformation pipelines, leveraging Databricks and Azure technologies, while demonstrating a strong understanding of data architecture, analytics, and business problem-solving.

• Design, develop, and maintain scalable data pipelines using PySpark, Python, Databricks, and Azure
  .
• Build robust data ingestion, transformation, and processing frameworks for large-scale datasets.
• Develop and optimize ETL/ELT workflows within the Databricks ecosystem.
• Utilize advanced Databricks capabilities, including:
• Unity Catalog
• Databricks Workflows
• Photon
• Cluster optimization and performance tuning
• Analyze source data and understand the underlying business context and data relationships.
• Collaborate with business stakeholders to translate business requirements into scalable data solutions.
• Perform data modeling, data quality validation, and performance optimization.
• Create data solutions that support reporting, analytics, and business intelligence initiatives.
• Work closely with cross-functional teams including Data Architects, Analysts, and Business users.
• Troubleshoot complex data issues and recommend best practices for data engineering processes.

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with:
• Databricks
• Py Spark
• Python
• SQL
• Extensive experience building PySpark-based data pipelines.
• Strong understanding of data ingestion, transformation, and orchestration frameworks.
• Advanced SQL skills, including data analysis, query optimization, and complex transformations.
• Experience working with large-scale structured and unstructured datasets.
• Ability to interpret data, identify patterns, and derive meaningful business insights.
• Strong problem-solving and analytical skills.
• Experience working in cloud-based data environments.

• Experience with in the Financial Services industry.
• Experience supporting enterprise analytics and reporting initiatives.
• Knowledge of data warehousing and modern lakehouse architectures.
• Experience collaborating directly with business stakeholders to solve complex business problems through data.

• Understand the story behind the data.
• Connect technical solutions to business outcomes.
• Analyze datasets and generate meaningful insights.
• Design end-to-end data solutions that solve real business challenges.
• Leverage the full capabilities of Databricks and Azure to build scalable, high-performing data platforms.

Required Skills: Azure, Databricks, PySpark, Python, SQL, Data Engineering, ETL/ELT, Delta Lake, Unity Catalog, DLT, Data Analytics, Data Warehousing, Cloud Data Platforms.

A reasonable, good-faith estimate of the minimum and maximum for this position is $130K/year to $140K/year with benefits