Senior Data Engineer

Senior Data Engineer
*** Sponsorship is not provided**
* *** No Third Party Resumes
*** Onsite in Peachtree Corners, GA
- Hybrid Salary
- Competitive Position Summary We are seeking an experienced, highly motivated Senior Data Engineer to help design, develop, and optimize our enterprise data infrastructure. This role requires a strong technical background in data engineering, cloud-based technologies, and modern data architecture, with a particular focus on Microsoft Azure, Azure Databricks, Azure Data Factory, and PySpark. The ideal candidate is a hands-on technical leader who can translate business requirements into scalable data solutions, establish best practices, and drive continuous improvement across our data environment.

This position offers the opportunity to take ownership of key projects, influence technical strategy, and collaborate with cross-functional teams. While there are no direct reports, the Senior Data Engineer will provide technical guidance, mentor junior team members, and lead project initiatives.

Key Responsibilities Data Engineering & Architecture Design, build, and maintain scalable batch and streaming data pipelines using Azure Databricks, Azure Data Factory, and related technologies. Develop and optimize data ingestion, transformation, and integration processes (ETL/ELT). Architect and support enterprise-wide data platforms, data lakes, and data warehousing solutions. Integrate Azure Databricks with Azure Data Lake Storage (ADLS), Power BI, Azure Dev Ops, and Azure Key Vault.

Develop efficient data models and ensure data quality, reliability, and accessibility. Optimize Apache Spark jobs, cluster configurations, serverless policies, and cloud infrastructure to improve performance and manage costs. Cloud Infrastructure, Security & Automation Implement and manage Unity Catalog to support data governance, access control, lineage, and security compliance. Build and maintain CI/CD pipelines for Databricks notebooks and workflows using Azure Dev Ops and Terraform.

Establish and enforce technical standards, development practices, and cloud architecture guidelines. Identify opportunities to improve automation, scalability, and overall platform efficiency. Technical Leadership & Collaboration Partner with business stakeholders, analysts, and technical teams to understand current and future data requirements. Help define and execute the organization's long-term data strategy. Provide technical direction and recommendations for new tools, platforms, and technologies.

Lead data engineering projects from planning through implementation. Mentor junior data engineers and promote knowledge sharing across the team. Collaborate with data engineers, analysts, and other technical professionals to support AI and machine learning initiatives. Evaluate emerging technologies and recommend solutions that enhance data capabilities. Assist with project planning, resource allocation, and assessment of future staffing needs.

Required Qualifications Education & Experience Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, Statistics, or a related technical field. 4+ years of hands-on experience in data engineering, including working with large datasets and database systems. 2+ years of experience working with cloud-based data technologies, preferably Microsoft Azure. Experience designing and supporting enterprise-scale data platforms and architectures. Strong background in ETL/ELT development, data transformation, integration, and modeling.

Technical Skills Strong hands-on experience with Azure Databricks and Azure Data Factory. Proficiency in Python, PySpark, and SQL.

Experience with Apache Spark, Spark SQL, Delta Lake, and Databricks Jobs.

Experience with Azure Data Lake Storage (ADLS), Azure Key Vault, and Power BI. Knowledge of Azure Dev Ops and CI/CD development practices.

Experience with data warehousing, data lake architecture, and modern cloud data platforms. Understanding of cloud storage, compute resources, data orchestration, and reporting technologies. Experience optimizing data pipelines and processing large-scale datasets.

Preferred Qualifications Experience implementing Unity Catalog for data governance and security. Experience using Terraform for infrastructure automation. Knowledge of serverless computing, Spark performance tuning, and cloud cost optimization. Experience supporting AI/ML data workloads. Experience establishing enterprise data engineering standards…