Data Engineer II - Data Architect
-
IT/Tech
Job Description & How to Apply Below
Premise Health is Different on Purpose
Premise Health serves large organizations and their people with exceptional healthcare. The result? Better experiences, better health, and better value, all while helping organizations lower their healthcare costs.
Premise's mission is to help people get, stay, and be well. Come join us and see for yourself why amazing health starts with amazing healthcare. For more information, visit
The Data Engineer II will be responsible for building, and maintaining scalable data pipelines, managing our azure delta lake architecture, and supporting analytics initiatives across the organization. The ideal candidate will have a strong background in Azure Databricks, data engineering, and data architecture. The successful candidate will work closely with our platform architects and data science teams to ensure seamless integration of data pipelines and Lakehouse architecture.
This role should ensure coding standard and coding migration to higher environment and CI/CD.
This is a Full Time, hybrid,
Data Engineer II - Data Architect role located in Brentwood, TN.
What You'll Do
- Design and maintain conceptual, logical, and physical data models for operational, analytical, reporting, and data science use cases.
- Design, develop, and maintain data pipelines using Azure Databricks, Apache Spark, and Python.
- Collaborate with data engineering teams to integrate data from EPIC, Clarity, and Caboodle and other on-premise source systems to Azure.
- Ensure data quality, integrity, and security throughout the data pipeline.
- Automate unit test cases as part of Data Pipeline.
- Contribute to the development and maintenance of our Lakehouse architecture using Azure Databricks and Delta Lake.
- Ensure data governance and security in the Lakehouse environment.
- Able to execute POCs to evaluate new tool, concepts introduced by Azure/ Databricks.
- Troubleshoot data pipeline and Lakehouse architecture issues, identifying and resolving bottlenecks and errors.
- Optimize data pipeline and Lakehouse architecture performance, scalability, cost and reliability.
- Collaborate with data science teams to integrate data pipelines and Lakehouse architecture with data science workflows.
- Ensure seamless data exchange between data engineering and data science teams.
- Participate in peer review for coding standards.
- Subject matter expert for py Spark and Scala.
- Implement and maintain monitoring solutions for Databricks pipelines (Bronze, Silver, Gold layers) to ensure data quality, performance, and reliability.
- Develop dashboards and alerts for tracking DLT pipeline health, job execution status, and SLA adherence using Databricks metrics and Azure Monitor.
- Configure and manage logging frameworks across ADF pipelines, Databricks jobs, and Storage Account to enable end-to-end traceability.
- Set up proactive alerting mechanisms for failures, latency issues, and resource utilization using Azure Monitor, Log Analytics, and custom metrics.
- Ensure lineage and auditability by leveraging Unity Catalog for tracking data movement and access patterns across the Medallion architecture.
- Collaborate with Data Engineering teams to define observability standards and integrate them into CI/CD workflows.
- Perform root cause analysis (RCA) for pipeline failures and maintain operational documentation for recurring issues.
- Optimize cost and performance monitoring for Databricks clusters and ADF activities through usage analytics and reporting.
- Provide extended support during off-hours or weekends for critical deployments or troubleshooting production issues to ensure platform stability.
- Other duties as assigned.
What You'll Bring
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
- Certifications such as AWS Certified…
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction