Azure Data Engineer

Responsibilities

• Analyze, manipulate, and process large sets of structured and semi-structured data including healthcare datasets (claims, patient, and provider data) using Python (PySpark), SQL, and Spark SQL within the Azure data platform.
• Apply data mining, data modeling, and statistical analysis techniques to extract and analyze information from large datasets stored in Azure Data Lake Storage (ADLS) and Delta Lake across Bronze, Silver, and Gold layers.
• Identify business problems and management objectives that can be addressed through data analysis; translate business requirements into analytical workflows and data solutions using Azure Data Factory and Azure Databricks.
• Develop and maintain data models and analytical datasets to support reporting, business intelligence, and downstream data consumption; identify relationships, trends, and factors that could affect the results of analysis.
• Apply feature selection and data profiling methods to identify patterns, anomalies, and data quality issues that may impact analytical outcomes; recommend data-driven improvements.
• Clean, manipulate, and prepare raw data for analysis using ETL/ELT processes; implement data quality frameworks including validation rules, cleansing, and reconciliation techniques to ensure accuracy and consistency.
• Write new functions and applications in Python (PySpark) and SQL to conduct data transformation, validation, and analysis of large-scale datasets; optimize code for performance and efficiency.
• Test, validate, and reformulate data pipelines and datasets to ensure accurate and reliable data delivery; monitor, troubleshoot, and resolve data and performance issues using Azure monitoring tools.
• Analyze data patterns and recommend data-driven solutions to key stakeholders; deliver oral and written presentations of data analysis findings to management and end users to support informed decision-making.
• Support data governance, security, and compliance practices aligned with HIPAA requirements; prepare documentation for data processes and analytical workflows to support governance and audits.
• Develop and manage workflow orchestration, scheduling, and automation for consistent and timely data delivery using Azure-based tools.
• Implement version control and deployment processes using Azure Dev Ops and Git to support reproducible and reliable data workflows.
• Integrate data from multiple sources including APIs, databases, and flat files into a unified, analytics-ready data platform.
• Read technical articles, research publications, and conference papers to identify emerging analytic trends and technologies; continuously evaluate new tools and methodologies to improve data processing, analysis, and platform efficiency.
• Collaborate with business stakeholders, analysts, and data scientists to gather requirements, advise on appropriate analytical techniques, and recommend data-driven solutions aligned with organizational objectives.