Azure Databricks Engineer

Plateau Software Inc

Plateau GRP is seeking a highly skilled Azure Databricks Engineer to support a large-scale data modernization initiative for the Federal Deposit Insurance Corporation (FDIC). The selected candidate will design, develop, and optimize enterprise‑grade data lakehouse solutions using Azure Databricks, Apache Spark, and related Azure data services.

This position requires deep technical expertise in data engineering, advanced analytics enablement, and large‑scale data pipeline optimization. The ideal candidate will collaborate with cross‑functional teams to deliver secure, scalable, and high‑performing data infrastructure within a modernized cloud environment.

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark to process large volumes of structured and unstructured data.
• Integrate data from multiple sources using Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics.

• Architect and optimize data lakehouse solutions leveraging Delta Lake format and partitioning strategies.
• Implement data modeling best practices for efficient data storage, retrieval, and analytical performance.

• Configure and manage Databricks work spaces, clusters, and compute resources.
• Implement security controls, access management, and cost optimization strategies aligned with FDIC standards.

• Develop streaming data solutions using Structured Streaming and implement batch processing jobs for transformation, aggregation, and integration.
• Ensure performance, scalability, and reliability of both real‑time and batch data workflows.

• Collaborate with data scientists to deploy machine learning models and integrate them into data pipelines.
• Implement MLflow for model lifecycle management and automate ML pipelines for training and inference.

• Monitor pipeline performance and system health across Databricks and Azure environments.
• Troubleshoot job failures, implement data quality checks, and optimize Spark jobs for improved efficiency and reduced processing cost.

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
• Over 5 years of experience as a Data Engineer, with 3+ years specifically in Azure Databricks and related Azure data services.
• U.S. Citizenship and the ability to pass/obtain any required government background check or security clearance. An active security clearance is preferred.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala.
• Hands‑on experience designing, developing, and optimizing large‑scale data pipelines in cloud environments.
• Technical proficiency with Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics.
• ETL/ELT processes, data modeling, and big data architectures.
• Strong understanding of data governance, data security, and cloud operations in enterprise environments.
• Experience working within Agile delivery frameworks.

• Microsoft Certified:
  Azure Data Engineer Associate (DP‑203) or Databricks Certified Data Engineer certification.
• Experience supporting federal data modernization or enterprise cloud transformation programs.
• Familiarity with Dev Ops/CI‑CD pipelines, Git Hub Enterprise, and Infrastructure as Code (Terraform).
• Experience supporting AI/ML initiatives and integrating data science workflows within Azure Databricks.

Plateau is an equal opportunity/affirmative action employer; all qualified applicants will receive consideration for employment without regard to age, gender, gender identification, sex, sexual orientation, color, race, creed, national origin, religion, marital status, parental status, citizenship status, ancestry, physical or mental disability, genetic information, veteran status, military status, or any other classification protected by federal, state, or local laws.

Plateau is committed to providing access and reasonable accommodations for individuals with disabilities. If you require reasonable accommodation, please email the Plateau HR Department at