Databricks Architect

Location:

Houston, TX (Hybrid)

Employment Type:

Full-Time

Experience Level: Senior / Principal (10+ Years)

Jade Business Services (JBS) is seeking an experienced, visionary Senior Databricks Architect / Data & AI Architect to spearhead enterprise-scale Data and AI transformations. In this role, you will define end‑to‑end Lakehouse architectures, set technical strategy, drive presales/RFP technical responses, and engage directly with client executive leadership (VP/C‑suite).

The ideal candidate blends deep, hands‑on architectural expertise across the modern Databricks ecosystem—including real‑time streaming, Lakeflow pipelines, Unity Catalog governance, and agentic AI architectures— with proven experience in cloud platforms (Azure, GCP), data warehousing (Big Query), and strategic consulting.

• Architect, design, and validate enterprise‑grade Databricks Lakehouse, Real‑Time Streaming, Data Engineering, and Generative/Agentic AI solutions.
• Design robust data pipelines leveraging Lakeflow Designer, Lakeflow Connect, and Lakeflow Declarative Pipelines.
• Establish unified enterprise governance, access controls, auditability, and data lineage across catalogs using Unity Catalog (RBAC/ABAC).
• Evaluate architectural trade‑offs across performance, scalability, security posture, and cloud compute/storage costs (Fin Ops).
• Define interoperability strategies across Delta Lake and Apache Iceberg open table formats.

• Lead the architecture stream for major RFP, RFI, and client proposal responses, delivering high‑impact technical narratives and architectural diagrams.
• Serve as the trusted technical advisor in high‑stakes discussions with client executives (CTOs, CDOs, VPs).
• Translate complex business problems into modular, cloud‑agnostic, and future‑proof Data & AI blueprints.
• Conduct architecture reviews, gap analyses, and proof‑of‑concept (PoC) implementations for strategic enterprise clients.

• Pipeline Orchestration & Ingestion:
  In‑depth hands‑on experience with Databricks Lakeflow, Lakeflow Declarative Pipelines, Lakeflow Designer, and Lakeflow Connect.
• Storage & Interoperability:
  Delta Lake, Delta Uni Form, and Apache Iceberg interoperability.
• Unified Governance:
  Advanced Unity Catalog implementation, fine‑grained access control (RBAC/ABAC), dynamic masking, row‑level filtering, and system tables/lineage.
• AI & Agentic Systems:
  Databricks Vector Search / AI Search, Unity AI Gateway, Databricks Genie / Genie Code, Managed Agent Memory, and Agent Bricks.
• Core Compute & Frameworks:
  PySpark, Apache Spark Structured Streaming, MLflow, Retrieval‑Augmented Generation (RAG) frameworks, and agentic AI orchestration.
• Cloud Platforms:
  Strong architectural experience across Microsoft Azure (ADLS Gen2, Azure Synapse, AKS) and Google Cloud Platform (GCP) (GCS, Dataproc, Big Query).
• Cross‑Platform Analytics:
  Hands‑on experience integrating Databricks with Google Big Query, cross‑cloud querying, and data virtualization.
• Dev Ops & CI/CD:
  Terraform for Databricks/Cloud provisioning, Databricks Asset Bundles (DABs), and automated CI/CD pipeline deployments.

• Experience:
  
  10+ years in Data Engineering, Data Platform Architecture, Cloud Architecture, or Enterprise Solution Architecture.
• Databricks Expertise: 5+ years of specialized Databricks architecture experience, demonstrating mastery of the latest platform features and architecture patterns.
• Presales & Advisory:
  Proven track record in RFP/RFI authoring, presales solution design, client‑facing consulting, and executive‑level presentations.
• Education:
  
  Bachelor’s or Master’s degree in Computer Science, Information Systems, Software Engineering, or related field (or equivalent professional experience).

• Databricks Certified Solutions Architect (Professional)
• Microsoft Certified:
  Azure Solutions Architect Expert / GCP Professional Data Engineer