Databricks Data Architect

architecture end to end: how data is modeled, ingested, transformed, stored, governed, and served to every consumer

Skill Requirements Expert: SQL, Python, PySpark, data modeling, Databricks, Delta Lake, Snowflake, Kafka, Spark Structured

Job Summary We are hiring a Principal Data Architect — a hands-on, senior individual contributor who will design, build, govern, and evolve the enterprise data platform that serves as the single source of truth for the organization. You will own the architecture end to end: how data is modeled, ingested, transformed, stored, governed, and served to every consumer that depends on it — from executive reporting and operational analytics to business applications and, increasingly, ML and AI systems.

This is a data architecture role first. The large majority of your time goes to data modeling, platform design, pipeline engineering, data quality, and governance. A smaller but growing part of it goes to making that foundation AI-ready —so that ML models and AI applications consume governed, documented, trustworthy data instead of building their own shadow pipelines.

Key Responsibilities
15+ years of hands-on data engineering and data architecture experience, with a track record of owning enterprise-scale platforms end to end.
• Deep data modeling expertise — dimensional, normalized, and Data Vault — with real judgment about grain, conformance, and when each pattern applies.
• Proven experience architecting lakehouse and/or data mesh platforms:
Databricks, Delta Lake, PySpark, Snowflake, Kafka, Spark Structured Streaming, and cloud-native data services (AWS, Azure).Cisco Confidential Cisco Confidential
• Strong command of SQL and performance engineering: query optimization, partitioning and clustering, and cost management at scale.
• Demonstrated ownership of data governance, data quality, master data, metadata, and lineage programs — not just the tooling, but the operating model.
• Experience with data security and compliance in regulated industries (financial services, payments, cybersecurity, healthcare).
• Experience migrating legacy on-prem warehouses and ETL estates to cloud-native platforms.
• Working familiarity with the data requirements of ML and AI systems — feature pipelines, training data curation, and retrieval/vector data layers — enough to architect for them credibly and partner well with AI teams

Skill Requirements Expert: SQL, Python, PySpark, data modeling, Databricks, Delta Lake, Snowflake, Kafka, Spark Structured Streaming, AWS (S3, Glue, Redshift, Kinesis, EKS, Lambda), Airflow, Terraform, Git-based CI/CD.
• Strong:
Azure data services, dbt, data quality frameworks (Great Expectations or equivalent), catalog and lineage platforms (Unity Catalog, Collibra, Alation, Purview), MDM tooling, BI semantic layers (Power BI, Tableau, Looker), Docker, Kubernetes, FastAPI, observability tooling (Grafana, Cloud Watch).
• Familiar: MLflow, feature stores, vector databases (Open Search, Pinecone, FAISS, ChromaDB), knowledge graphs (Neo4j), LLM APIs and RAG frameworks (Lang Chain, Llama Index, AWS Bedrock, Claude, OpenAI)
Other Requirements BS / MS in Computer Science, Information Systems, Data Engineering, or a related quantitative field.
• Prior experience at a global financial institution (payments, risk, AML, compliance) or large enterprise SaaS, operating under strict regulatory oversight.
• Experience in presales solutioning for large data programs: RFP/RFI responses, SOW shaping, effort estimation, and CXO-level solutioning.
• Certifications such as Databricks Data Engineer Professional, AWS Certified Data Engineer or Solutions Architect, or Snowflake Snow Pro Advanced Architect.
• Familiarity with emerging standards for AI and agent data access (e.g., MCP — Model Context Protocol) and agent entitlement models