Big Data Specialist
Job in
New York, New York County, New York, 10261, USA
Listed on 2026-07-01
Listing for:
Hyrhub
Full Time
position Listed on 2026-07-01
Job specializations:
-
IT/Tech
Data Engineering, Data Warehousing, Cloud Computing: Infrastructure & Operations
Job Description & How to Apply Below
We are seeking a highly skilled and experienced Databricks SME to lead the architecture, design, and implementation of enterprise‑scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have at least 12 years of hands‑on experience architecting big data solutions with Databricks, with proven expertise in migrating data and pipelines from legacy platforms like Alteryx, handling streaming data, and implementing CI/CD practices for data pipelines.
This role is critical for driving data modernization initiatives, ensuring robust data governance, and enabling scalable data ingestion and transformation across multiple source systems into the Databricks ecosystem.
Key Responsibilities- Architect enterprise Big Data solutions on Databricks using Bronze/Silver/Gold (Medallion) layers with Delta Lake and Delta Live Tables
- Migrate data and pipelines from legacy platforms (Alteryx, Informatica, Data Stage, Talend etc.) to Databricks, including source‑to‑target mapping and data validation
- Design streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables
- Implement CI/CD practices for data pipelines using Git, Azure Dev Ops, and Git Hub Actions
- Develop data governance framework using Unity Catalog ABAC/RBAC policies and collaborate with stakeholders to translate requirements into scalable solutions
- Experience with Databricks Workflows, Jobs, and Databricks Asset Bundles for deployment automation
- Mentor junior team members and provide technical leadership across data engineering projects
- 12+ years architecting Big Data solutions with Databricks Lakehouse Platform
- Proven experience migrating pipelines from Alteryx or similar legacy ETL tools
- Hands‑on expertise with Delta Live Tables, Delta Lake, and Medallion architecture
- Strong experience building streaming pipelines with Structured Streaming and Auto Loader
- Demonstrated ability implementing CI/CD for data workloads
- Unity Catalog Data governance framework with ABAC / RBAC policies
- Proficient in Python, SQL, PySpark, and strong experience in performance tuning of large‑scale data
- Proven experience on at least one major cloud platform — Azure (preferred) or AWS — and its native data services
- Unity Catalog for data governance and lineage
- Data quality tools:
Great Expectations, Informatica DQ, Collibra - Data modeling tools:
Erwin, dbt, ER/Studio - AI/ML experience with GenAI, LLMs, MLOps
- Databricks Certified Data Engineer Associate/Professional
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×