Big Data Practice Lead – Spark​/Databricks​/AWS

Job Overview

A global consulting and technology services organization is seeking a Big Data / Spark Principal Architect to lead the design and implementation of large-scale data systems. This senior-level role combines deep technical expertise in Spark, Databricks, and cloud-based data platforms with a strategic understanding of solutioning and consulting across enterprise environments. The successful candidate will architect high-performance data pipelines, guide technical teams, and act as a subject matter expert (SME) for big data and streaming solutions within the Media & Entertainment domain.

• Architecture & Design:
  Define and implement scalable, high-performance big data architectures using Spark (Data Frame, Spark SQL, PySpark/Scala Spark) and Databricks.
• Solutioning & Consulting:
  Lead solution design engagements, act as SME for big data technologies, and provide consulting expertise for enterprise data strategies.
• Pipeline Development:
  Build and optimize batch and streaming data pipelines leveraging Spark, AWS Glue, and Kafka (or equivalent streaming tools).
• Data Governance & Quality:
  Establish frameworks for data quality, lineage, and compliance across multi-source data ecosystems.
• Cloud Enablement:
  Architect and deploy solutions on AWS or Azure, ensuring performance, scalability, and cost efficiency.
• Practice Development:
  Contribute to the growth of the Big Data practice through mentoring, pre-sales support, and best practice documentation.
• Collaboration:
  
  Work closely with data engineers, analysts, and business stakeholders to translate requirements into robust technical solutions.
• Continuous Improvement:
  Stay current with emerging technologies and promote adoption of modern data engineering practices.

• 10–18 years of total experience in Big Data Architecture, Engineering, and Consulting.
• Strong hands-on expertise in Apache Spark (Data Frame, SQL, PySpark/Scala Spark) and Databricks.
• Proven track record in solutioning and consulting as a Subject Matter Expert (SME) in Big Data.
• Experience with AWS Glue or Azure Data Services for data ingestion and transformation.
• Familiarity with streaming technologies such as Kafka, Kinesis, or Flink.
• Deep understanding of ETL design, data governance, and data quality frameworks.
• Strong coding skills and willingness to be hands-on with development.
• Prior experience in Media & Entertainment or related domains.
• Must be a U.S. Citizen or Green Card Holder (mandatory).
• Demonstrated job stability and long-term project ownership.

• Exposure to AI/ML integration within big data ecosystems.
• Experience leading practice development or solution selling for enterprise clients.
• Knowledge of modern data mesh or lakehouse architectures.

Location: USA – Remote (Full-Time)

Work Type:
Permanent Remote

Experience

Required:

10 – 18 Years

Compensation: USD 150,000 – 200,000 (plus bonus, based on experience and performance)