Databricks Architect; AWS | Spark | PySpark

Databricks Architect (AWS | Spark | PySpark)

Our client, a IT Services and Consulting company, is looking for a Databricks Architect for their Bridgewater, NJ location. Responsibilities:

• This Architect role focuses on designing and governing scalable data platforms using Databricks Spark PySpark and AWS Cloud for a global organization in a hybrid work model.
• The Architect will own end to end data solution architecture oversee Databricks administration and optimize cloud resources to enable reliable analytics and machine learning outcomes that support critical business decisions and create societal value through data driven innovation.
• Architect robust and scalable Databricks based data platforms that enable secure high performance data processing and analytics for business critical workloads in a hybrid work environment.
• Design end to end data solutions leveraging Spark and PySpark that integrate batch and streaming data pipelines to deliver reliable curated data sets for downstream consumption across diverse teams.
• Oversee Databricks workspace configuration including cluster policies access controls and environment standards to ensure consistent governance compliance and operational efficiency. Implement AWS Cloud architectures for data and analytics including storage compute and networking components to optimize cost performance reliability and alignment with enterprise standards.
• Administer Databricks environments by monitoring job performance capacity utilization and error trends to proactively tune clusters and improve overall platform stability and throughput. Provide technical guidance on PySpark coding standards reusable frameworks and data transformation patterns to enhance maintainability reduce defects and accelerate delivery of data products.
• Coordinate with data engineers data scientists and product teams to translate business requirements into data architecture blueprints that maximize value creation and support strategic initiatives.
• Establish and enforce best practices for data security privacy and access management across Databricks and AWS including role based controls encryption and audit monitoring.
• Evaluate and introduce optimization techniques for Spark and PySpark such as efficient partitioning caching and data format choices to reduce processing time and resource consumption. Collaborate with cloud operations teams to define backup recovery and high availability strategies for Databricks and AWS data components ensuring resilience and business continuity. Drive continuous improvement of data platform capabilities by assessing new Databricks and AWS features piloting enhancements and advocating adoption that benefits organizational goals and societal impact.
• Create clear technical documentation including solution designs runbooks and standards so delivery and operations teams can implement and support data platforms with confidence and consistency.
• Mentor peers and junior practitioners by sharing domain expertise on Databricks Spark PySpark and AWS Cloud helping build internal capability and fostering a culture of quality and innovation.

Requirements:

• 10+ years in Data Engineering, Data Architecture, or Cloud Data Platforms
• 5+ years of hands-on Databricks architecture and administration
• Strong experience designing enterprise-scale data platforms on AWS
• Pharmaceutical, Biotechnology, Healthcare, or other highly regulated enterprise environments
• Experience supporting enterprise analytics, AI/ML, and large-scale data modernization initiatives
• Experience:
  
  12to16
  
  Yrs
• Demonstrate extensive experience in Databricks administration including workspace setup cluster management governance configuration and troubleshooting in complex enterprise contexts. Bring strong practical proficiency in Spark and PySpark for building scalable data pipelines transformations and performance tuned jobs applied to large and varied data sets.
• Possess deep hands on experience with AWS Cloud services relevant to data platforms such as data storage compute analytics and security features to design integrated solutions.
• Show capability in AWS administration including identity and access configuration monitoring resource…