Data Platform Engineer

Title:

Data Platform Engineer

Location:

San Jose, CA 95110 (Hybrid) Duration: 06 Months (possible extension) Job Description

• Setup and maintain production-scale Databricks environments on public cloud platforms such as Microsoft Azure and AWS
• Handle production-scale data storage platforms like ADLS and S3 for multiple tenant teams within our Data Platform
• Coordinate production-scale microservices that support the daily operations of our data platform, including job scheduling, security, financial, and administrative services
• Maintain enterprise-scale SQL Server and SSAS environments
• Provide triage and mentorship to the team on various support issues raised by our tenants
• Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting
• Ensure security and privacy compliance, implementing Adobe Security & Compliance solutions to secure data in our data lake
• Explore GenAI technologies to integrate technical improvements and improve user experience
• Collaborate with various 3rd party vendors to resolve issues, conduct proof of concepts, and improve our product

• AWS
• Azure

• ADLS Gen2
• S3
• MongoDB
• Vector Databases
• AI Agent Development and Maintenance:
• Langchain

• Databricks
• SQL Server & SSAS
• Airflow
• Azure Kubernetes Service or Elastic Kubernetes Service

• DBX System Tables
• Prometheus
• Splunk
• MLFlow

• Linux Servers
• Kubernetes Environment

• Jira
• Service Now