Lead Data Platform Engineer; Palantir or Databricks

Project description

Project description

We are seeking an expert with deep proficiency as a Palantir Platform Engineer, possessing experience in data engineering and designing, building, and operationalizing AI-powered workflows, agents, and applications that drive tangible business outcomes. The ideal candidate is a self-starter, able to translate complex business needs into scalable technical solutions, and confident working directly with stakeholders to maximize the value of Foundry and AIP.

• Manage and optimize Palantir data platform.
• Ensure high availability, security, and performance of data systems.
• Provide valuable insights about data platforms usage.
• Optimize computing and storage for large‑scale data processing.
• Design and maintain system libraries (Python) used in ETL pipelines and platform governance.
• Optimize ETL Processes – Enhance and tune existing ETL processes for better performance, scalability, and reliability.
• AIP & AI Enablement
  • Support the design and deployment of AIP use cases such as copilots, retrieval workflows, and decision‑support agents.
  • Ground agents and logic flows using RAG (retrieval‑augmented generation) by connecting to relevant data sources, embedding/vector search, ontology content.
  • Use Ontology‑Augmented Generation (OAG) when needed: operational decision‑making where logic, data, actions and relationships are embedded in the Ontology.
  • Collaborate with senior engineers on agent design, instructions, and evaluation using AIP’s native features.

• Minimum 10 Years of experience in IT/Data.
• Minimum 5 years of experience as a Data Platform Engineer/Data Engineer.
• Minimum 3 years of experience with Palantir Foundry.
• Practical experience using or supporting AIP features such as RAG workflows, copilots, or agent‑based applications.
• Bachelor’s in IT or related field.
• Infrastructure & Cloud:
  Azure, AWS (expertise in storage, networking, compute).
• Proficiency in PySpark for distributed computing.
• Proficiency in Python for ETL development.
• SQL:
  Expertise in writing and optimizing SQL queries, preferably with experience in databases such as Postgre
  
  SQL, MySQL, Oracle, or Snowflake.
• ETL Tools:
  Familiarity with ETL tools & processes.
• Data Modelling:
  Experience with dimensional modelling, normalization/denormalization, and schema design.
• Version Control:
  Proficiency with version control tools like Git to manage codebases and collaborate on development.
• Data Pipeline Monitoring:
  Familiarity with monitoring tools (e.g., Prometheus, Grafana, or custom monitoring scripts) to track pipeline performance.
• Data Quality Tools:
  Experience implementing data validation, cleaning, and quality frameworks, ideally Monte Carlo.

• Containerization & Orchestration:
  Docker, Kubernetes.
• Infrastructure as Code (IaC):
  Terraform.
• Understanding of Investment Data domain (desired).