Lead Software Engineer - Databricks​/AI

Salary: $61, per year

• Formal training or certification in software engineering, with at least 5 years of applied experience
• Expert Python and PySpark programming skills, supported by a portfolio of production-grade code
• Extensive hands‑on experience with Databricks and AWS services, including Glue, S3, SQS/SNS, and Lambda
• Deep expertise in Spark and SQL
• Hands‑on experience with Lakehouse and Delta Lake architecture, application development, testing, and operational stability; experience with Snowflake, Terraform, and LLMs; and knowledge of data observability, data quality, query optimization, and cost optimization
• In‑depth knowledge of enterprise‑scale big data and data warehousing
• Extensive experience with CI/CD processes and automated testing frameworks
• Understanding of agile methodologies, Dev Ops practices, application resiliency, and security measures
• Understanding of agentic AI concepts, including how autonomous agents plan, reason, use tools, and execute multi‑step workflows, and of the data infrastructure that supports them
• Experience building APIs, data services, and retrieval systems connecting AI agents with enterprise data
• Ability to lead through hands‑on engineering, mentor colleagues, and drive team delivery
• Preferred:
  Experience with agentic AI frameworks such as Lang Graph, Auto Gen, CrewAI, or OpenAI Assistants API
• Preferred:
  Familiarity with tool‑use and function‑calling patterns for LLM‑based agents, including building APIs and data endpoints for autonomous use
• Preferred:
  Experience with vector databases such as Pinecone, FAISS, or Chroma, and embedding workflows for agent memory, semantic search, and retrieval‑augmented generation
• Preferred:
  Exposure to agent memory and state‑management patterns, including short‑ and long‑term memory and conversation or task history
• Preferred:
  Familiarity with guardrails and safety frameworks, including input/output validation, action approvals, and human‑in‑the‑loop controls
• Preferred:
  Understanding of agentic‑system observability and monitoring, including tracing decisions, logging tool use, and debugging multi‑step workflows
• Preferred:
  Understanding of responsible AI principles, including autonomous decision‑making, data provenance, and auditability

• Build and optimize data pipelines and workflows that provide agentic AI systems with reliable, timely access to high‑quality structured and unstructured data
• Develop data retrieval and indexing layers so AI agents can search, query, and synthesize information across multiple sources
• Build and maintain APIs, data services, and function endpoints that agents use to execute tasks, retrieve data, and interact with enterprise systems
• Apply data management best practices and uphold quality, security, compliance, and governance for data used or generated by autonomous agents
• Write secure, high‑quality production code following AWS best practices, and deploy through CI/CD pipelines
• Build orchestration and state‑management layers for multi‑step agent workflows, including memory, context persistence, and task chaining
• Write and review code regularly, conduct thorough code reviews, and raise the teams technical standards
• Mentor and guide junior and mid‑level engineers through pairing, code reviews, and technical coaching
• Work with product owners, data scientists, and business stakeholders to turn business needs into production‑ready agentic AI solutions
• Evaluate and adopt emerging agentic AI frameworks, tools, and data engineering practices to improve the teams capabilities

• Agentic AI
• AI
• AI Agents
• API
• AWS
• Lambda
• Big Data
• CI/CD
• Databricks
• Dev Ops
• Support
• LLM
• Marketing
• Python
• Py Spark
• SQL
• Security
• Snowflake
• Spark
• Terraform
• Cloud
• AWS Glue
• Backbone
• RAG
• Windows

We are collaborating with hackajob to…