Lead AI Engineer

Lead AI Engineer

Location: Austin, Texas (Hybrid)

Duration: Longterm Contract

We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based Elastic Search system by integrating state‑of‑the‑art semantic search, dense retrieval, and LLM‑powered ranking techniques. This role will drive the transformation of traditional search into an intelligent, context‑aware, personalized, and high‑precision search experience.

The ideal candidate has hands‑on experience with Elastic Search internals, information retrieval (IR), embedding‑based search, BM25, re‑ranking, LLM‑based retrieval pipelines, and AWS cloud deployment.

• Analyze limitations in current regex & keyword‑only search implementation on Elastic Search.
• Enhance search relevance using:
  • BM25 tuning
  • Synonyms, analyzers, custom tokenizers
  • Boosting strategies and scoring optimization
• Introduce semantic / vector‑based search using dense embeddings.

• Implement LLM‑powered search workflows including:
  • Query rewriting and expansion
  • Embedding generation (OpenAI, Cohere, Sentence Transformers, etc.)
  • Hybrid retrieval (BM25 + vector search)
  • Re‑ranking using cross‑encoders or LLM evaluators
• Build RAG (Retrieval Augmented Generation) flows using Elastic Search vectors, Open Search, or AWS‑native tools.

• Build and optimize search APIs for latency, relevance, and throughput.
• Design scalable pipelines for:
  • Indexing structured and unstructured text
  • Maintaining embedding stores
  • Real‑time incremental updates
• Implement caching, failover, and search monitoring dashboards.

• Deploy and operate solutions on AWS, leveraging:
  • Open Search Service or EC2‑managed Elastic Search
  • Lambda, ECS/EKS, API Gateway, SQS/SNS
  • Sage Maker for embedding generation or re‑ranking models
• Implement CI/CD for search models and pipelines.

• Develop search evaluation metrics (nDCG, MRR, precision@k, recall).
• Conduct A/B experiments to measure improvements.
• Tune ranking functions and hybrid search scoring.
• Partner with product teams to refine search behaviors with real usage patterns.

• 5–10 years of experience in AI/ML, NLP, or IR systems, with hands‑on search engineering.
• Strong expertise in Elastic Search/Open Search: analyzers, mappings, scoring, BM25, aggregations, vectors.
• Experience with semantic search:
  • Embeddings (BERT, SBERT, Llama, GPT‑based, Cohere)
  • Vector databases or ES vector fields
  • Approximate nearest neighbor (ANN) techniques
• Working knowledge of LLM‑based retrieval and RAG architectures.
• Proficient in Python; familiarity with Java/Scala is a plus.
• Hands‑on AWS experience (Open Search, Sage Maker, Lambda, ECS/EKS, EC2, S3, IAM).
• Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker.
• Familiar with typical IR metrics and search evaluation frameworks.

• Knowledge of cross‑encoder and bi‑encoder architectures for re‑ranking.
• Experience with query understanding, spell correction, autocorrect, and autocomplete features.
• Exposure to LLMOps / MLOps in search use cases.
• Understanding of multi‑modal search (text + images) is a plus.
• Experience with knowledge graphs or metadata‑aware search.