Lead AI Engineer; Search Modernization

Lead AI Engineer (Search Modernization)

Location: Austin, TX (3 days work from office)

Client: Tavant / Move Realtor

During the discovery stage, it will be 5 days working from office for the first 4 weeks of discovery

Mandatory

Skills:

Elastic Search,Open Search,Python,LLM,GenAI,Semantic Search,Re-Ranking,AWS,Search Engineer

Job Description:

We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based Elastic Search system by integrating state-of-the-art semantic search, dense retrieval, and LLM-powered ranking techniques.

This role will drive the transformation of traditional search into an intelligent, context-aware, personalized, and high-precision search experience
.

The ideal candidate has hands-on experience with Elastic Search internals
, information retrieval (IR),
embedding-based search
, BM25
, re-ranking
, LLM-based retrieval pipelines
, and AWS cloud deployment
.

• Analyze limitations in current regex & keyword-only search implementation on Elastic Search.
• Enhance search relevance using:
• BM25 tuning
• Synonyms, analyzers, custom tokenizers
• Boosting strategies and scoring optimization
• Introduce semantic / vector-based search using dense embeddings.

• Implement LLM-powered search workflows including:
• Query rewriting and expansion
• Embedding generation (OpenAI, Cohere, Sentence Transformers, etc.)
• Hybrid retrieval (BM25 vector search)
• Re-ranking using cross-encoders or LLM evaluators
• Build RAG (Retrieval Augmented Generation) flows using Elastic Search vectors, Open Search, or AWS-native tools.

• Build and optimize search APIs for latency, relevance, and throughput.
• Design scalable pipelines for:
• Indexing structured and unstructured text
• Maintaining embedding stores
• Real-time incremental updates
• Implement caching, failover, and search monitoring dashboards.

• Deploy and operate solutions on AWS
  , leveraging:
• Open Search Service or EC2-managed Elastic Search
• Lambda, ECS/EKS, API Gateway, SQS/SNS
• Sage Maker for embedding generation or re-ranking models
• Implement CI/CD for search models and pipelines.

• Develop search evaluation metrics (nDCG, MRR, precision@k, recall).
• Conduct A/B experiments to measure improvements.
• Tune ranking functions and hybrid search scoring.
• Partner with product teams to refine search behaviors with real usage patterns.

• 5–10 years of experience in AI/ML, NLP, or IR systems
  , with hands-on search engineering.
• Strong expertise in Elastic Search/Open Search
  : analyzers, mappings, scoring, BM25, aggregations, vectors.
• Experience with semantic search
  :
• Embeddings (BERT, SBERT, Llama, GPT-based, Cohere)
• Vector databases or ES vector fields
• Approximate nearest neighbor (ANN) techniques
• Working knowledge of LLM-based retrieval and RAG architectures
  .
• Proficient in Python
  ; familiarity with Java/Scala is a plus.
• Hands-on AWS experience (Open Search, Sage Maker, Lambda, ECS/EKS, EC2, S3, IAM).
• Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker
  .
• Familiar with typical IR metrics and search evaluation frameworks.

• Knowledge of cross-encoder and bi-encoder architectures for re-ranking.
• Experience with query understanding
  , spell correction, autocorrect, and autocomplete features.
• Exposure to LLMOps / MLOps in search use cases.
• Understanding of multi-modal search (text images) is a plus.
• Experience with knowledge graphs or metadata-aware search.