Lead AI Engineer

Job Description:

We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based Elastic Search system by integrating state-of-the-art semantic search, dense retrieval, and LLM-powered ranking techniques.

This role will drive the transformation of traditional search into an intelligent, context-aware, personalized, and high-precision search experience
.

The ideal candidate has hands-on experience with Elastic Search internals
, information retrieval (IR),
embedding-based search
, BM25
, re‑ranking
, LLM‑based retrieval pipelines
, and AWS cloud deployment
.

• Analyze limitations in current regex & keyword-only search implementation on Elastic Search.

• Enhance search relevance using:

• BM25 tuning

• Synonyms, analyzers, custom tokenizers

• Boosting strategies and scoring optimization

• Introduce semantic / vector-based search using dense embeddings.

• Implement LLM-powered search workflows including:

• Query rewriting and expansion

• Embedding generation (OpenAI, Cohere, Sentence Transformers, etc.)

• Hybrid retrieval (BM25 vector search)

• Re‑ranking using cross‑encoders or LLM evaluators

• Build RAG (Retrieval Augmented Generation) flows using Elastic Search vectors, Open Search, or AWS‑native tools.

• Build and optimize search APIs for latency, relevance, and throughput.

• Design scalable pipelines for:

• Indexing structured and unstructured text

• Maintaining embedding stores

• Real‑time incremental updates

• Implement caching, failover, and search monitoring dashboards.

• Deploy and operate solutions on AWS
  , leveraging:

• Open Search Service or EC2‑managed Elastic Search

• Lambda, ECS/EKS, API Gateway, SQS/SNS

• Sage Maker for embedding generation or re‑ranking models

• Implement CI/CD for search models and pipelines.

• Develop search evaluation metrics (nDCG, MRR, precision@k, recall).
• Conduct A/B experiments to measure improvements.
• Tune ranking functions and hybrid search scoring.
• Partner with product teams to refine search behaviors with real usage patterns.

• 5 10 years of experience in AI/ML, NLP, or IR systems
  , with hands‑on search engineering.

• Strong expertise in Elastic Search/Open Search
  : analyzers, mappings, scoring, BM25, aggregations, vectors.

• Experience with
  semantic search
  :

• Embeddings (BERT, SBERT, Llama, GPT‑based, Cohere)

• Vector databases or ES vector fields

• Approximate nearest neighbor (ANN) techniques

• Working knowledge of LLM‑based retrieval and RAG architectures
  .

• Proficient in Python
  ; familiarity with Java/Scala is a plus.

• Hands‑on AWS experience (Open Search, Sage Maker, Lambda, ECS/EKS, EC2, S3, IAM).

• Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker
  .

• Familiar with typical IR metrics and search evaluation frameworks.

• Knowledge of cross‑encoder and bi‑encoder architectures for re‑ranking.
• Experience with query understanding
  , spell correction, autocorrect, and autocomplete features.
• Exposure to LLMOps / MLOps in search use cases.
• Understanding of multi‑modal search (text images) is a plus.
• Experience with knowledge graphs or metadata‑aware search.