VLM Engineer Information Technology Herndon, VA

This role is anticipated to end at the end of July, with an expectation of mod/additional funding through 2027.

• Design and execute fine-tuning pipelines for Vision-Language Models (VLMs) on domain-specific imagery datasets, including data preprocessing, training orchestration, and hyperparameter optimization
• Develop and implement evaluation frameworks for multimodal model performance, including task‑specific metrics for image understanding, visual question answering, and spatial reasoning
• Build scalable training infrastructure on AWS (Sage Maker, EC2 GPU instances) for distributed fine‑tuning of large multimodal models
• Engineer data pipelines for curating, annotating, and transforming geospatial imagery datasets into model‑ready formats for supervised and instruction‑tuning workflows
• Collaborate with applied scientists and solutions architects to iterate on model architectures, adapter strategies (LoRA/QLoRA), and inference optimization techniques

• TS/SCI CI Poly Clearance with current NGA eligibility and SBU/Sec Net/COE accounts
• Must be willing to work in SCIF daily or as needed
• 5+ years of professional machine learning engineering experience with a focus on deep learning
• 4+ years of advanced Python development for ML workloads
• 3+ years of experience with computer vision or multimodal models
• 3+ years of experience with AWS ML infrastructure
• Sage Maker Training jobs, Processing jobs, and endpoint deployment
• GPU instance selection, multi‑node training, and cost optimization on EC2 (P4/P5/G5/G6e)
• 2+ years of experience building ML evaluation pipelines
• Automated benchmarking, metric computation, and result analysis
• Experience with both quantitative metrics and qualitative/human evaluation approaches
• 1+ years of hands‑on experience fine‑tuning large foundation models (LLMs or VLMs)
• Experience with parameter‑efficient fine‑tuning methods (LoRA, QLoRA adapters)
• Familiarity with supervised fine‑tuning, instruction tuning, and RLHF/DPO alignment techniques
• Strong proficiency with PyTorch and the Hugging Face ecosystem (Transformers, PEFT, Datasets, Accelerate)
• Experience with distributed training frameworks (Deep Speed, FSDP, or Megatron)
• Understanding of vision transformer architectures (ViT, CLIP, LLaVA‑family models, or similar)
• Experience processing and augmenting image datasets at scale
• Strong software engineering fundamentals (version control, CI/CD for ML workflows, testing)

• 2+ years of experience with geospatial or remote sensing imagery
• 2+ years of experience with Authority to Operate (ATO) processes in government environments
• Familiarity with electro‑optical and SAR satellite imagery formats and characteristics
• Understanding of geospatial metadata, coordinate systems, and imagery preprocessing
• Experience with model quantization and inference optimization (vLLM, Tensor
  
  RT, ONNX)
• Experience with MLOps and experiment tracking tools (MLflow, Weights & Biases, Sage Maker Experiments)
• Familiarity with data annotation platforms and active learning workflows for imagery
  
  Experience with containerized ML workflows (Docker, ECR, ES/EKS)
• Implementation of NIST 800‑53 controls and security compliance for ML systems
• Experience deploying models in air‑gapped or disconnected environments
• Familiarity with multimodal evaluation benchmarks (MMMU, MMBench, GQA, or domain‑specific equivalents)
• Publications or demonstrated contributions in computer vision, VLMs, or multimodal AI
• Experience with synthetic data generation for training data augmentation