Senior AI Engineer

Job Title: Senior AI Engineer
Location: Remote, USA
Clearance Required: Active Secret Clearance
Position Type: Full-Time

At Viv Soft, we aim to solve complex federal problems using emerging and open technologies in a collaborative and rewarding environment. Viv Soft is a diverse team of strategists, engineers, designers, and creators experienced in building high-performance, effective software, with a focus on impactful organizational design and software delivery dynamics. We build secure Software Factories based on DoD reference designs and NIST Frameworks for Cloud and Dev Sec Ops .

These factories deliver AI/ML Applications, Data Science Platforms, Blockchain, and Microservices for DoD, Healthcare, and Civilian Agencies.

We are seeking a Senior AI Engineer to design, develop, and deploy advanced Artificial Intelligence and Machine Learning (AI/ML) solutions within complex, large-scale data environments. This role will focus on building distributed machine learning models, developing agentic AI workflows, and integrating Large Language Models (LLMs) with enterprise data platforms to support predictive analytics, intelligent automation, and data-driven decision-making.

The ideal candidate will have extensive experience in software engineering, data engineering, and applied AI/ML, with expertise in distributed computing, statistical modeling, and modern AI technologies. This individual will work closely with data engineers, data scientists, and cloud/platform engineering teams to develop scalable AI solutions, optimize model performance, and deploy AI/ML capabilities within secure and disconnected environments.

• Design, develop, and deploy distributed machine learning models using Apache Spark and Apache Iceberg, applying rigorous statistical validation and classical machine learning techniques where appropriate.
• Build and implement agentic AI workflows using LLMs, tool calling, Model Context Protocol (MCP), retrieval-augmented generation (RAG), and multi-step planning to query and analyze enterprise datasets.
• Develop and optimize AI-driven data analysis workflows that leverage distributed query engines such as Amazon Athena, Trino, and Spark SQL.
• Optimize agent-generated SQL queries through partition pruning, file layout awareness, query guardrails, and distributed query performance techniques.
• Design, deploy, and maintain self-hosted AI/ML model-serving environments using Kubernetes, including deployments within disconnected or restricted network environments.
• Develop and orchestrate automated model training, evaluation, and agent workflows using Apache Airflow, AWS services, and Kubernetes.
• Leverage AWS technologies, including Amazon S3, EC2, and EMR, to support distributed AI/ML processing and deployment.
• Evaluate LLM and AI agent outputs using structured test datasets, evaluation rubrics, regression testing, and performance metrics.
• Monitor AI/ML solutions for model drift, performance degradation, and operational failure modes.
• Collaborate with data engineers, data scientists, and cloud/platform engineers to integrate AI/ML models into production data pipelines and enterprise environments.
• Apply responsible AI principles, including bias assessment, model transparency, data handling, and risk management.
• Provide technical leadership, mentor junior engineers, and contribute to engineering best practices.
• Develop and maintain technical documentation, architecture designs, operational runbooks, and implementation guidance.
• Communicate model behavior, limitations, and technical risks to technical and nontechnical stakeholders.

• Active U.S. Government Secret security clearance.
• U.S. Citizenship required.
• Minimum of 8 years of professional experience in software engineering, data engineering, or machine learning, including at least 4 years of applied AI/ML experience.
• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Science, or a related technical field. Additional relevant professional experience may substitute for education on a year-for-year basis.
• Demonstrated experience developing and deploying distributed machine learning solutions using Apache Spark, including Spark MLlib, and lakehouse table formats such as Apache Iceberg.
• Strong statistical background, including probability, statistical inference, regression analysis, experimental design, and model evaluation.
• Experience applying classical machine learning algorithms and…