AI Evaluation Subject Matter Expert Security Clearance

Position: AI Evaluation Subject Matter Expert with Security Clearance
Work Arrangement:
Hybrid Clearance:
Active Secret w/TS Capability Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader in navigating technology and security challenges, Foxhole delivers mission-focused innovations to answer evolving and complex needs. Our talented employee-owners provide agile, scalable services and solutions that solve operational gaps, operate critical systems, and protect and secure the enterprise - across the organization and around the world Foxhole Technology is seeking an AI Evaluation SME to join an existing program.

The AI Evaluation SME will support the assessment, testing, validation, and operational evaluation of artificial intelligence, machine learning, automation, analytics, and decision-support capabilities being considered for or integrated into the Navy's Next Generation CANES environment. This role will help ensure AI-enabled capabilities are mission-relevant, reliable, secure, explainable, measurable, and suitable for deployment within afloat, tactical, disconnected, intermittent, limited-bandwidth, and multi-security-domain environments.

The SME will develop evaluation frameworks, test methods, metrics, datasets, scenarios, risk assessments, and reporting products that help Navy and CACI stakeholders determine whether AI-enabled capabilities improve network operations, cyber defense, system administration, predictive maintenance, anomaly detection, configuration management, mission planning, or other CANES-related functions. .

KEY RESPONSIBILITIES:

* Serve as a senior technical advisor for AI evaluation, test planning, performance assessment, and operational suitability analysis in support of Next Generation CANES modernization.
* Develop AI evaluation strategies, test plans, measures of effectiveness, measures of performance, success criteria, risk indicators, and evaluation scorecards.
* Assess AI, machine learning, generative AI, automation, analytics, and decision-support capabilities for operational relevance, technical maturity, cyber risk, reliability, maintainability, explainability, human oversight, and fleet suitability.
* Evaluate AI-enabled tools for use cases such as network monitoring, cyber anomaly detection, event correlation, predictive maintenance, help desk automation, configuration compliance, system health monitoring, log analysis, vulnerability prioritization, and operational decision support.
* Design test scenarios that reflect Navy afloat operating conditions, including limited bandwidth, disconnected operations, contested cyber environments, cross-domain constraints, variable data quality, and platform-specific operational limitations.
* Define data requirements, ground truth methods, evaluation datasets, labeling approaches, validation methods, and performance baselines for AI-enabled capabilities.
* Assess AI model performance using appropriate metrics such as accuracy, precision, recall, false positive rate, false negative rate, latency, robustness, drift, confidence calibration, explainability, and operational impact.
* Evaluate risks associated with hallucination, model brittleness, adversarial manipulation, data poisoning, prompt injection, bias, over-reliance, model drift, cybersecurity exposure, and failure modes in operational environments.
* Support AI red teaming, cyber survivability assessment, adversarial testing, safety reviews, and responsible AI evaluation activities.
* Develop human-machine teaming concepts, operator-in-the-loop workflows, trust calibration approaches, escalation procedures, and recommended guardrails for AI-enabled tools.
* Produce technical reports, evaluation findings, executive summaries, test observations, data analysis products, and recommendations for Navy and CACI leadership.
* Collaborate with systems engineers, cybersecurity engineers, software developers, data scientists, network engineers, operational testers, fleet users, and government stakeholders.
* Support technical interchange meetings, design reviews, test readiness reviews, operational assessments, demonstrations, and acquisition decision support.
* Provide SME input on AI governance, responsible AI implementation, model lifecycle management, configuration control, sustainment, monitoring, and continuous evaluation.

REQUIRED QUALIFICATIONS:

* Bachelor's degree in computer science, data science, artificial intelligence, engineering, mathematics, statistics, cybersecurity, operations research, information systems, or a related technical discipline preferred. Advanced degree preferred.
* Additional years of directly relevant AI evaluation, test, cybersecurity, Navy, or DoD mission system experience may be considered in lieu of a degree.
* Demonstrated experience evaluating AI, machine learning, data analytics, automation, or decision-support systems in defense, intelligence, cybersecurity, network operations, enterprise IT, or mission system environments.
*…