Business Intelligence Analyst Senior; AI Evaluation & Monitoring

Job Summary

Senior BI Analyst serves as a partner to the business or clinical area supported (AI Evaluation & Monitoring) and uses analytic expertise—including knowledge of data structures, programming, and statistics—to help the organization make data‑driven decisions. The role involves collecting, cleaning, and interpreting data sets to answer questions or solve problems, identifying improvement opportunities, and supporting the evaluation of existing AI/ML projects.

Senior BI Analysts act as the analytics expert in meetings and can operate independently. The position requires strong oral and written communication skills, including the ability to translate analyses to non‑technical stakeholders, and excellent initiative, organizational, and customer‑relationship skills. Mentorship of junior BI developers, code reviews, and training are also part of the role.

Work is typically performed in an office or remote environment and the analyst is accountable for satisfying all job‑specific obligations and complying with all organization policies and procedures.

• Independently consults with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes.
• Executes evaluation work that keeps the AI portfolio safe, performant, and accountable—running validation studies, instrumenting production monitoring, analyzing results, and turning findings into clear written work products for governance and program teams.
• Collaborates with internal partners to design and develop reporting solutions and analytic studies to support business needs.
• Plans projects and requests completion tasks.
• Provides time and effort estimates based on requirements gathering; manages multiple projects independently.
• Demonstrates working knowledge of relevant source systems and proficiency with databases/data marts within the first year in the role. Stays abreast of industry trends and becomes familiar with business intelligence software systems used at Geisinger.
• Recommends analysis approaches and additional analyses or reporting needed based on business questions, requirements, and initial findings.
• Develops, validates, and executes complex queries; creates complex reports and performs complex data analysis that answers business questions for internal customers; interprets results and makes recommendations.
• Presents analysis and reports to internal customers and stakeholders.
• Uses statistical methods during analyses and selects visualization techniques that accurately reflect the intended message and limitations.
• Reports and troubleshoots data quality issues during profiling, testing, or validation.
• Collaborates and shares knowledge with team members.
• Participates in strategic departmental initiatives.
• Acts as a mentor to other analysts.
• May participate in industry advisory and/or user groups.
• Maintains strong attention to detail and clarity when describing results of data analyses.
• Proactively offers suggestions, provides resources, volunteers for assignments, and removes barriers to help the team accomplish its goals.
• Follows all department policies and procedures, including request intake/fulfillment, change management, and data governance practices.

• Healthcare, clinical, or regulated‑industry data experience strongly preferred.
• Applied statistics: hypothesis testing, confidence intervals, power and sample size, regression, and non‑parametric methods.
• SQL experience with large, messy clinical data warehouses, including window functions, CTEs, debugging query plans, and performance tuning.
• Python for analysis (pandas, numpy, scikit‑learn, stats models, common evaluation libraries).
• Experience designing and executing production monitoring: drift detection, performance decay tracking, adoption and outcome metrics.
• Deployment and evaluation of LLM and generative AI: golden sets, judge‑based scoring, hallucination and grounding checks.

High School Diploma or Equivalent (GED) – Required

Minimum of 8 years relevant experience – Required

None specified.

We are an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.