AI​/ML Engineer - Lead Analyst

The Digital Solutions and Innovation (DSI) function works with members of Internal Audit (IA) to identify automation opportunities and develop solutions and capabilities to support the development of audit analytics. A key aspect of DSI’s strategy is to develop and deploy augmented intelligence (AI) and machine learning (ML) capabilities that are integrated with applications used by IA teams. The overarching objectives of these capabilities is to provide “in the moment” guidance to auditors, glean broader and deeper insights, enable efficiencies, and promote better quality and consistency.

The AI/ML Engineer - Lead Analyst is a critical role with the objective of exploring, developing, testing, deploying, enhancing, and maintaining machine learning and artificial intelligence capabilities integrated with Internal Audit’s management systems.

A broad understanding of AI/ML concepts and techniques, agile methodologies, cloud computing, intelligent chatbots, and related aspects is essential for the role. Additionally, a key aspect of this role involves the design and refinement of prompts to effectively communicate with generative AI models and to guide them, optimizing their output for a variety of technical applications and producing high-quality and contextually relevant content.

The role will entail collaborating with DSI, audit teams, external third parties and technology teams on ML use case ideation and prioritization, developing plans and timelines for implementation, thorough testing, deployment in a systemic way, maintenance to keep the capabilities up to date and relevant, and support in the onboarding of new technology for DSI.

The role will also involve collaborating with DSI team members and audit teams on new use case ideation, exploration, and eventual implementation.

Another success factor for the role will be documentation of model performance and other metrics and communication/presentation of results to pertinent audiences, including senior leadership. Along with a strong technical foundation, a creative, imaginative, forward-looking attitude and mindset is needed to continue to exert a positive impact on DSI and Internal Audit. Furthermore, a total quality assurance orientation is indispensable in all activities and outputs.

• Design, test, deploy and maintain machine learning use cases leveraging IBM Watson SaaS environment and toolset, and integrate with the audit management systems
• Enhance existing AI/ML capabilities through ideation, exploration, testing, prioritization, and implementation
• Design, test, and refine prompts to elicit the desired outputs from generative AI models, ensuring clarity, accuracy, effectiveness, and contextual relevance.
• Perform analytics on AI/ML capabilities usage to understand trends, patterns, and pain points and implement a sustainable feedback loop to ensure positive, continually improving user experiences.
• Explain the analysis of results in a methodical, analytical, visually-appealing manner
• Collaborate with Innovation, technology, vendor, and information security teams on updates, enhancements, releases, and other procedural requirements
• Assess current capabilities and chart a path for continuous improvement of the AI/ML capabilities value and impact
• Prepare, manage, and maintain model risk management (MRM) documentation on model performance and accuracy
• Engage with MRM team on adhering to and fulfilling requirements for MRM process
• Engage with relevant stakeholders on use case ideation, planning, development, and deployment
• Apply strong project management skills to develop project plans, timelines, milestones, and demonstrate progress toward objectives
• Utilize excellent communication skills to ideate, educate, and present internally, often at senior levels (across functions); expected to communicate with external parties
• Help shape the direction of machine learning and artificial intelligence capabilities in DSI
• Apply a strong understanding of business processes, risks and controls to identify opportunities to enhance audit efficiencies and effectiveness through the development and delivery of automated processes
• Drive…