Scheduler Administrator​/Infrastructure Engineer Security Clearance

Position: Workload Scheduler Administrator / Infrastructure Engineer with Security Clearance
We are seeking a highly skilled (3-5+ years dedicated experience administering) IBM Workload Scheduler (IWS) Administrator to manage, maintain, and optimize our Partner's enterprise batch scheduling infrastructure. The successful candidate will be responsible for the end-to-end administration of the IWS environment hosted primarily on Red Hat Enterprise Linux (RHEL). This role requires a strong blend of IWS expertise, Linux system administration, and scripting to ensure high availability and seamless execution of critical business workloads.

Responsibilities
* Administer Production IBM Workload Scheduler (aka Tivoli Workload Scheduler) environment with 28,000 unique daily jobs across about 350,000 daily job runs, 44 servers, and three other change control environments
* Administer, install, configure, and patch/upgrade IWS components (Master Domain Manager, Dynamic Agents, Dynamic Pool, Dynamic Workload Console)
* Work with Product Owner on communicating work streams in Jira
* Manage job promotions with Workload Application Template-based process, ensuring safety checks for platform stability assessed on each job promotion
* Manage change control of four (4) separate change control environments, enforce change control standards and policies
* Maintain and continuously promote 99.17% Production platform uptime per calendar month (uptime calculated excluding planned change control outages, planned weekly maintenance windows) using SOP’s, Dev Ops tools, and disciplined change control across change control environments
* Orderly communication of platform-impacting news to user community of about 500 developers and data engineers
* Production consists of 44 servers between MDM, DWC, and agents
* Resolve complex job failures, performance bottlenecks, agent issues, and infrastructure issues
* Advise on complex job scheduling design questions that less experienced scheduling support team
* Monitor the health of the scheduler environment, manage database maintenance, and perform backup/disaster recovery, and monthly failovers
* Define and maintain security policies, user authorizations, and authentication for the DWC
* Respond to Cybersecurity vulnerability assessments and PCI (and other regulatory) audit inquiries
* Design and implement Ansible automation and self-healing mechanisms to continuously reduce unplanned outages
* Coordinate with offshore performing SOP’s during non-working hours
* Scripting Python with IWS REST API Requirements
* US Citizenship
* Strong experience with IBM Workload Scheduler architecture, especially Dynamic Workload Broker, V10.1+, high availability of MDM’s managing Fault Tolerant Agent and Dynamic Agent agent architectures
* Strong conceptual understanding of Master Domain Manager (MDM), Backup MDM (BMDM), Dynamic Workload Console (DWC), Fault Tolerant Agent (FTA), Dynamic Agent (DA)
* Strong grasp of conman CLI to monitor and control production plan, check job/job stream/resource status
* Strong grasp of composer CLI to define, modify and extract scheduling objects
* Strong grasp of planman CLI to control pre-production plan and GUI mirroring
* Strong grasp of lifecycle of daily production planning process, phases of JNextplan/FINAL
* Proficiency in navigating the DWC web-based GUI to monitor workloads, manage user access security, and define scheduling objects
* Experience installing IWS components, applying Fix Packs, and Interim Fixes
* Troubleshooting with logs under TWSDATA/stdlist, adjusting trace level for netman, batchman, writer, mailman, etc.
* Strong experience with IBM Web Sphere Liberty
* Strong grasp of reading messages.log, traces.log, FFDC logs
* Strong grasp of configuring JVM heap sizes
* Strong grasp of configuring tracing scope, tracing levels, tracing retention
* Strong experience with Red Hat Enterprise Linux 8+
* Deep familiarity with bash/shell commands for text processing (for example, grep, awk, sed), file manipulation, and system navigation
* Ability to manage, start, stop, and troubleshoot System

D services using systemctl and journalctl for IWS agents and MDM
* Managing user accounts, groups, service accounts and deep knowledge of Linux…