Software Engineer Calibration

Position Description:

• Own the formal Deliverable Acceptance process for TOP's external vendor engagement, reviewing each submitted Deliverable against the Acceptance Criteria defined in the vendor SOW and issuing written acceptance or a specific, actionable defect list
• Verify vendor container image deliverables against the Software Bill of Materials (SBOM), confirming that all declared components are present and no unapproved components are included
• Validate that vendor-delivered AI engine deployments comply with Client's technical architecture requirements, including confirming that fine-tuned model weights are stored in Client's Vertex AI Model Registry and not embedded in container images
• Design, build, and maintain automated test suites for TOP platform backend services, APIs, and data pipelines
• Develop AI engine output evaluation frameworks that test inference quality against defined accuracy benchmarks for each dealer service use case
• Own the UAT (User Acceptance Testing) process for dealer-facing interfaces, coordinating with Client Service stakeholders to recruit pilot users, design test scenarios, and capture structured feedback
• Manage Jira project bug triage workflow: creating Bug records for confirmed Tier 2 issues, assigning priority in alignment with SLA tiers, tracking vendor acknowledgment and resolution timelines, and reporting SLA compliance metrics
• Perform regression testing for each new container image version before Client authorizes production deployment
• Define test environments within Client's GCP project space in collaboration with the GCP Cloud Engineer, ensuring test environments accurately reflect production configurations
• Produce quality metrics reports for Client program leadership covering defect rates, SLA compliance, and test coverage across all TOP platform components

• API - 3-5 years designing, executing, and validating API test cases using tools such as Postman or REST-assured. This includes verifying request/response contracts, authentication flows, error handling, and integration behavior across microservices within the Telemetry & Observability Platform.
• Cloud Infrastructure - 3-5 years of working knowledge of cloud-native infrastructure concepts including containerization, networking, IAM, and storage. Experience validating deployments and testing service behavior in cloud environments is expected.
• Google Cloud Platform - 2-5 years of hands-on experience working within GCP, including familiarity with services such as Pub/Sub, Big Query, GCS, or Cloud Run as they relate to testing data pipelines and telemetry workloads.
• User Acceptance Testing - 2-5 years coordinating and executing UAT cycles with internal stakeholders and platform consumers, ensuring delivered features meet business and functional requirements before production release.
• Application Testing - 3-5 years developing and executing test plans covering functional, regression, and smoke testing across platform applications and services throughout the SDLC.
• Software Testing - 3-5 years of broad software testing experience including unit, integration, system, and end-to-end testing, with the ability to contribute to or maintain automated test suites.
• Test Cases - 2-5 years authoring clear, traceable, and reusable test cases tied to acceptance criteria, with documentation maintained in Jira or a comparable test management tool.
• Jira
  - Ad Hoc Reporting - 2-5 years creating ad hoc Jira queries, dashboards, and filters to surface test coverage, defect trends, and sprint quality metrics for engineering and leadership audiences.
• Artificial Intelligence & Expert Systems - 1-3 years of familiarity with AI-assisted testing approaches, including the use of LLM-based tools for test generation, anomaly detection, or intelligent defect triage within developer workflows.
• Dynatrace - 1-3 years using Dynatrace for observability-driven testing, including leveraging traces, dashboards, and alerting to validate system health and performance during test cycles.
• Quality Assurance Concepts and Standards - 2-5 years applying QA methodologies such as shift-left testing, risk-based testing, and test-driven development, with experience maintaining consistent quality standards across teams.
• Quality Assurance/Control - 2-5 years owning quality gates within a CI/CD pipeline, including defining acceptance thresholds, managing test environments, and working with engineering teams to resolve defects efficiently.
• Testing
  - Performance - 2-4 years designing and running load, stress, or scalability tests against platform services, with the ability to interpret results and collaborate with engineers to resolve bottlenecks.
• Test Integration Testing - 3-5 years planning and executing integration tests that validate end-to-end data and event flows across platform services. including ingestion, processing, and telemetry delivery pipelines.

• 4 or more years of professional software quality assurance experience, with demonstrated experience in…