Lakehouse Performance Engineer

Overview

At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation.

With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.

Introduction at IBM Software emphasizes transforming client challenges into solutions, and building AI-powered, cloud-native products that shape the future of business and society.

IBM is building the next generation of watsonx.data: a GPU-accelerated, open data lakehouse engineered to deliver category-leading price‑performance for analytics and AI workloads. We are hiring a Performance Engineer to focus on measuring, defending, and improving the performance and cost‑per‑performance of the platform across every release.

You will run the benchmarks, build the harnesses, and operate the backing infrastructure that the entire watsonx.data organization relies on to characterize performance. That includes the dedicated benchmark labs, GPU and CPU test fleets, dataset stores, result warehouses, and the automation that ties them together. Engineering, product, field, and competitive intelligence will all consume what you produce: regression signals in CI, executive scorecards, customer-facing dashboards, and the data behind claims that we are the market-leading open lakehouse.

• Industry-standard benchmarks:
  Run, maintain, and continuously improve reproducible benchmarks across watsonx.data configurations and against competitive offerings.
• Customer-representative workloads:
  Build and curate workload suites that reflect real customer query mixes, data volumes, concurrency profiles, and freshness requirements: not just synthetic benchmarks.
• Reproducibility & rigor:
  Ensure every published result is reproducible end-to-end: controlled environments, pinned versions, locked datasets, documented methodology, variance analysis, and statistically defensible reporting.
• Cost-per-performance metrics:
  Operationalize the canonical price‑performance KPIs ($/query, $/TB scanned, $/training‑token, queries/sec/$, TCO at workload mix); instrument workloads, collect data, and produce repeatable scorecards.

• Telemetry pipeline:
  Build and maintain the metrics, traces, profiles, GPU/CPU utilization, query plan, and IO telemetry that flow from benchmark runs into the performance data store.
• Dashboards & scorecards:
  Develop dashboards that surface trends, regressions, and competitive position to engineering, leadership, and external audiences.
• Regression gates:
  Operate performance regression gates in CI/CD; triage failures, file and drive issues with engine, storage, and GPU teams, and verify fixes.
• Root-cause analysis:
  Drill into slow queries and GPU/CPU bottlenecks using profilers (Nsight, perf, async-profiler, pprof, flame graphs) and query plan inspection to pinpoint regressions and improvement opportunities.

• Performance environment ownership:
  Own the lifecycle of the dedicated performance environment(s) supporting watsonx.data: GPU and CPU clusters, networking, storage, and the orchestration that schedules workloads onto them.
• Test fleet automation:
  Build and maintain infrastructure-as-code (Terraform/Ansible/Helm) for provisioning, configuring, and resetting test environments deterministically across on-prem hardware and cloud (IBM Cloud and partner clouds).
• Benchmark harness platform:
  Develop and operate the benchmark harness itself: job scheduler, run orchestration, dataset provisioning, result capture, artifact storage, and…