Senior Experimentation Engineer

Responsibilities

• Design and build the core experimentation platform infrastructure: experiment assignment service, randomization and traffic splitting, multi-layer experiment conflict detection, experiment configuration management (low-code, ≤30 min setup), and full lifecycle tooling (creation, monitoring, graduation, rollback) — supporting App, Web, and backend surfaces simultaneously
• Build automated metric pipelines that connect experiment assignments to full-funnel business outcomes — Signup CVR, eFTD CVR, eFTT CVR, deposit amount, trading volume — with sub-day latency; solve the cross-device identity bridging problem (device  user s the registration boundary) and implement "time-to-convert" metrics (e.g., eFTD within N minutes of page exposure) that are currently unavailable
• Implement advanced statistical methods: CUPED and stratified variance reduction to improve experiment sensitivity without increasing sample size, sequential testing for early stopping (mSPRT / always-valid inference), network interference correction for referral and social experiments, and pre-experiment SRM (Sample Ratio Mismatch) checks; design and execute conversion lift studies to quantify the causal impact of product changes on business metrics
• Build self-serve experiment creation and analysis tooling for product managers, growth marketers, and data scientists — including experiment design wizards, power calculators, automated significance reporting, and decision support dashboards; reduce the experiment launch cycle from "requires 2 days of engineering" to "self-serve in under 30 minutes" for all three platforms (App, Web, backend)
• Establish experiment quality standards: event instrumentation requirements, guardrail metric monitoring, automated anomaly detection to prevent shipping regressions, and an experiment knowledge base that documents winning patterns, failed hypotheses, and domain-specific learnings — ensuring teams learn from each other rather than rediscovering the same findings
• Partner with Growth Product, Personalization (千人千面), Trade
  
  GPT, ByX Community, and Asia-Pacific data engineering teams to design and analyze high-impact experiments across personalization, campaign optimization, user journey, push notifications, community feed, and AI product features; serve as the internal authority on causal inference and experiment design across all business units
• Build the experimentation platform as the feedback engine for Bybit's AI strategy: design automated attribution systems and causal inference pipelines that deliver real-time feedback signals to AI models (Trade
  
  GPT response ranking, personalization algorithms, push notification optimization) — enabling AI models to self-iterate based on causal experiment results rather than correlation-only metrics; build experiment bloodline tracking that traces how each AI model version performs across user segments, and end-to-end observability for recommendation and growth systems that accelerates the iteration cycle from research to production deployment
• Define engineering standards, conduct design reviews, and mentor junior engineers; drive cross-team adoption of experimentation best practices as the US team grows

• 5+ years of industry experience in experimentation engineering, data engineering, or growth engineering at a consumer-scale internet company
• Proven track record building and operating large-scale A/B testing infrastructure: experiment assignment, metric pipelines, statistical analysis, and self-serve tooling serving hundreds of experiments simultaneously
• Deep expertise in causal inference and experimental statistics: hypothesis testing, power analysis, CUPED/variance reduction, sequential testing (mSPRT, always-valid inference), network effects and interference correction, conversion lift studies, and treatment effect estimation; ability to apply statistical test theories to optimize user experience and validate business decisions
• Strong proficiency in Python and SQL; hands‑on experience with real-time data processing frameworks — Flink Streaming or Spark Structured Streaming — as well as data warehouses (Hive, Click House, or equivalent),…