Experimentation and Data-Driven Decision Making in eCommerce
Simulation Statistics
Duration: 50 minutes
Modality: synchronous; in-class or remote learning
Author: Kartik Hosanagar & Alex Miller
Subject: Marketing, Strategy, Teamwork
Learner Level: Advanced High School, Undergraduate, Graduate, and Executive Education
Team Size: Individual Practice, 2-5 players/team during tournament
This simulation challenges users to design and analyze real-world experiments that drive data-driven marketing and product decisions. Through hands-on testing and competition, students develop practical intuition for experimentation, optimization, and the application of machine learning in business strategy.
Simulation Summary
A/B Testing places students in the role of Director of eCommerce at Nano, where they are tasked with a series of controlled online experiments to determine the best strategies for driving smartphone sales. Operating in a fast-moving digital environment, participants must decide how to test website features, ranging from layout changes to promotional strategies, all while weighing the tradeoff between acting quickly on limited data vs. waiting to collect more reliable information.
The simulation unfolds across both an optional practice phase and a competitive tournament. As they design and run randomized experiments, participants must interpret performance and statistical data, continuously refining their optimization approach. Whether working independently or in teams, players experience the challenges of sequencing tests, allocating web traffic, and determining when results are actionable in the face of uncertainty.
As decisions compound, the simulation reveals how premature conclusions or overly cautious strategies can undermine performance, highlighting the importance of disciplined experimentation. Students begin to see how even small changes in test design or interpretation can significantly impact outcomes, mirroring the complexities faced by modern digital businesses. The experience also introduces how machine learning can be utilized to improve optimization, reinforcing the value of structured experimentation in data-rich environments.
Designed for flexible classroom and complexity settings, A/B Testing supports both synchronous and asynchronous play in different play modes and includes built-in analytics enabling instructors to review decisions, compare performance, and guide debrief discussions.
Developed by Professors Kartik Hosanagar and Alex Miller of the Operations, Information, & Decisions Department, this hands-on simulation offers a practical introduction to experimentation strategy, equipping students with the tools and mindset needed to make informed, evidence-based decisions in the ever-changing world of marketing and product development.
Gameplay Preview
About the Authors
Kartik Hosanagar is the John C. Hower Professor of Technology and Digital Business and a Professor of Marketing at The Wharton School of the University of Pennsylvania. Kartik’s research work focuses on the digital economy, in particular the impact of AI on consumers and society, Internet media, Internet marketing, and e-commerce. Read more
Alex Miller is a full-stack scientist and engineer with expertise in applied statistics, A/B testing, causal inference, and machine learning. As a former academic (Quant Marketing @ USC, Wharton PhD in Information Systems), he specialized in applying advanced statistical, econometric, and ML techniques to solve problems in personalized targeting, online experimentation, and recommendation systems. Read more.
Play Now!
Wharton affiliated?
- Contact Learning Lab to use in your classes: learninglab@wharton.upenn.edu
Outside of Wharton?
- Available on these markets