Finding What Works, Faster

Every time a company changes the layout of a website, introduces a new feature or tweaks the way an online platform works, there’s a question behind the change: Will this actually make the experience better?

Increasingly, companies answer that question through digital experimentation, testing different options with customers before deciding which one to adopt. But as the number and complexity of those experiments grow, organizations face another question: How much data do they actually need to make a confident decision?

A forthcoming paper in Information Systems Research offers a new way to answer that question.

“Fast Selection From Multiple Treatments: A Sequential Method for Principled Digital Experimentation” is co-authored by Will Stamey, a Ph.D. candidate in Analytics at the Mendoza College of Business and an affiliated student of the Human-centered Analytics Lab (HAL) at the University of Notre Dame; Ken Kelley, professor of IT, Analytics, and Operations and HAL co-director; and co-authors Bhargab Chattopadhyay and Tamal Bandyopadhyay. The researchers developed a statistical method to identify the best option among multiple treatments while avoiding both over- and under-sampling.

How much data is enough?

Stamey described the problem as a two-part question.

“How much data does an organization need to gather to confidently choose the most effective treatment option [an approach or intervention being tested] that makes a practical improvement to their current way of doing things?” he said. “Second, how should we actually gather that data sequentially in a way that does not over- or under-sample and results in unbiased estimates of each option’s effectiveness?”

Those questions have become increasingly important as experimentation has become commonplace in digital business. Companies test everything from small changes to websites to major product and user-experience updates before rolling them out broadly.

Experiments require participants, who are often customers interacting with a company's digital platform. Those participants are a limited resource. Collect too little data and the organization may not have enough evidence to make a reliable decision. Collect too much and it wastes resources while potentially exposing more customers than necessary to an inferior experience.

The researchers’ new sequential method is designed to find the balance. The approach determines how much data is needed to confidently choose among several alternatives, allowing data to be gathered sequentially without sampling beyond what is necessary to achieve the desired statistical power. It is also designed to produce precise estimates of the required sample size while reducing bias in estimates of the effectiveness of the different treatments.

For Stamey, the project exemplifies what draws him to this area of research in the first place.

“It lies at a fascinating intersection of practical tools that can help organizations improve their operations and theoretically engaging, elegant statistics that are enjoyable to work on,” he said.

Stamey’s broader research agenda examines the complex behavioral data organizations encounter from customers and other stakeholders in digital settings, and developing statistical frameworks that can make those data understandable and useful for decision-making. That means helping organizations act on their findings sooner while ensuring the evidence behind their decisions is sound.

Making better decisions without sacrificing rigor

Speed matters in digital experimentation. An organization may want to know as quickly as possible which website design generates more clicks, which feature increases engagement or which offer produces more sales.

Quickly finding a winner can come with a tradeoff. Methods optimized for fast decisions do not necessarily produce unbiased estimates of how well each option actually performs.

Kelley said the new approach helps firms do both: make an efficient choice and obtain nearly unbiased estimates of the effectiveness of the different options. That brings together practical business decision-making and the kind of statistical rigor more traditionally associated with scientific research.

The applications could extend well beyond a particular kind of website experiment. Kelley said the method can be useful whenever data arrive sequentially and an organization is considering multiple alternatives to its current way of doing things — including online surveys, website designs and algorithms used for functions such as search.

For practitioners, the researchers also compare their approach with existing methods, providing guidance for choosing an approach suited to a particular setting.

And they have taken an additional step toward putting the research into practice: an R package [a collection of tools for the R statistical software] that allows analysts to implement the method themselves.

“We have developed an R package that can help analysts directly apply our method in real-world operations,” Stamey said, “which has the potential to greatly improve their efficiency with their online data.”

Experimentation with people in mind

For all its statistical sophistication, Stamey sees the research as fundamentally concerned with people.

“This work is all about the problem of learning what works for people,” he said. “We might have many theories about which different treatments or experiences will benefit people, but at the end of the day, we have to test these theories empirically.”

The challenge is doing that learning efficiently and confidently.

An experiment that takes longer than necessary may expose additional customers to a less effective experience. But ending one prematurely can leave an organization without enough evidence to make a well-supported decision.

The researchers’ approach is designed to navigate that tension.

“This research proposes a way to do this that is efficient, so that not too many people receive an inferior experience as we are learning,” Stamey said, “But also confidently, so that we have strong evidence at the end of the experiment supporting our final decision.”

From a course project to Information Systems Research

The forthcoming publication also represents a milestone in Stamey’s development as a scholar.

The research grew out of a course project, and Kelley said the most rewarding aspect of the work was collaborating with Stamey and watching the project develop into his first publication in Information Systems Research, a top-tier journal in the information systems field.

Stamey credits Mendoza’s Ph.D. in Analytics program’s combination of quantitative training and broader conceptual thinking with helping him develop the project.

“Although my work is grounded in statistical methods development, methods are not interesting or useful if they are not focused on engaging, important problems,” Stamey said. “The holistic conceptual guidance we receive as Ph.D. students in Mendoza is extremely valuable for deciding what problems are important and how to think about them at a higher level.”

Faculty mentorship has been equally important.

“This project owes a large part of its success to the numerous conversations for which Ken constantly made time,” Stamey said, “and I am extremely grateful to him for that.”

For Kelley, that relationship works in both directions. Faculty can help doctoral students navigate the long path from an initial idea to publication, he said, while Ph.D. students bring new ideas, energy and perspectives that strengthen the research.

“It is truly synergistic,” Kelley said.

What comes next

The experience has also helped shape the direction Stamey wants to take his research.

“Writing this article was an entire education in itself,” he said. Through the project, he developed a deeper appreciation for sequential experimentation and best-treatment selection — a field he sees as particularly appealing because it “solves practical problems with elegant solutions.”

That combination of practical value and intellectual challenge is one he intends to continue pursuing.

One of his next projects focuses on experiments involving many treatments, but asks a more targeted question. Rather than simply determining which treatments work, the research uses anomaly detection to identify patterns across treatments and different groups of participants.

The goal, Stamey said, is to help organizations learn “not just which treatments work, but who they work for.”

It’s a natural extension of the question at the heart of his forthcoming paper: how organizations can learn what works — efficiently, rigorously and with the people behind the data in mind.

 

Jessica McManus Warnell is the managing director of research strategy, operations and engagement for the Human-centered Analytics Lab (HAL), and a teaching professor in the Department of Management & Organization.