Allen Tran
The shelf and the aim
A recommendation does two things at once: it puts a title on the shelf, and it aims that title at the people most likely to want it. At Netflix, separating the two shows that the aim is worth far more than the shelf space.
Open Netflix and a single title fills the top of the screen. Below it, rows: New on Netflix, Top Picks for You, and on down. Most engagement with recommendations happens in the top-left corner—roughly the first five titles in the first five rows. Suppose you press play on something in that corner. Did the system understand your taste, or did it just put the title where you couldn’t miss it?
Almost always, it did both, and the two are hard to separate for a reason any economist will recognize. Recommendations are targeted. They go to the people the system already expects to watch. Compare viewers who were shown a title with viewers who weren’t, and most of the gap reflects who was chosen, not what being chosen did. The better the recommender, the worse that comparison gets.
The usual tools of demand estimation offer little help here. Industrial organization learns what people value from how they respond to prices, but once someone subscribes, every title costs the same: nothing. Film and television have long frustrated characteristic-based demand models, because cast and genre do a poor job of predicting what people watch. Choices depend on each other—watch a sports movie and you’re likelier to try a sports series next. And the scale is large: we model two million members choosing among roughly seven thousand titles.
···
Our model is a discrete-choice model in which each day a member watches one title or nothing. A member’s utility for a title has two parts. One is intrinsic: a low-rank product of a member preference vector and a title embedding. The other is a recommendation bonus, which depends on where the title appears: the billboard at the top, the top-left block of twenty-five, or the wider block of a hundred.
Two design choices let this run at Netflix scale. First, title characteristics are not supplied; they are learned from choices. The learned space makes sense: Love Is Blind sits next to Married at First Sight and far from the action film Faster; Despicable Me 4 sits next to Sonic the Hedgehog 2 and Minions and far from A Man in Full. Second, members get no free parameter vector of their own. Their preferences are computed from their viewing history by an attention-weighted sequence model, so tastes can shift over time and the parameter count grows with titles rather than members. It is a state-dependent demand model, in other words, built with a transformer-style attention mechanism.
Identification comes from something every production recommender does: explore. To keep learning, the system injects some randomness into what it shows. Conditional on everything the recommender itself conditions on, the remaining variation in exposure is as good as random. That is an assumption, and we treat it as one. So we tested it.
···
The test borrows a tool from antitrust. When regulators evaluate a merger, they ask about diversion: if one product became unavailable or worse, where would its customers go—to a rival, or nowhere? Diversion ratios are normally estimated from price changes or stockouts. We had neither, but we could run an experiment. We randomized members into eight arms, each built around a category of titles—scripted versus unscripted, a particular language, film versus series. Within each arm, impressions of other titles were swapped for impressions of titles in the focal category. Focal titles became more salient; others less. Each arm had about a million members and ran for five weeks, with perturbations comparable to those in routine algorithm tests. From these we computed model-free diversion ratios: how viewing moved among titles, and how much of it moved to not watching at all.
Then we fit the model on the control arm alone, simulated each intervention, and compared. Out of sample, the model’s predicted diversion ratios correlate with the experimental ones at 0.86, with an R2 of 0.73. A model fit only on observational exploration reproduces what happens when recommendations are deliberately perturbed.
···
With a validated model, we can swap out the recommender. Holding the number of recommendations fixed, replacing the current system with random recommendations reduces engagement by 16 percent. A popularity-based system, the best option that doesn’t personalize at all, reduces it by 12 percent. A vanilla matrix-factorization recommender, a stand-in for the state of the art of about a decade ago, reduces it by 4 percent. In 2019, a much-discussed study asked whether neural recommenders were really beating well-tuned classical baselines. At this scale, the gain over matrix factorization is real and economically large. Our numbers are probably conservative, since the recommendation bonus partly reflects the trust members have built in the current system.
Diversity tells the same story. A common worry is that personalization narrows consumption. Here the opposite holds. Popularity-based and matrix-factorization recommendations concentrate viewing sharply, raising its Herfindahl index by 42.5 and 37.5 percent respectively. The current system is nearly as diverse as recommending at random—random lowers concentration by only 2.5 percent—while producing far more viewing.
Then comes the question in the title. For each title, take the gap between how often targeted members watch it when it is recommended and how often an average member watches it when it isn’t. That gap has three parts. Selection: targeted members were already likelier to watch. Exposure: recommending the title to anyone raises viewing. Targeting: the people actually chosen respond more strongly than an average member would.
Exposure is powerful on its own. Putting a title in front of an average member roughly triples the chance they watch it. Yet across recommended titles, exposure explains only 6.8 percent of the gap. Selection explains 51.3 percent and targeting 41.9 percent. Targeting is worth nearly seven times as much as shelf space. Under matrix factorization, by contrast, the three channels are roughly equal, which suggests that what a decade of progress improved was the aim.
The gains from targeting fall unevenly across the catalog. They are smallest for the biggest hits, which appeal to nearly everyone and need no help being found. They are also small for the deepest niches, whose audiences may be too small to find reliably. The largest gains go to titles of middling popularity, which have sizeable but specific audiences who would probably not find them unaided. This is not the long-tail story, in which algorithms mainly lift obscure titles. The beneficiaries are in the middle.
···
The same model answers a question that matters for running a catalog: how much viewing does a particular title create, net of what members would have watched instead? Remove it in simulation and see how much of its viewing vanishes versus how much moves elsewhere. For titles that don’t exist yet, we swap the learned embeddings for embeddings built from metadata and human-applied tags. That version predicts viewing shares with an R2 of 0.96 and experimental diversion with an R2 of 0.65. The two approaches trade off as you’d expect. Learned embeddings need scale: trained on 1 percent of members, the model’s validation R2 is nearly 60 percent worse, and they overtake the metadata version only at around 30 percent.
The model has limits. The recommendation bonus is the same for every member, so responsiveness varies only through preferences and what each member is shown; it approximates a consideration-set model rather than fully specifying one. Because the formulation is reduced-form, we measure engagement and diversity, not consumer welfare, though both are known to track long-run satisfaction and retention. And identification rests on the recommender’s exploration being as good as random once we condition on its inputs. The salience experiment is our check on that assumption, and the estimates hold up when the model is fit on experimental data directly.
Shelf space is easy to measure and easy to overvalue: show something to millions of people and some of them will watch. The harder thing to measure, and the thing that turns out to matter most, is whether the recommender reached the right people.