Allen Tran
Measuring forever
The decision is permanent; the experiment lasts two weeks. The standard fix quietly answers a different question. A better one comes from treating an A/B test as a measurement of how the world moves, rather than of where it ends up.
A streaming service redesigns its homepage. If the redesign ships, members will live with it for years. The test that decides whether it ships will run for a few weeks. A drug meant to be taken daily for decades is approved on the strength of a trial measured in months. A city weighs a permanent change to air quality using studies far shorter than a childhood. In each case, the thing we want to know unfolds over a horizon no experiment will ever cover.
The established workaround is the surrogate. Find an early outcome that sits on the causal path to the late one—a biomarker, a week-two engagement metric—measure the treatment’s effect on it, and use a separate observational dataset to map surrogates to long-run outcomes. The surrogate index made this approach far more practical, and for many problems it is the right tool. It works best for shocks: a one-time discount, a training course, a short course of medication. When the intervention is over by the time the surrogate is measured, it is reasonable to believe the surrogate carries everything the intervention did.
A homepage redesign is not a shock. Neither is a daily pill or a polluted river. These are long-term treatments: continual exposures that keep acting on people after the experiment stops watching. Next month’s homepage has its own effect next month. No early outcome can mediate an effect that hasn’t happened yet.
Our first result makes this concrete, and it is a little unsettling. In our setting, even when every surrogate assumption holds, the surrogate index does not estimate a noisy version of the long-term effect of a long-term treatment. It estimates something else exactly: the long-term effect of a treatment that stops at the moment the surrogates are measured. Ask it what a permanent change will do, and it answers what a two-week change would have done. Nothing in the output warns you that the question was swapped.
···
The way out is to change what we ask the experiment to measure. A short test cannot show the distant outcome, but it does show one step of the process that produces it: from the state each unit is in, under treatment and under control, what happens this period, and where does the unit end up next? If the state is rich enough to summarize where a trajectory is heading—the Markov assumption—and the gap between treated and control dynamics holds steady over time, then one observed step can be chained into an entire future.
Stated that way, the problem is offline reinforcement learning. The state is the member’s or patient’s condition; the action is treatment or control; the reward is the per-period outcome. “Treat forever” and “never treat” are two policies, and the long-term average treatment effect is the difference in their discounted values. The experiment is simply logged data from a behavior policy that randomized. No long-run observational dataset is needed, and no surrogate has to mediate anything. The state does not need to carry the effect. It needs to describe where things are going.
One detail deserves its own paragraph, because it is the kind of trick that makes the formalism usable. Suppose we want the effect of treating for T periods and then stopping. That policy depends on the clock, so its value function does too, and non-stationary value functions are awkward to learn from a single transition. But any policy, stationary or not, induces a discounted occupancy measure—how much weight each state and action receives over time—and two policies with the same occupancy have the same value. We show that a T-period treatment has a stationary stochastic equivalent whose average treatment probability is exactly 1 − γT. In practice, “treat for a year” becomes “flip a coin with that probability every period.” Under discounting, duration and probability are interchangeable, and a single stationary Q-function can price treatments of any length, from one period to forever. (The state-independent version is an approximation, and as we’ll see, it is weakest for the shortest durations.)
For estimation we use a doubly robust estimator built on the efficient influence function from the off-policy evaluation literature. It combines two learned pieces: the Q-function, and a density ratio comparing how often each state would be occupied under the target policy with how often it appears in the experiment. Either piece can be estimated with flexible machine learning, each converging slower than root-n, as long as the product of their errors shrinks fast enough; if either one is right, the estimate is consistent. With cross-fitting, the estimator is asymptotically normal and efficient, so the long-range estimate comes with a confidence interval rather than a bare number.
···
The assumptions do real work, and they are worth naming plainly. The state must be Markov, which in practice means rich. The difference between treated and control dynamics must be stable, though overall levels can drift. And overlap is more demanding than in a standard experiment: the test has to reach every kind of unit whose future we care about, so a test run only on new members cannot speak for tenured ones.
We first checked the method in a simulation built to resemble a subscription service: tenure drifts upward unless a member churns, revenue rises with tenure but levels off, and the treatment both lifts revenue and nudges members toward longer tenure. The true effect therefore grows with how long the treatment lasts. Our estimates track it at every duration from a single period to permanent, using two periods of experimental data. A surrogate index that sees the same two periods returns a flat answer—correct for a one-period treatment, and increasingly too small after that. Give the surrogate index as many periods of data as the treatment lasts and it matches the truth, which is to say it works once the experiment is as long as the question.
We also tried to break it. Thinning out the experiment’s coverage of part of the state space inflates the variance, as theory predicts, though it takes heavy downweighting before it matters. Hiding a state variable is more dangerous: as the hidden variable becomes more relevant to the dynamics, estimates are pulled toward the mean and effects are understated, sometimes badly. If you adopt this approach, invest in the state.
Finally, we turned to a sepsis simulator from the clinical reinforcement-learning literature, with vital signs as states and discharge or death as the outcome. We treated each combination of antibiotics, vasopressors, and mechanical ventilation as a candidate long-term treatment. Ventilation alone has effects close to zero, and our confidence intervals cover zero at every duration. Ventilation with antibiotics has a large effect that grows the longer it continues, and our estimates follow that growth. They are least accurate for the shortest durations, where the coin-flip approximation is weakest. The surrogate index is accurate only for one-period treatments. For ventilation alone, it looks right at every duration—but only because duration doesn’t matter for that treatment. It is right for the wrong reason, which is the most dangerous way to be right.
···
The practical shift is in how one reads an experiment. We usually treat an A/B test as a measurement of an outcome and then argue about how far that outcome extrapolates. Read instead as a measurement of a transition law—how the world steps forward under each arm—the same two weeks of data say something precise about year three. They can’t show year three directly. But if the state is rich enough, year three is made of nothing except those steps.