What counts as a good gain over the market in Benter's second-stage test?

Bill Benter reported a pseudo-R-squared gain of .0178 for one of his models, and .0090 in a later out-of-sample run on 2,313 races, describing the first as enough for significant profits. The scale depends on the sport and the market, so treat those as reference points. A gain indistinguishable from zero across cross-validation folds is a fail.

Why blend model and market probabilities in log space?

Benter's paper states that the natural log of probability is used rather than probability because the transformation provides a better fit. Averaging raw probabilities treats a disagreement between 2 and 4 percent the same as one between 50 and 52 percent. Log space treats the first as the larger disagreement, which matches how prices behave.

Does the second-stage test work on soccer 1X2 markets?

Yes. The formula in Benter's paper is already N-way. Replace the two-term normaliser with a sum over every outcome and swap the coin-flip baseline for log(1/N). Soccer 1X2 gives a wider spread of probabilities than a baseball moneyline, so the pseudo-R-squared values come out larger.

If my model passes the test, will it make money?

Not automatically. The test measures information rather than profit. Converting a positive gain into money requires a price better than the combined fair number and a stake the bookmaker will accept, so check the limit field on live outcomes and the exchange depth ladders before sizing anything.

Combine Your Model With Market Odds: Benter’s Second-Stage Test

Q: Can I test a betting model without a results feed?

Yes. The OddsPapi /historical-odds endpoint keeps recording price snapshots after kick-off, so the winning side's price collapses toward 1.00 and the final snapshot of the day works as a result label. Require every bookmaker on the fixture to agree on which side collapsed, and drop games that never resolve.

How To Guides August 10, 2026

Your model says the Dodgers win 61% of the time. Pinnacle says 58%. Do you have an edge, or do you have a rounding error?

Bill Benter published the answer in 1994, in a horse racing paper more often cited than read: “Computer Based Horse Race Handicapping and Wagering Systems: A Report”, collected in Efficiency of Racetrack Betting Markets. It reports significant positive results across five years of live operation in Hong Kong. The paper spends less time on his handicapping model than on what he calls the second stage: a small logit model that takes his probability and the public’s probability, combines them, and returns one number telling him whether his model was worth running at all.

That number is the test in this post. It needs three things per game: your probability, the market’s probability, and what happened. OddsPapi hands you the market for free across 383 bookmakers, and the free /historical-odds endpoint turns out to carry the result as well, hidden in plain sight. Everything below runs on the free tier.

What Benter found in 2,313 races

Benter compared two forecasters. The first was his own logit handicapping model built on nine significant fundamental factors. The second was a probability derived from tallying the picks of roughly 48 newspaper tipsters, scoring 6 points for a first-place pick, 3 for second and 1 for third.

On their own the two looked interchangeable. The fundamental model scored an out-of-sample pseudo-R² of .1016. The tipster tally scored .1014. Then he ran the second stage on 2,313 races from September 1988 to June 1993, combining each one with the public odds:

Forecaster	Alone (R²)	Combined with the public (R²)	Gain over the public
The public odds	.1237	n/a	n/a
Fundamental model	.1016	.1327	+.0090
48 newspaper tipsters	.1014	.1239	+.0002

Two forecasters, indistinguishable on their own scores, and one of them is worthless. Benter’s words on the tipster: carrying out the second stage “would have saved that player from losing money”, because the combined model’s output stays virtually identical to the public estimate and therefore never flags an advantage bet.

The lesson generalises past horse racing. Standalone accuracy tells you how much your model knows. It does not tell you how much your model knows that the market does not already know, and only the second quantity pays.

Why this beats a straight ROI backtest

The usual way to check a model is to simulate a season of bets and look at the profit. That works, and our backtest walkthrough covers it. It also has three problems: it needs a lot of games before the noise settles, it mixes model quality up with staking and price selection, and a single hot streak reads as skill.

	ROI backtest	Benter’s second stage
What it measures	Money, given a staking plan	Information, on its own
Confounded by	Stake sizing, vig, which book you priced against	Nothing but the two probabilities
Games needed	Hundreds, and more if the edge is thin	Fewer, because every game contributes a graded probability
Answers	Would this have made money last season	Does this know anything the market missed
Data needed	Prices, results, a bankroll model	Prices and results
Cost here	Free tier	Free tier

Run the second stage first. It is cheaper, and a model that fails it will never make money no matter how you size the bets.

Grading a forecast when the feed has no results

OddsPapi carries schedules, status and odds. No scores, no box scores, no settlement flags. Our free sports data guide spells out what the fixture object does and does not hold. So the obvious blocker: how do you grade a probability without knowing who won?

Read it off the prices. /historical-odds keeps recording snapshots after first pitch, all the way through the game, and by the final out the winner’s price has walked down toward 1.00 while the loser’s has run away. One real example from this study, Pinnacle on a Giants game:

Snapshot	Side 1 (Giants)	Side 2 (Rockies)
Opening price	1.769	2.18
Last price before first pitch	1.806	2.14
Final snapshot of the night	1.041	11.5

Side 1 at 1.041 is the feed telling you the Giants won. Two checks keep this honest. Every book on the fixture has to agree on which side collapsed, and prediction markets settle harder than sportsbooks do: pull the same fixture with bookmakers=polymarket and the winning outcome’s last snapshot is exactly 1, the loser 1000. On 176 labelled games here, the books disagreed on 1.

Step 1: Pull the finished fixtures

Authentication is a query parameter. /fixtures caps its window at 10 days, so walk the calendar, and filter on tournamentName because sport 13 returns every baseball competition on the planet, including the Dominican Summer League and Single-A California.

import math
import time
from datetime import datetime, date, timedelta

import requests

API_KEY = "4040b6fc-c40e-48eb-9896-c682d9a92150"
BASE_URL = "https://api.oddspapi.io/v4"

MONEYLINE = "131"          # Winner (incl. extra innings), sportId 13
SIDE_1, SIDE_2 = "131", "132"


def api(path, **params):
    """One GET with the free-tier 429 handled. apiKey is a query param, not a header."""
    for _ in range(8):
        r = requests.get(f"{BASE_URL}/{path}", params={"apiKey": API_KEY, **params}, timeout=600)
        body = r.json()
        if r.status_code == 200 and not (isinstance(body, dict) and body.get("error")):
            return body
        time.sleep((body.get("error") or {}).get("retryMs", 3000) / 1000 + 0.5)
    raise RuntimeError(f"{path} kept rate-limiting")


def moment(stamp):
    return datetime.fromisoformat(stamp.replace("Z", "+00:00"))

def finished_fixtures(sport_id, tournament, start, end):
    """Every completed fixture in one competition. /fixtures caps the window at 10 days."""
    found = {}
    day = start
    while day < end:
        stop = min(day + timedelta(days=9), end)
        for f in api("fixtures", sportId=sport_id,
                     **{"from": day.isoformat(), "to": stop.isoformat()}):
            if f["tournamentName"] == tournament and f["statusName"] == "Finished":
                found[f["fixtureId"]] = f
        day = stop + timedelta(days=1)
        time.sleep(1.0)
    return sorted(found.values(), key=lambda f: f["startTime"])

Finished fixtures come back with hasOdds: false, because the live feed drops prices once a game ends. The history is still queryable, which is the whole point.

Step 2: Take the closing price out of the history

Two traps live in this step. The historical endpoint returns bookmakers and a list of snapshots per outcome, while the live /odds endpoint returns bookmakerOdds and a single price dict. Mixing them up is the most common way this code fails. And /historical-odds accepts a maximum of three bookmakers per call, so loop if you want more.

def moneyline_snapshots(fixture, books=("pinnacle", "draftkings", "fanduel")):
    """{book: {side: [snapshot, ...]}} for the moneyline. Max 3 books per call."""
    hist = api("historical-odds", fixtureId=fixture["fixtureId"], bookmakers=",".join(books))
    out = {}
    for book in books:
        markets = (hist.get("bookmakers", {}).get(book) or {}).get("markets", {})
        outcomes = (markets.get(MONEYLINE) or {}).get("outcomes", {})
        sides = {}
        for side in (SIDE_1, SIDE_2):
            snaps = [s for s in (outcomes.get(side, {}).get("players", {}) or {}).get("0", [])
                     if s.get("price")]
            if snaps:
                sides[side] = snaps
        if len(sides) == 2:
            out[book] = sides
    return out


def closing_prices(sides, kickoff):
    """The last price posted before kick-off. Later snapshots are in-play."""
    prices = {}
    for side, snaps in sides.items():
        pre = [s for s in snaps
               if s.get("active") is not False and moment(s["createdAt"]) < kickoff]
        if not pre:
            return None
        prices[side] = max(pre, key=lambda s: s["createdAt"])["price"]
    return prices

The createdAt < kickoff filter is doing real work. Leave the in-play snapshots in and you will grade the market on prices set while the game was being played, which makes any forecaster look terrible by comparison.

Step 3: Read the result off the collapse

def outcome_label(book_snapshots, threshold=1.10):
    """Which side won, read off the price collapse after the final whistle.

    The feed keeps recording through the game. By the last out the winner's
    price has walked down to about 1.00 and the loser's has run away, so the
    final snapshot is a result label. Every book present has to agree.
    """
    votes = []
    for sides in book_snapshots.values():
        p1 = sides[SIDE_1][-1]["price"]
        p2 = sides[SIDE_2][-1]["price"]
        if min(p1, p2) > threshold:      # never resolved, no vote
            continue
        votes.append(SIDE_1 if p1 < p2 else SIDE_2)
    if not votes or any(v != votes[0] for v in votes):
        return None
    return votes[0]

The 1.10 threshold is deliberately loose. A price at 1.10 or below is 91% or better, which no pre-game baseball line reaches, so anything under it is a settled game rather than a lopsided one. Books that stopped recording earlier abstain instead of voting: Pinnacle’s last snapshot on one Rangers game sat at 1.355, so DraftKings and FanDuel carried that label between them.

Step 4: De-vig to get a probability

Raw prices are not probabilities. They sum past 100% by the book’s margin, 2.06% on average for Pinnacle across this sample. Proportional de-vig is the simplest of the three methods in our no-vig odds guide and it is fine for a two-way market.

def devig(prices):
    """Proportional de-vig of a two-way market: implied probability per side."""
    q = {side: 1.0 / price for side, price in prices.items()}
    total = sum(q.values())
    return {side: value / total for side, value in q.items()}

Step 5: Benter’s equation in Python

Here is the formula from the paper, in his notation. For a race with entrants 1 to N, the combined win probability of entrant i is

c_i = exp(alpha * f_i + beta * pi_i) / sum_j exp(alpha * f_j + beta * pi_j)

where f_i is the log of your out-of-sample model probability and pi_i is the log of the market’s implied probability. Benter’s own parenthetical explains the logs: “Natural log of probability is used rather than probability as this transformation provides a better fit.” Blending odds behaves better than blending probabilities, because a move from 2% to 4% matters more than a move from 50% to 52%.

A moneyline has two outcomes, so the sum runs over two terms.

def combined(p_model, p_market, alpha, beta):
    """Benter's equation (1) for a two-outcome market.

    exp(alpha * log p_model + beta * log p_market), normalised across the sides.
    The logs are the whole trick: it blends odds, not probabilities.
    """
    top = alpha * math.log(p_model) + beta * math.log(p_market)
    bot = alpha * math.log(1 - p_model) + beta * math.log(1 - p_market)
    shift = max(top, bot)
    a, b = math.exp(top - shift), math.exp(bot - shift)
    return a / (a + b)


def log_likelihood(rows, alpha, beta):
    total = 0.0
    for r in rows:
        p = combined(r["model"], r["market"], alpha, beta)
        p = min(max(p, 1e-9), 1 - 1e-9)
        total += math.log(p if r["won"] else 1 - p)
    return total


def fit_alpha_beta(rows):
    """Coarse-to-fine grid search for the (alpha, beta) that maximise likelihood."""
    lo, hi = -2.0, 4.0     # a negative weight means that forecaster is worse than noise
    best = (0.0, 0.0, -1e18)
    step, span_a, span_b = 0.5, (lo, hi), (lo, hi)
    for _ in range(5):
        alpha = span_a[0]
        while alpha <= span_a[1] + 1e-9:
            beta = span_b[0]
            while beta <= span_b[1] + 1e-9:
                ll = log_likelihood(rows, alpha, beta)
                if ll > best[2]:
                    best = (alpha, beta, ll)
                beta += step
            alpha += step
        span_a = (max(lo, best[0] - step), min(hi, best[0] + step))
        span_b = (max(lo, best[1] - step), min(hi, best[1] + step))
        step /= 4
    return best

Alpha and beta are what you came for. Benter: “The estimated values of alpha and beta can be interpreted roughly as the relative correctness of the model’s and the public’s estimates. The greater the value of alpha, the better the model.”

Step 6: Score it with pseudo-R² and the gain

The score is a likelihood ratio against a forecaster with no information. Benter uses the version from Bolton and Chapman (1986); the two-outcome case below compares against a coin flip, so 0 means you know nothing and 1 means you called every game.

The number that decides your model’s fate is his equation (4): the R² of the combined model minus the R² of the market on its own. Fit alpha and beta on one set of games and score them on another, or the two free parameters will manufacture a gain out of noise. Benter is explicit about this, and it is the single easiest way to fool yourself here.

def solo_log_likelihood(rows, key):
    total = 0.0
    for r in rows:
        p = min(max(r[key], 1e-9), 1 - 1e-9)
        total += math.log(p if r["won"] else 1 - p)
    return total


def pseudo_r2(log_lik, n):
    """1 minus the ratio to a coin-flip forecaster. 0 is no skill, 1 is perfect."""
    return 1 - log_lik / (n * math.log(0.5))


def delta_r2(rows, folds=5):
    """Benter's test, cross-validated: what the model adds on top of the market.

    Fit alpha and beta on four fifths of the fixtures, score the held-out fifth,
    and compare the combined forecast against the market on its own. Fitting and
    scoring on the same rows inflates this, which is the trap Benter warns about.
    """
    order = list(range(len(rows)))
    parts = [order[i::folds] for i in range(folds)]
    ll_combined = ll_market = 0.0
    weights = []
    for f in range(folds):
        held_out = {i for i in parts[f]}
        test = [rows[i] for i in parts[f]]
        train = [rows[i] for i in order if i not in held_out]
        alpha, beta, _ = fit_alpha_beta(train)
        weights.append((alpha, beta))
        ll_combined += log_likelihood(test, alpha, beta)
        ll_market += solo_log_likelihood(test, "market")
    n = len(rows)
    return pseudo_r2(ll_combined, n) - pseudo_r2(ll_market, n), weights

Step 7: Run it on real games

Two functions left: one that assembles the rows, and one standing in for your model. The stand-in is the de-vigged consensus of DraftKings and FanDuel, which is a real forecaster with real opinions and a good test case, since it is exactly the kind of number people treat as ground truth.

def build_rows(fixtures, your_model):
    """One row per gradeable fixture: your probability, the market's, and what happened."""
    rows = []
    for fixture in fixtures:
        snaps = moneyline_snapshots(fixture)
        time.sleep(4.6)                          # /historical-odds is slow, respect it
        if "pinnacle" not in snaps:
            continue
        winner = outcome_label(snaps)
        if winner is None:
            continue
        closers = closing_prices(snaps["pinnacle"], moment(fixture["startTime"]))
        if not closers:
            continue
        market = devig(closers)[SIDE_1]
        model = your_model(fixture, snaps)
        if model is None:
            continue
        rows.append({"market": market, "model": model, "won": winner == SIDE_1,
                     "fixture": f'{fixture["participant1Name"]} v {fixture["participant2Name"]}'})
    return rows


def retail_consensus(fixture, snaps):
    """Stand-in model: what DraftKings and FanDuel think, de-vigged and averaged."""
    kickoff = moment(fixture["startTime"])
    probs = []
    for book in ("draftkings", "fanduel"):
        if book not in snaps:
            continue
        closers = closing_prices(snaps[book], kickoff)
        if closers:
            probs.append(devig(closers)[SIDE_1])
    return sum(probs) / len(probs) if probs else None

season = finished_fixtures(13, "MLB", date(2026, 7, 10), date(2026, 7, 29))
rows = build_rows(season, retail_consensus)
n = len(rows)
print(f"{n} gradeable fixtures")
print("market  R2 = %.4f" % pseudo_r2(solo_log_likelihood(rows, "market"), n))
print("model   R2 = %.4f" % pseudo_r2(solo_log_likelihood(rows, "model"), n))
alpha, beta, _ = fit_alpha_beta(rows)
print("alpha = %.2f   beta = %.2f" % (alpha, beta))
gain, _ = delta_r2(rows)
print("delta R2 = %+.4f" % gain)

Swap retail_consensus for anything that returns a probability. A Poisson fit, an Elo rating, a gradient-boosted model, the power ratings we built from closing lines, or a number you typed in by hand. The harness does not care where the probability came from.

What 176 MLB games say

Sport 13, tournament MLB, 10 to 28 July 2026. 191 finished fixtures pulled, 176 gradeable from the price collapse, and every one of those carried both a Pinnacle close and a retail close. Pinnacle is the market here: 2.06% average margin on this sample, the highest limits in the sport, and the book the rest of the field follows.

Forecaster	Alone (R²)	alpha	beta	Gain over the market (cross-validated)
Pinnacle close (the market)	+0.0043	n/a	n/a	n/a
DraftKings + FanDuel consensus	+0.0050	2.26	-1.59	-0.0517
Pinnacle’s own opening price	+0.0070	0.87	-0.12	-0.0621
Control: market plus a 5% peek at the result	+0.0758	4.00	-2.00	+0.2396

Read the first column first. The three forecasters score +0.0043, +0.0050 and +0.0070 standing alone, which on 176 games is the same number three times over. One opinion, quoted three ways.

The last column settles it. The retail consensus scores -0.0517, with a bootstrap interval of -0.0769 to -0.0286 that never touches zero. Blending DraftKings and FanDuel into Pinnacle’s price makes the forecast measurably worse.

A negative gain is not your model betting against you. It is the fit spending two free parameters on nothing and paying for it out of sample, and you can watch it happen in the fold-by-fold weights: (2.28, -2.0), (2.49, -1.69), (-1.9, 2.32), (3.83, -2.0), (-2.0, 2.22). Alpha and beta land on opposite sides of zero in 5 folds out of 5 while their sum stays between 0.22 and 1.83. When your candidate duplicates the market, the likelihood surface flattens into a ridge where only the sum of the weights is identified, so the fitter picks a different arbitrary point on that ridge every fold. Opposite-signed weights that flip fold to fold are the fingerprint of a repackaged market price.

The raw prices say it more plainly. Across the 176 games, the de-vigged retail consensus sat an average of 0.49 percentage points from Pinnacle’s. Half a point. DraftKings and FanDuel are quoting the same opinion with a wider margin, so there is nothing to blend.

Pinnacle’s own opening price scores -0.0621 on top of Pinnacle’s close, and the open was not a trivial input: it sat 1.99 percentage points from the close on average, so the market moved. The close already contains the open plus everything that arrived after it. Textbook efficiency, and a second sanity check on the harness.

The bottom row is a control, and it exists so you can trust the three above it. It nudges the market’s own probability 5% of the way toward the actual result, making a forecaster with genuine inside information and no other skill. Same fitter, same two free parameters, same 176 games, and the test scores it at +0.2396. The harness finds skill when skill is there.

How small an edge would this sample catch?

Weaken the control and rerun it. This is the sensitivity check people skip, and it is what turns “we found nothing” into a statement with a number attached.

Control peeks at the result	Gain the test reports
1%	+0.0040
2%	+0.0626
3%	+0.1245
5%	+0.2396

A forecaster carrying 2% genuine information shows up loudly on 176 games. One carrying 1% vanishes into the noise. So the honest verdict on the retail consensus is that it holds less than roughly 1% independent information about an MLB moneyline, which is a different and more useful claim than “zero”.

Reading your own alpha and beta

What you see	What it means
Positive gain, both weights positive and stable	Two forecasters that each know something. Bet the combined number, never your raw one.
Weights flip sign fold to fold, gain zero or negative	Your model is a repackaged market price. Find an input the market cannot see.
Alpha large, beta near zero	Suspect a leak. Confirm your model never touched anything from after kick-off, including the closing price itself.
Alpha stable and negative	Genuinely anti-correlated. Check for a flipped side before you celebrate a contrarian edge.
Gain swings sign across folds and the control barely registers	Not enough games. Benter used 2,313 races.

Row two applies to the study above, and row five is the reason the control matters. Without it you cannot tell an uninformative model from an underpowered sample.

Where this test lies to you

Four honest limits, because a screening test you trust blindly is worse than none.

Sample size. 176 games resolves a 2% edge and misses a 1% one, per the control curve above. Benter had 2,313 races for a reason. Bootstrap your own gain before you act on it, and resample the held-out predictions rather than re-running cross-validation on resampled rows, or a duplicated fixture lands in the training and test folds at once and hands you a friendlier interval than you earned.

Pseudo-R² does not travel between sports. The metric is bounded by how much spread the market’s probabilities have. Baseball moneylines sit between roughly 0.35 and 0.75, so the market itself only scores +0.0043 here, while Benter’s .12 came from twelve-horse fields where the favourite is 0.40 and the tail is 0.005. Soccer 1X2 and outright markets score higher for the same reason. Compare models within one sport and one market.

The labels can be wrong. The collapse rule reads the last snapshot the feed recorded, not an official result. A game that turns over after the final snapshot gets mislabelled. Requiring cross-book agreement and dropping the 14 fixtures that never resolved handles most of it, and spot-checking against Polymarket’s settled 1 and 1000 handles the rest. Grade a stats provider against a sample of your labels before you trust them at scale, using the externalProviders IDs on every fixture object.

Information is not profit. A positive gain means your model knows something. Turning that into money still needs a price you can actually get and a size the book will accept. Our guide to the limit field covers what is on offer at the top of the market, and the Kelly walkthrough covers how much of it to take.

What to do with a model that passes

Bet the combined probability, not your raw one. That is the practical payoff of the second stage and the reason Benter spends pages on it: the raw model estimate is biased, consistently toward being too far from the market, so the advantage you compute from it is inflated. Feed the combined number into an expected value calculation instead, which our EV and closing line value guide sets up, then size it against real limits.

Then keep the harness running. Refit alpha and beta every month. A model whose alpha decays toward zero is a model the market has caught up with, and you will see it in the weights months before you see it in the bankroll.

Get the data

Everything above runs on the free tier: 383 bookmakers including Pinnacle and the sharp Asian books, 69 sports, and full historical price snapshots that competitors put behind a paywall. Grab a free API key and screen your model this afternoon.

Stop guessing whether your model is any good. Measure what it adds.

FAQ

Can I test a betting model without a results feed?

Yes. /historical-odds keeps recording after kick-off, so the winner’s price collapses toward 1.00 and the final snapshot works as a result label. Require every bookmaker on the fixture to agree on which side collapsed, and drop the games that never resolve.

What counts as a good gain over the market?

Benter reported .0178 for one of his models and .0090 for a later out-of-sample run on 2,313 races, and called the first enough for significant profits. Treat those as reference points rather than thresholds, since the scale depends on the sport and the market. Anything indistinguishable from zero across folds is a fail.

Why blend in log space instead of averaging the probabilities?

Benter’s paper gives the reason in a parenthesis: the log transformation “provides a better fit”. Averaging treats a 2% to 4% disagreement the same as 50% to 52%. Log space treats the first as the bigger one, which matches how prices behave.

Does this work on soccer 1X2 and other three-way markets?

The formula in the paper is already N-way. Replace the two-term normaliser in combined with a sum over every outcome and swap the coin-flip baseline in pseudo_r2 for log(1/N). Soccer 1X2 gives a wider spread of probabilities than a baseball moneyline, so the scores come out larger.

My model passed the test. Am I going to make money?

Not necessarily. The test measures information, and information gets converted into profit only if you can get on at a price better than the combined fair number, at a size worth the effort. Check the limit field and the exchange depth ladders before you count anything.