Everyone says prediction markets are more accurate than sportsbooks. Polymarket predicted the 2024 election better than polls. Kalshi beat the consensus on rate cuts. But is this true for sports? Are Polymarket’s odds actually sharper than Pinnacle’s?<\/p>\n
You can’t answer that with anecdotes. You need hundreds of events, closing prices, actual outcomes, and a calibration curve. In this tutorial, you’ll build exactly that \u2014 a prediction market accuracy backtest<\/strong> using free historical data from OddsPapi.<\/p>\n Accuracy claims without data are marketing. To actually test whether Polymarket or Pinnacle is better calibrated, you need:<\/p>\n Getting this data is the hard part. Polymarket’s historical prices live on-chain (slow to extract). Pinnacle’s API is closed to the public. Most odds APIs charge extra for historical data.<\/p>\n OddsPapi gives you historical odds for free on every tier \u2014 including prediction market exchanges.<\/p>\n A Python script that:<\/p>\n The fixtures endpoint supports date filtering. We’ll fetch recently completed soccer fixtures.<\/p>\n The The Now we loop through completed fixtures, get closing prices from both Polymarket and Pinnacle, and record whether each predicted probability matched reality.<\/p>\n The Brier score measures prediction accuracy. Lower is better. A perfect predictor scores 0.0, random guessing scores 0.25 on binary outcomes.<\/p>\n For context on what these numbers mean:<\/p>\nWhy Historical Data Matters<\/h2>\n
\n
Getting Historical Data: The Options<\/h2>\n
\n\n
\n \nSource<\/th>\n Historical Data<\/th>\n Cost<\/th>\n Speed<\/th>\n<\/tr>\n<\/thead>\n \n Polymarket on-chain<\/td>\n Needs blockchain indexing<\/td>\n Free (gas costs)<\/td>\n Slow (GraphQL + parsing)<\/td>\n<\/tr>\n \n The Odds API<\/td>\n Available on paid plans only<\/td>\n $79+\/month<\/td>\n Fast<\/td>\n<\/tr>\n \n Scraping sportsbooks<\/td>\n Terms of service violation<\/td>\n Free (until you get banned)<\/td>\n Fragile<\/td>\n<\/tr>\n \n OddsPapi<\/td>\n \u2705 Free on all tiers<\/td>\n Free<\/td>\n Fast (REST API)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n What We’re Building<\/h2>\n
\n
Step 1: Fetch Completed Fixtures<\/h2>\n
import requests\nfrom datetime import datetime, timedelta\n\nAPI_KEY = \"YOUR_API_KEY\"\nBASE = \"https:\/\/api.oddspapi.io\/v4\"\n\ndef fetch_completed_fixtures(sport_id=10, days_back=7):\n \"\"\"Fetch completed fixtures from the last N days.\"\"\"\n end = datetime.utcnow()\n start = end - timedelta(days=days_back)\n\n params = {\n \"apiKey\": API_KEY,\n \"sportId\": sport_id,\n \"from\": start.strftime(\"%Y-%m-%dT00:00:00Z\"),\n \"to\": end.strftime(\"%Y-%m-%dT23:59:59Z\"),\n \"limit\": 300\n }\n r = requests.get(f\"{BASE}\/fixtures\", params=params, timeout=15)\n r.raise_for_status()\n\n # Filter to completed fixtures only (statusId 2 = completed)\n return [f for f in r.json() if f.get(\"statusId\") == 2]\n\nfixtures = fetch_completed_fixtures(days_back=7)\nprint(f\"Found {len(fixtures)} completed fixtures in last 7 days\")<\/code><\/pre>\nStep 2: Get Historical Odds (Closing Prices)<\/h2>\n
\/historical-odds<\/code> endpoint returns tick-by-tick odds history for up to 3 bookmakers per call. The closing price is the last recorded price before the match started.<\/p>\ndef get_closing_odds(fixture_id, bookmakers=[\"polymarket\", \"pinnacle\"]):\n \"\"\"Get closing (last pre-match) odds from historical endpoint.\"\"\"\n params = {\n \"apiKey\": API_KEY,\n \"fixtureId\": fixture_id,\n \"bookmakers\": \",\".join(bookmakers)\n }\n r = requests.get(f\"{BASE}\/historical-odds\", params=params, timeout=15)\n r.raise_for_status()\n return r.json()\n\ndef extract_closing_price(hist_data, slug, market_id, outcome_id):\n \"\"\"Extract the closing price from historical odds data.\"\"\"\n try:\n book = hist_data[\"bookmakerOdds\"][slug]\n market = book[\"markets\"][market_id]\n outcome = market[\"outcomes\"][outcome_id]\n # Historical data is sorted newest-first\n # Closing price = first entry (most recent before match)\n snapshots = outcome[\"players\"][\"0\"]\n if snapshots:\n return snapshots[0].get(\"price\")\n except (KeyError, IndexError):\n return None<\/code><\/pre>\nStep 3: Get Match Results<\/h2>\n
\/scores<\/code> endpoint returns period-by-period scores for completed fixtures.<\/p>\ndef get_result(fixture_id):\n \"\"\"Get match result (fulltime score).\"\"\"\n r = requests.get(f\"{BASE}\/scores\",\n params={\"apiKey\": API_KEY, \"fixtureId\": fixture_id},\n timeout=15)\n r.raise_for_status()\n data = r.json()\n periods = data.get(\"scores\", {}).get(\"periods\", {})\n ft = periods.get(\"fulltime\", periods.get(\"result\", {}))\n return {\n \"home_score\": ft.get(\"participant1Score\"),\n \"away_score\": ft.get(\"participant2Score\")\n }\n\ndef determine_1x2_outcome(result):\n \"\"\"Determine 1X2 outcome from score.\"\"\"\n h, a = result[\"home_score\"], result[\"away_score\"]\n if h is None or a is None:\n return None\n if h > a:\n return \"101\" # Home win\n elif h == a:\n return \"102\" # Draw\n else:\n return \"103\" # Away win<\/code><\/pre>\nStep 4: Build the Calibration Dataset<\/h2>\n
import pandas as pd\n\ndef build_calibration_data(fixtures, max_fixtures=200):\n \"\"\"Build dataset: closing probability vs actual outcome.\"\"\"\n records = []\n checked = 0\n\n for fx in fixtures[:max_fixtures]:\n fid = fx[\"fixtureId\"]\n name = f\"{fx['participant1Name']} vs {fx['participant2Name']}\"\n\n # Get historical odds\n try:\n hist = get_closing_odds(fid, [\"polymarket\", \"pinnacle\"])\n except Exception:\n continue\n\n # Check if polymarket data exists\n if \"polymarket\" not in hist.get(\"bookmakerOdds\", {}):\n continue\n\n # Get result\n try:\n result = get_result(fid)\n actual_outcome = determine_1x2_outcome(result)\n except Exception:\n continue\n\n if actual_outcome is None:\n continue\n\n checked += 1\n\n # Extract closing prices for each outcome\n for oid, label in [(\"101\", \"Home\"), (\"102\", \"Draw\"), (\"103\", \"Away\")]:\n poly_price = extract_closing_price(hist, \"polymarket\", \"101\", oid)\n pin_price = extract_closing_price(hist, \"pinnacle\", \"101\", oid)\n\n if poly_price and pin_price:\n poly_prob = 1 \/ poly_price\n pin_prob = 1 \/ pin_price\n occurred = 1 if oid == actual_outcome else 0\n\n records.append({\n \"match\": name,\n \"outcome\": label,\n \"poly_prob\": round(poly_prob, 4),\n \"pin_prob\": round(pin_prob, 4),\n \"occurred\": occurred\n })\n\n print(f\"Checked {checked} fixtures with Polymarket data\")\n return pd.DataFrame(records)\n\ndf = build_calibration_data(fixtures)\nprint(f\"\\n{len(df)} data points collected\")\nprint(df.head(10))<\/code><\/pre>\nStep 5: Calculate Brier Scores<\/h2>\n
import numpy as np\n\ndef brier_score(probs, outcomes):\n \"\"\"Calculate Brier score: mean squared error of predictions.\"\"\"\n return np.mean((np.array(probs) - np.array(outcomes)) ** 2)\n\nif len(df) > 0:\n poly_brier = brier_score(df[\"poly_prob\"], df[\"occurred\"])\n pin_brier = brier_score(df[\"pin_prob\"], df[\"occurred\"])\n\n print(f\"\\nBrier Scores (lower = better):\")\n print(f\" Polymarket: {poly_brier:.4f}\")\n print(f\" Pinnacle: {pin_brier:.4f}\")\n\n if poly_brier < pin_brier:\n print(f\"\\n Polymarket is better calibrated by {(pin_brier - poly_brier):.4f}\")\n else:\n print(f\"\\n Pinnacle is better calibrated by {(poly_brier - pin_brier):.4f}\")<\/code><\/pre>\n