Predict

01What it does

You give it an event and, optionally, your own exposure. It returns a question, mutually exclusive outcomes with probabilities and intervals, a rationale per outcome, cited sources, and — if you ask — actions worth taking under each one. The API takes answers as a JSON contract, which is the whole point: forecasts you can put in a pipeline, not a chat window.

02What's inside

• Four estimators — direct, log-probability, self-consistency sampling and a Bayesian prior — pooled into one distribution
• Search grounding through Tavily, with citations attached to each outcome
• Streaming playground that shows retrieval, estimation and pooling as they happen
• A backtesting harness: multi-class Brier, log loss, expected calibration error, and paired bootstrap intervals against the market

03How it works

Estimators are plugins behind a registry, so adding a method is one file plus a preset entry, and the contract test suite covers it automatically. Backtests run with the clock rewound: searches are date-bounded at the forecast date and undated hits are dropped. Search responses are cached so a re-run is identical and costs nothing.

04Measured against Polymarket

Unit tests prove the arithmetic. They cannot prove the probabilities are any good, so backend/eval forecasts Polymarket questions and scores them against what actually happened, beside the market’s own price — a liquid prediction market is a strong, well-calibrated crowd baseline, which makes "as good as the market" the bar rather than "better than a coin".

• Two kinds of run. A backtest takes resolved questions, forecasts each as of a date before it settled (seven days by default) and scores it against the price that day. A live run takes open markets, forecasts them at today’s price, and scores them as they settle — you can run against current Polymarket, not only the past
• Nothing can leak in a live run, because the answer does not exist yet. That is why the honest track record will come from live runs, and why the backtest is for comparing engine versions quickly
• Datasets come from Polymarket’s public Gamma and CLOB APIs, no key: one question per event, negRisk events become multi-outcome with the long tail folded into "Other", everything else contributes its highest-volume Yes/No market. $100k volume floor, skip markets the crowd had already called above 97%, esports and price ladders excluded, no category allowed more than 30% of the set
• Leakage guard for backtests: search is cut off at the as-of date, undated and later hits are dropped, forecast and wiki domains are blocked, and every search is traced. The model’s own memory can still leak an answer, which is the known weakness of any backtest
• Scoring: multi-class Brier, log loss, and expected calibration error with a reliability diagram — computed for the engine, the market, a uniform guess, a 50/50 engine-market blend and each estimator on its own. A paired bootstrap over per-question differences gives a 95% interval on engine minus market, and the engine only "beats" it when that whole interval sits below zero
• Where it currently stands: Brier 0.565 against the market’s 0.506 over 150 resolved questions, with the difference not statistically significant

05The estimators

Four run on every request, independently and in parallel, and a weighted logarithmic opinion pool combines them — p(outcome) is proportional to the product of each estimator’s probability raised to its weight. Agreement is rewarded, disagreement widens the intervals, and self-reported confidence is ignored because the model that was most sure of itself was the one that scored worst.

• direct — the heaviest weight (3.0). It asks the model for P(outcome) one outcome per call, so a competing outcome cannot anchor the answer; the prompt asks for a base rate first and an adjustment for the evidence after. Three repeats, pooled by median
• logprob — a one-token yes/no question per outcome, reading P(yes) off the first token’s log-probabilities summed over the top ten variants. A token is far more confident than the odds behind it, so the log-odds are halved. Skipped on yes/no questions, where it sat far from every other estimator
• sampling — self-consistency: sixteen free-text "single most likely outcome" answers at temperature 1, each mapped onto the outcome list, shares Dirichlet-smoothed and counted as votes. Mutually exclusive sets only
• bayes — priors times likelihood ratios. A prior per outcome, anchored on a quoted market price when the evidence contains one (that evidence is then excluded from the update, so the price is not counted twice), then Jev rates every evidence-outcome pair on a five-level scale worth a likelihood ratio of ¼, ½, 1, 2 or 4. Correlated evidence is discounted: one vote per source, combined as the sum over the square root of the count, capped at a 4x move overall
• jev — the decision model asked straight for the distribution. In backtests it was overconfident (its 90%-plus picks came true 45% of the time) and scored worse than direct in every category, so its weight is zero and it does not run as an estimator. It is right there in the results as a skipped plugin, not hidden

06Where the Jev API is used

Jev is TypeSafe’s calibrated decision model, called through its SDK as typed questions over one state: a Choice over the outcome list, or a Noul per outcome when the set is not exclusive, with a short client timeout and a single retry. Forecasting is the one job it lost to its own backtest results. It stays on the critical path where the work is judgement over text rather than a probability: rating each piece of evidence for the Bayesian update, mapping the self-consistency samples onto the outcome list, and scoring how useful each proposed action would be if a given outcome happened — which is what turns a list of actions into hedging ones.