The thinking behind it

Questions that can be settled

A question you can forecast says exactly what counts, by when, and which source will settle it, so that once the date passes, everyone agrees on the answer.

“Will electric cars take over?” can't be scored: nobody agrees what taking over means, or when to check. Forecasting research calls the fix resolution criteria.

Pinning a question down also changes its answer. The date, the threshold and what counts as an exception each move the chance, so they have to be settled before anyone estimates it.

Before any research starts, GameOut works the question out with you: what counts, the date and the source. You approve it, and every later step forecasts that exact wording.

The same question, before and after it's defined.

Base rates: start from history

A base rate is how often things like this have happened before. Starting from it, before the details of this case, is one of the most reliable habits in forecasting.

The psychologists Daniel Kahneman and Amos Tversky called it the outside view. Left alone, people start from the story in front of them, the inside view, and forget how stories like it usually end. Big projects run late and over budget however confident the plan.

The craft is picking the reference class: the past cases that are truly like this one. Then the details of this case move the estimate up or down from there.

Every forecast starts with the history: what has already happened, where things stand now, and how often similar events have gone each way. The estimates start there and move only for reasons that are written down.

Out of 24 similar past cases, 8 went this way. The estimate starts near one in three, then the details of this case move it.

Start from how often it happens

Before reading the news about a question, list a few groups of similar past situations and how often the thing happened in each, then a few specific past cases that look like this one and how each ended.

The forecasting researchers Kesten Green and J. Scott Armstrong called the second step structured analogies: name the look-alike cases, rate how similar each one is, and note how it turned out. In their studies of conflicts, experts who worked this way forecast better than experts left to their own judgment.

One reference class is rarely enough, because a case belongs to several groups at once. Comparing two or three shows how much the starting point depends on the choice of group. Writing down what makes this case different shows how far, and in which direction, the evidence should move it.

Right after looking up what has already happened, GameOut searches for two or three groups of similar past situations and a handful of look-alike cases, each with its source. A rate no source states stays unknown. Every forecaster states where it started, which cases it leaned on and why it moved, and one that ends far from every group without naming new evidence is questioned in the review.

Past stadium projects like a new one, how alike each is, and how each ended: the starting point before the details of this one.

Decomposition: break it down

Decomposition splits a hard question into the smaller questions that decide it, each easier to research and estimate than the whole.

Nobody can judge “Will this bill become law this year?” in one go. Split it, and each part has its own evidence and its own history: will it pass the House, will it pass the Senate, will it be signed in time?

Splitting takes care. The parts must cover every route to the answer, and parts that depend on each other have to be judged together, not multiplied as if they were separate.

Each question is broken into sub-questions, and those into smaller ones that can be checked against evidence. Agents research the branches side by side, then the parts are put back together.

A bill's chance of becoming law, split into the steps it has to clear and the facts each step turns on.

Fermi estimates: unknowns from knowns

A Fermi estimate works out a number nobody knows from a chain of smaller numbers that can be estimated, the way the physicist Enrico Fermi famously did.

Fermi asked his students how many piano tuners worked in Chicago. Nobody knew. But anyone could estimate the city's people, the homes with a piano, how often a piano is tuned and how many tunings a tuner can do in a year. Multiply and divide, and you land close to the real figure.

It works because the errors in the separate guesses tend to cancel out rather than pile up, and because every step is in the open, where a wrong one can be spotted.

Hard numbers are built the same way: from a base rate and the factors that move it, each factor written down with its reason, so anyone can see which assumption matters most.

Fermi's piano tuners: five rough guesses, multiplied out. Each one can be checked on its own.

Game theory trees

A game tree maps a situation as a series of choices: who decides, what they can do, what's left to chance, and how each path ends for everyone involved.

Game theory studies choices that depend on other people's choices. In a tree, a square is a player deciding, a circle is chance (a ruling, a vote count, the weather), and each ending lists what every player gets.

Reading the tree from its endings back to the start, called backward induction, shows which moves each player would actually make. That turns every possible path into a likely one.

When an outcome depends on people who want different things, GameOut maps the decision-makers and their incentives, draws the tree of paths to each outcome and marks the likeliest one.

A union decides whether to strike, knowing the company's reply. If the court case is a coin flip, holding firm is worth 1 to the company on average and raising its offer is worth 2, so it raises. Knowing that, the union strikes.

Where the bargain lands, and who can block it

When an outcome is negotiated, place every player on one scale for the issue, weight each by its clout and by how much it cares, and the weighted middle is a good first guess at where the deal ends up.

The political scientist Bruce Bueno de Mesquita built this into his expected-utility model of bargaining: a player who is powerful but doesn't care much pulls less than a smaller one that cares about nothing else. The weighted middle often lands close to the negotiated result, which is why the model has been used to study everything from budgets to arms talks.

A second idea comes from George Tsebelis: veto players. Anyone who can stop the outcome alone, such as a chamber, a court or a party to the deal, has to be won over. The more veto players there are, and the further apart they stand, the harder it is to move away from the way things are now.

For a yes or no question, the step that maps the players also places each one on a 0 to 100 scale for the question's main issue and says who can block it. Code works out the weighted middle and the veto players; a player whose position is unknown is left out, never placed in the middle. The forecasters and the reviewer read both as evidence about the deal, not as a chance, and the full file shows the scale.

Three made-up players on a city park budget. The mayor cares most, so the weighted middle sits nearer to her than the plain average does.

Why deals fail, and which signals to trust

Talk costs nothing, so it proves little. Votes, money, signed contracts and troop movements cost something, so they show what a player really intends.

The political scientist James Fearon gave three reasons why sides fail to reach a deal both would prefer: each hides what it knows and may be bluffing; a promise can't be enforced, so neither trusts the other to keep it; and some stakes can't be split. Economists call the difference between words and deeds cheap talk versus costly signals.

Thomas Schelling showed that bargains cluster at hard deadlines and obvious split points, like a round number or halfway. Leaders who commit in public pay a price at home for backing down, which Fearon called audience costs, and players who meet again and again keep promises more easily. Where talks are normally kept secret, as in company mergers, silence is not a sign of no.

Each player gets a short checklist: its signals so far, each marked cheap or costly with its sources, whether its promises can be enforced, the deadline or split point that matters to it, and what it is probably bluffing about. The reviewer must raise one challenge from these ideas, such as "this rests on a statement that cost nothing" or "three players can each block this". The forecasters can reject it with their reasons, and it may not drag the forecast toward even odds without evidence.

A company threatens to leave town. The threat is talk until it signs a lease somewhere else.

War-gaming the players

Instead of asking an expert what each side will do, have someone play each side and let them actually make their moves against each other. How the role-play ends is a good guide to how the real conflict ends.

The forecasting researcher Kesten Green tested this on real conflicts, such as strikes, takeovers and diplomatic standoffs. People who played the parties and interacted picked the actual decision about 60% of the time. Experts on their own, game theorists, and people who only imagined each side's view without playing it out did little better than chance, at around 30%.

Playing it out works because the moves react to each other: an offer brings a reply, a threat brings a counter-move, and a deadline forces a choice. Repeating the game with separate groups helps too. The most common ending across many groups was right in most of the conflicts Green studied.

For a yes or no question with at least two players who make decisions, AI players each play one key player from a brief built from the research: its goals, limits, position, signals so far and deadline. They take turns making moves for a few rounds, a referee says how the game ended and which move decided it, and the game is played several times on its own. The forecasters and the reviewer read the result, such as how many of the games ended yes, as evidence beside the history and the tree, never as the chance itself. The full file shows every move.

A union and a company play out a pay dispute several times over. Most games end in a deal after the company raises its offer, so a deal is the likely ending.

Independent panels

Several forecasters who each estimate a question on their own, then have their answers combined, are usually more accurate than any one of them.

It's the wisdom of crowds, and it holds only while their errors are independent. Forecasters who see each other's numbers first make the same mistakes together. The classic one is anchoring: the first number heard pulls every later one toward it.

Variety matters as much as numbers. Forecasters who think differently cover each other’s blind spots; copies of one mind share them.

GameOut's forecasters run on AI models from rival labs, trained by different teams on different data. Each estimates alone, without seeing the others, and the estimates are combined with equal weight.

Five forecasters estimate on their own. The combined estimate weighs each of them equally.

Different starting angles

A panel does better when its members think differently. Giving each forecaster a different place to start makes their mistakes less alike, so more of them cancel out when the answers are combined.

The economist Scott Page put this into a formula, known as his diversity theorem: the combined estimate's error equals the members' average error minus how far apart they are. With the same skill, a panel that disagrees for good reasons beats one that agrees.

Philip Tetlock found the same in people. The best forecasters look at a question like a dragonfly's eye, through many lenses at once: what usually happens, how the process works, what the people involved want, what the numbers say. Starting from one lens and testing it against the others guards against seeing only what fits the first story.

Each forecaster leads with its own angle: history (how often this kind of thing happens and how look-alike cases ended), mechanics (the steps that must happen and who must act), the players (what each wants, the signals it has sent and who can block), or numbers and timing (data, polls, schedules and deadlines). Each still weighs all the evidence and says what its angle showed. No angle is told to doubt, and none pulls toward even odds. The full file shows each forecaster's angle.

Four made-up forecasters estimate the same count, each starting from a different angle. Their misses point different ways, so the combined estimate lands nearer the real count than most of them do.

The devil's advocate

A devil's advocate argues against an estimate on purpose, to find what its author missed before reality does.

The name comes from the Catholic Church, which appointed an official to argue against making someone a saint. Modern versions include red teams and the premortem: imagine the forecast turned out wrong, then explain why.

It counters confirmation bias, the habit of noticing only what fits. AI models need it even more than people do, because they tend to agree with whoever they are talking to.

A devil's advocate attacks each forecaster's estimate with the strongest case against it. Each forecaster answers the critique and may revise on the evidence, never just to split the difference.

After the critique, one forecaster revises and the other holds, because the case against didn't stand up to the evidence.

Evidence you can check

Every claim a forecast rests on should point to where it came from, so a reader can check it instead of trusting it.

Language models can state invented facts with total confidence. The defence is old-fashioned: sources, quoted and linked, and a separate check that each one really says what the forecast claims it says.

Hard data, such as polls, prices and official counts, is evidence that moves an estimate. It is never the answer by itself.

Before anything is published, the evidence is audited: every number links to the page it came from, and claims their source doesn't support are caught.

Each claim points to its source, and the one its source doesn't support is caught.

Probabilities, scored

A probability says how often calls made with that confidence come true. A good forecaster is calibrated: of all their 70% calls, about 70% happen.

That's why a forecast is a chance rather than a yes or no, and why it never says 0% or 100%: nothing about the future is certain.

Accuracy is measured with the Brier score: the square of the gap between the chance given and what happened, counted as 1 if it happened and 0 if not. Lower is better. A confident right call scores near 0, a confident wrong one near 1, and saying 50% every time scores 0.25.

Every forecast publishes one whole percent from 1% to 99% and takes a side. Once the outcome is known it gets its Brier score, and our public forecasts stay up, won or lost.

Brier scores for four calls. Lower is better.

Updating on news

Updating means moving a forecast as evidence arrives: by as much as the news is surprising, and no more.

This is Bayes' rule in plain words. News you'd expect to see anyway should barely move you; news that would be unlikely unless the outcome were coming should move you a lot.

People tend to get both wrong, brushing off news that cuts against them and overreacting to vivid headlines.

With Live, GameOut keeps reading the news after a forecast is published and checks whether each development should change the chance, until the question is decided.

Most news barely moves the forecast. One surprising development moves it a lot.

When nothing happens

For a question about whether something happens by a date, a quiet stretch is evidence too: with less time left and nothing new, the chance should usually ease down a little.

Researchers call one common mistake scope insensitivity: people, and language models, give about the same chance whether the window is a month or a year, though a longer window leaves more time for the thing to happen.

Good forecasters update often and in small steps, and ask how the chance is spread over time. Some things tend to happen early; many come late, the way deals close near their deadlines; some are settled on one day, like a vote. When part of the time passes with nothing new, the share of the chance that was due in it is gone.

Each forecaster also gives its chance for an earlier date about halfway to the deadline and says when it expects the event, if it comes; code checks that the earlier chance is never above the one for the deadline. With Live, when the checks find nothing new, the published chance eases along that timing in whole points, at most once a day, marked "Time passing". Real news takes over from the current number, questions settled on one day never drift, and the original forecast is the one scored.

A made-up forecast easing down on quiet days, then moving with one piece of news.

Questions people ask

What is a game theory decision tree?

A drawing of a situation as a sequence of choices. Squares are players deciding, circles are chance events, and each ending lists what every player gets. Working back from the endings shows which choices each player would rationally make, and so which path is likeliest.

What is the expected utility model of bargaining?

A way to forecast a negotiation, developed by Bruce Bueno de Mesquita. Each player is placed on a scale for the issue and weighted by its power and by how much it cares; the weighted middle of their positions is a first guess at where the deal lands.

What is the difference between cheap talk and a costly signal?

Cheap talk is a statement that costs nothing to make, so a bluffer can make it as easily as someone who means it. A costly signal, like a vote, money spent or a signed contract, is expensive to fake, so it says more about what a player will actually do.

What is Scott Page's diversity theorem?

A result from the economist Scott Page: a group's combined estimate is off by the members' average error minus how much their estimates differ. So when the members are equally skilled, the group that thinks more differently gets closer.

What is a war game or simulated interaction in forecasting?

A way to forecast a conflict by having people play each side and actually interact, making offers, threats and concessions in turn, instead of asking an expert what will happen. In Kesten Green's tests on real conflicts, it picked the actual decision about twice as often as unaided experts or game theorists.

What is a Fermi estimate?

A way to estimate a number nobody knows by breaking it into smaller numbers that can be estimated, then multiplying them out. It's named after the physicist Enrico Fermi, who was known for good answers to questions like how many piano tuners work in Chicago.

What is a base rate in forecasting?

How often events like the one in question have happened before. It's the starting point for an estimate, before the specific details of the case move it up or down.

Why use several AI models instead of one?

The standard score for probability forecasts: the square of the difference between the chance given and the outcome (1 if it happened, 0 if not). It runs from 0 to 1, and lower is better. Always saying 50% scores 0.25.

What is a Brier score?

One model's mistakes repeat every time you ask it. Models from different labs are trained differently and tend to make different mistakes, so when each estimates on its own, combining them cancels part of the error. It's the same reason a panel of independent forecasters beats one.

Why does a forecast never say 0% or 100%?

Because nothing about the future is certain, and a forecast that claims certainty and turns out wrong scores as badly as possible. GameOut publishes whole percents from 1% to 99%.

Can AI forecast the future?

Nobody can know the future. What AI can do is the work good forecasting takes, reading widely, following a method and keeping a record of every step, faster and at a larger scale than a person can. The result is a probability, not a certainty, and it gets scored once the outcome is known.

How is this different from asking a chatbot?

A chatbot is quick and good for almost anything, but it answers in one pass from what it remembers, and tends to agree with you. A GameOut forecast defines the question, starts from history, breaks it down, researches each part, has rival models estimate independently, then challenges and audits the result before it publishes one chance.