These are the scoring rules in force now, version 12. Each of the eleven ingredients below is measured on the games the computer players played, and turned into marks from 0 to 1 on its own scale. Ingredients that answer the same question are averaged, the heavier ones counting for more, which gives six qualities, each out of 1. The qualities are then multiplied together, each raised to its weight, so one quality near zero pulls the whole score down instead of being outvoted: a game with nothing in it can look simple, varied and dramatic, and averaging let a deliberately dull test game score 0.64. No quality counts as lower than 0.02, so a game that fails one still differs from a game that fails three.
A weight is the share of the whole it carries; the weights of the ingredients add up to the weight of their quality.
Depth · 23.1% of the weight
Does thinking pay?
Does thinking longer winweight 23.1%
How much stronger a computer player gets each time it is allowed to think twice as long.
Why it counts. If more thought buys nothing, there is nothing to be good at. This is the biggest single ingredient, because skill paying is what makes a strategy game.
How it is worked out. The computer players that look ahead differ only in how many positions they may look at before each move. Each gets a rating from its results, and a straight line through the ratings, against how many times the thinking doubled, gives the rating points one doubling is worth. Chess measures about 120.
Its scale. Nothing at 8 rating points a doubling or less, rising in a straight line to full marks at 202 rating points a doubling or more.
Decisions · 20.9% of the weight
Is there something to think about on most turns?
Decisions that matterweight 11.0%
On a typical turn, how many of the moves on offer would cost you something.
Why it counts. Near none, every move is as good as any other and nothing needs thinking about; near all, there is one right move and the game is a puzzle. Of everything measured, it tells a broken game from a real one best.
How it is worked out. In positions taken from the games played, every legal move is scored the same way on every game, with the piece values the rules imply. This is the share of moves that come out clearly worse than the best one: worse by more than the cheapest piece is worth.
Its scale. Best in the middle: nothing at 0%, full marks from 29.5% to 65.3%, and nothing again at 100%.
Pieces that moveweight 5.5%
Whether the pieces on the board at the start ever move at all.
Why it counts. Use of the army looks at kinds of piece and cannot see five of one kind sitting at the back while a sixth does all the work. A designer testing a new game named that failure, so it is counted.
How it is worked out. The share of pieces on the board at the start that move at least once in a game, over the games played. When few move the game is dull, and when every single one must, the game may be a slog.
Its scale. Best in the middle: nothing at 0%, full marks from 51.4% to 88.5%, and nothing again at 100%.
Use of the armyweight 4.4%
Whether every kind of piece gets used, or some are scenery.
Why it counts. A piece nobody moves is a rule that adds nothing to the game.
How it is worked out. Counts how many moves each kind of piece made, and measures how evenly they are spread, from 0 (one kind makes every move) to 1 (every kind moves as often as the others), so a game with three kinds of piece compares with one with nine.
Its scale. Nothing at 0.735 or less, rising in a straight line to full marks at 1 or more.
Pacing · 16.5% of the weight
Are the choices a good size, and do games vary in length?
Choices per turnweight 8.2%
How many legal moves a player typically has.
Why it counts. Too few and there is nothing to decide; too many and a position is soup nobody can read.
How it is worked out. The number of legal moves, averaged over every turn of every game played.
Its scale. Best in the middle: nothing at 1.5 moves or less, full marks from 9.624 moves to 56.4 moves, and nothing again at 112.7 moves or more.
Length varietyweight 8.2%
Whether games vary in length or always run about the same.
Why it counts. A game that always ends on the same turn is running a script.
How it is worked out. How far game lengths spread around their average, divided by the average, so a long game and a short one compare fairly: 0 is every game the same length.
Its scale. Nothing at 0 or less, rising in a straight line to full marks at 0.971 or more.
Drama · 15.4% of the weight
Can a game turn round, and does a lead mean something?
Recovery from a mistakeweight 8.5%
Whether one bad move ends the game.
Why it counts. If a single slip is always fatal, the game is a memory test; if nothing you do matters, it is not a game. It is noisy, but it tells broken games from real ones better than almost anything else here.
How it is worked out. A bad move is forced into a real position from a game played, and the game is played on from there. This is the share of those trials the side that slipped still saved.
Its scale. Best in the middle: nothing at 0%, full marks from 25% to 75%, and nothing again at 100%.
Converting a leadweight 6.8%
Whether being ahead is worth something.
Why it counts. If a lead means nothing, the middle of a game is noise; if the first real lead always wins, the rest of the game is a formality.
How it is worked out. Among the won games where one side was clearly ahead in material part of the way through, the share that side went on to win. When a run has too few such games, or the computer players cannot judge who is ahead in this game, it is left out rather than scored as zero.
Its scale. Best in the middle: nothing at 50% or less, full marks from 61.6% to 93.6%, and nothing again at 100%.
Variety · 11.0% of the weight
Do two games go to different places?
Games that go different placesweight 11.0%
Whether each game goes somewhere the others did not.
Why it counts. If the same game keeps being replayed, there is little to explore.
How it is worked out. For each game where neither side moves at random, the positions it passed through are compared with the most similar other game. This is the share of positions the two did not have in common, averaged over the games.
Its scale. Nothing at 0%, rising in a straight line to full marks at 100%.
Balance · 13.2% of the weight
Is it fair to both sides, and do games finish?
Games that never finishweight 8.8%
How often a game is stopped by the turn limit or by a long run of moves with nothing taken, instead of reaching an ending its rules name.
Why it counts. A game that does not end is not a contest. Too many and the game is ruled out altogether.
How it is worked out. The share of games which are stopped this way, among those where neither side is the computer player that moves at random. A game drawn by repeating a position has reached an ending, so it does not count here.
Its scale. Full marks at 0%, falling in a straight line to nothing at 70% or more.
First-player advantageweight 4.4%
How much of an edge moving first is worth.
Why it counts. Both sides should be playing the same game. It counts for little, because it is the least certain thing measured: games differ on it by not much more than chance alone moves it.
How it is worked out. Only games where a computer player meets an exact copy of itself are read, so any difference comes from moving first. The first player's share of the points is compared with an even half.
Its scale. Full marks at 50% of the points, falling in a straight line to nothing at 75% of the points or more.
What rules a game out
Some failings are not marked down but rule a game out: its score still shows, and means nothing. A game is viable when it passes all of these.
- A computer player that looks ahead must take at least 55% of the points from one that moves at random.
- No more than 70% of games may be stopped by the turn limit or by a long run of moves with nothing taken.
- A player must have at least 1.5 legal moves on a typical turn.
- At least 20% of games must have a winner, rather than nearly all being drawn.
- Games must last at least 8 turns on average.
- Thinking twice as long must be worth at least 8 rating points.
- No more than 50% of games may be won inside 12 turns, which would mean an opening trap decides them.
Where a test is about how games end, it is read twice, on the games of the weaker computer players and on those of the stronger ones, and the half that does worse is the one that counts: one set of rules can be an opening trap for weak players and a dead shuffle for strong ones, and averaged together each failing would hide the other. One more test asks whether the strongest computer players win from one side nearly every time, whichever of them sits there: then the side decides the game, not the play.