I ran an expriment to build a ML model to determine the chance that someone used an engine in a game.
The problem is that to study cheating you need games where you know who cheated, ideally move by move. I don't think anyone really has many of these A banned account on a platform isn't proof, and it doesn't tell you which moves were assisted.
As a workaround, I generated 7000 games myself as so:
- The "humans" are Maia-3, a neural network trained on lichess games to play like a person of a given rating. It tries to play the move a human at a given rating would actually play, including blunders.
- A "cheater" is the same Maia bot, except before each move it flips a weighted coin and sometimes asks stockfish instead. Some cheated on 5% of their moves, some on every move. When they play a stockfish move, they don't always take the top move either.
- Ratings were 600 to 2600, and a third of games had no cheater at all.
In these games, I labelled every move as stockfish or human.
Results 1: one great move proves very little
Across all the games:
| | Matched Stockfish's #1 move | Average loss per move |
|---|
| Honest players' moves | 34% | 57 centipawns |
| Moves Stockfish chose for cheaters | 51% | 16 centipawns |
Honest players found the engine's top move 1/3 of the time. and because honest moves outnumber engine moves, 86% of moves that matched stockfish's top choice were played by the honest "human" not the cheater.
So "they found the only move" seems to be a weak argument. Humans find good moves all the time.
Results 2: light cheating is well hidden by normal variation
A player who uses stockfish on 1/5 moves only raises their top move rate by about 5%. Honest players swing between about 20% and 48% from game to game, so one game's accuracy can't really separate the two.
The signal only really appears when you combine lots of moves and several kinds of evidence (accuracy, mistakes, how they do in key positions, etc)
I trained a custom ML model on this (no LLMs). Cheaters who let stockfish pick over 75% of their moves were flagged by the model 94% of the time. Those who used it on a quarter of their moves or less were only flagged 33% of the time. Light cheaters are really hard to catch.
Results 3: strong honest players look like cheaters
This one makes a lot of intuitive sense A 2400 playing accurately looks a lot like a 1400 with an engine. Only 1% of honest players rated 600-1000 got flagged by the model, but 21% of those rated 2200-2600 did. My detector currently doesn't know the player's rating, which could be its biggest weakness. If you're accusing a strong player based on accuracy alone, keep this chart in mind. It's best not to accuse at all.
Other smaller findings
- Cheating works (even if you only do a little). In games with one cheater, the cheater won 66.5% and lost 23.5%. Most cheaters in the dataset used the engine on fewer than a quarter of their moves.
- A deeper engine was harder to catch. Cheaters using Stockfish at depth 18 to 20 were caught less often (47%) than those at depth 12 to 17 (61%). A deeper search can pick moves a quick engine check wouldn't rate as best.
- Humans are bad at pretty bad at this. In GM David Smerdon's study of 4,000+ players judging games for cheating, people were only right about 54% of the time. So my model is out performing humans.
Caveat
Everything here is measured on simulated players. Maia is a good imitation of Lichess players, but it isn't the same as having real data Real cheaters can also be smarter than a coin flip, eg only using the engine when losing. My next steps are to give the model the player's rating, look at patterns across many games rather than one and test on real games from closed accounts.
Disclosure: I built this tool into a free cheat-checking site, Chess Cheat Detector. The full technical write up for those who want more detail is on the blog . It is not affiliated with Lichess and only provides independent game analysis and doesn't replicate lichess' internal detection. Happy to answer questions.
I ran an expriment to build a ML model to determine the chance that someone used an engine in a game.
The problem is that to study cheating you need games where you know who cheated, ideally move by move. I don't think anyone really has many of these A banned account on a platform isn't proof, and it doesn't tell you which moves were assisted.
As a workaround, I generated 7000 games myself as so:
* The "humans" are [Maia-3](https://www.maiachess.com/), a neural network trained on lichess games to play like a person of a given rating. It tries to play the move a human at a given rating would actually play, including blunders.
* A "cheater" is the same Maia bot, except before each move it flips a weighted coin and sometimes asks stockfish instead. Some cheated on 5% of their moves, some on every move. When they play a stockfish move, they don't always take the top move either.
* Ratings were 600 to 2600, and a third of games had no cheater at all.
In these games, I labelled every move as stockfish or human.
# Results 1: one great move proves very little
Across all the games:
| |Matched Stockfish's #1 move|Average loss per move|
|---|---|---|
|Honest players' moves|34%|57 centipawns|
|Moves Stockfish chose for cheaters|51%|16 centipawns|
Honest players found the engine's top move 1/3 of the time. and because honest moves outnumber engine moves, 86% of moves that matched stockfish's top choice were played by the honest "human" not the cheater.
So "they found the only move" seems to be a weak argument. Humans find good moves all the time.
# Results 2: light cheating is well hidden by normal variation
A player who uses stockfish on 1/5 moves only raises their top move rate by about 5%. Honest players swing between about 20% and 48% from game to game, so one game's accuracy can't really separate the two.
The signal only really appears when you combine lots of moves and several kinds of evidence (accuracy, mistakes, how they do in key positions, etc)
I trained a custom ML model on this (no LLMs). Cheaters who let stockfish pick over 75% of their moves were flagged by the model 94% of the time. Those who used it on a quarter of their moves or less were only flagged 33% of the time. Light cheaters are really hard to catch.
# Results 3: strong honest players look like cheaters
This one makes a lot of intuitive sense A 2400 playing accurately looks a lot like a 1400 with an engine. Only 1% of honest players rated 600-1000 got flagged by the model, but 21% of those rated 2200-2600 did. My detector currently doesn't know the player's rating, which could be its biggest weakness. If you're accusing a strong player based on accuracy alone, keep this chart in mind. It's best not to accuse at all.
# Other smaller findings
* Cheating works (even if you only do a little). In games with one cheater, the cheater won 66.5% and lost 23.5%. Most cheaters in the dataset used the engine on fewer than a quarter of their moves.
* A deeper engine was harder to catch. Cheaters using Stockfish at depth 18 to 20 were caught less often (47%) than those at depth 12 to 17 (61%). A deeper search can pick moves a quick engine check wouldn't rate as best.
* Humans are bad at pretty bad at this. In GM David Smerdon's [study of 4,000+ players](https://www.chess.com/news/view/smerdon-research-catching-chess-cheaters) judging games for cheating, people were only right about 54% of the time. So my model is out performing humans.
# Caveat
Everything here is measured on simulated players. Maia is a good imitation of Lichess players, but it isn't the same as having real data Real cheaters can also be smarter than a coin flip, eg only using the engine when losing. My next steps are to give the model the player's rating, look at patterns across many games rather than one and test on real games from closed accounts.
Disclosure: I built this tool into a free cheat-checking site, [Chess Cheat Detector](https://chesscheatdetector.com). The full technical write up for those who want more detail is on the blog . It is not affiliated with Lichess and only provides independent game analysis and doesn't replicate lichess' internal detection. Happy to answer questions.
