How to Detect Cheaters

Referring back to the Blitz games I played on chessdotcom - approx. 2000+ games. I independently and objectively measured the 'quality of the game' played in ECO rating terms, this took a very long time - I was very puzzled as to why I hadn't improved one iota over a 2-3 years period. What I discovered was a surprise.

Fitting the game data into a frequency v rating chart, instead of an expected mighty peak at my chessdotcom rating what I got was a gently rolling hill roughly peaking at my game rating +/- 100. The extremes were about 20 games rated 2800+ and roughly the same number 1000-.

My conclusion (in this context) - based on a sample of one - was that at my level, roughly 1300 to 1600 (~1450 classical OTB), people play with a very high level of game quality variance, such that a single game can never be utilised as an indicator of cheating.

Of course I would expect the higher your level, considerably less variance would be displayed and consequently the easier - fewer number of games required - to detect cheating using statistical techniques.

Note the analysis ignored game results, a very different way of looking at things.

Another interesting effect - albeit a bit more dodgy, as sample sizes were heavily biased toward Blitz - was that the longer the duration of the game the tighter was the variance and the more to the right the peak occurred. Effect was not strong but was expected. There was also little difference between playing White or Black - maybe +50 for the former.

All of this analysis has lead me to mainly play a board range of old chess computers nowadays - even less variance and more shifting to the right! - the thinking being that the higher the quality of games played the more likely the improvement for the time spent. Jury is still out on this experiment but I will give it a year - it's not the only change I have made.

@AlexiHarvey Sounds interesting. But what method did you use to measure the quality of the games?

In my opinion you detect a cheater by their lack of short term plans. Bots don't really build a position to trap a piece or to prevent a king from getting castled. They plan for the worst case scenarios?
https://courses.grainger.illinois.edu/cs440/fa2022/lectures/games.html

With self-play a bot can play best move vs best reply, to build that best dreamed up line. But when it's not self-play the best move is not the best short term plan. You don't see a chess engine going for a fried liver attack. Humans have aims and goals that bots do not.

When a chess piece looks trapped, the human player will go for it, but the bot will not. The bot knows the piece is not trapped or it's poisoned. Humans don't play speed chess at the depths that bots do. Any experienced player knows when they are being out played by a bot. The choices of the moves to play are explainable by every player. You play a move for a reason. The chess moves don't explain why they moved there, but humans can explain why they move there. Bot's don't mouse slip. Not many bot's approach the game the same way humans can do it.

The funniest part is seeing players leaving the chessboard on their move. Their green dot beside their user name turns off, but not the green dot that's logged into the site. Making an error purposely to break the streak of good moves is a laugh too. I also get a laugh when a player realises they played too fast, and then solves the problem to fast. It takes time for an engine to solve. So multiply that time factor exponentially to pretend a human is solving it.

Bots need to slow down to human speeds to be hard to detect.

When you think a player used assistance, ask them for a proof-of-work. The chess notation does not explain the aims or speed of the move. The performance trend of a player is also looked at. I use this to give me some insight to how will the human or bot is performing. Try out playing a bot and see the performance and then play a few humans to see the difference. I think it's a useful feature but cannot be used as proof that the player used assistance. Every chess player has a profile. It only takes a few games to start seeing the performance of a player.
https://chrome.google.com/webstore/detail/lichess-opponent-form/lipplpkgbnhdfdchoibgafjdblpjdkpi

Try to compare the bot graph slopes (performance trends) to human graph slope performances. Players that use assistance can probably make a slope graph, like the one in the link, but do they really want to be detected?
https://srconstantin.wordpress.com/2017/01/28/performance-trends-in-ai/

I say we should send in the Spanish Inquisition to detect cheaters.. after all, nobody expects the Spanish Inquisition!

Chess games probably don't need to go through a Turing test, but something like it (using human statistics) might help detect bot use. Imagine playing chess and having to answer a Captcha after every move. The Captcha would be asking you the aim of your move:

1. Attack;
2. Capture;
3. Control the centre;
4. Defend;
5. Develop a piece;
6. Improve position;
7. All the above;
8. None of the above.

Each move can be explained by the chess player. Especially in slow time controls. If they guess wrong, then that alerts a bot detection system. To pass the test, the player needs to have 90% of the aims correct. If not the game is re-analysed by other means.

https://www.semanticscholar.org/paper/Embedded-non-interactive-CAPTCHA-for-Fischer-Random-McDaniel-Yampolskiy/747f7ba6c3307cf6a34ade4b816273020466212c

@Toscani said in #35:

[...]Imagine playing chess and having to answer a Captcha after every move.[...]

I think I'd give up online chess if I was required to that!

@PeynirSever2134 said in #1:

I have realized that the number of cheaters on Lichess has increased in the last few weeks. And I'm creating this forum topic to take other's advices to stop them.
Bro that's up to lichess why do you want to do figure it out?

@Brian-E said in #32:

@AlexiHarvey Sounds interesting. But what method did you use to measure the quality of the games?

I used the LucasChess Game Analysis - you analyse relative to an engine and get cumulative results for each player, each game stage (middle game & endgame) and overall. Not sure of the exact details but it's clearly anchored to the Stockfish engine strength utilising a centi-pawn loss like metric. Scaling is a bit funny but relative values should be reasonable - it's essentially the same as the % accuracy metrics but scaled relative to the engine's ELO rating, I find the elo rating scaling more meaningful that % differences. Certainly better than anything I could produce but more importantly objective.

I believe chessdotcom have produced or are developing something similar - i.e. instead of an accuracy % you get a ELO game rating measure. With LucasChess you have more direct control of the analysis, i.e. analysis time per move, fixed ply depth of search, % change in evaluation etc. I used a fairly simple analysis, all moves for 10 seconds each - i.e. a 60 move game would take ~20mins to analyse - using a reasonably powerful PC running on two cores. In retrospect it might have been wiser to have used the fixed depth method of say 6 to 8 ply (most engines with fixed ply limits will actually look an additional 2 or 3 ply deeper for tactical elements and during the endgame stage).

Also note with LucasChess can take all this data and produce puzzles based on your own games - which was the main reason for all this effort - at my level the process produced around 5 to 10 puzzles per game where a move differed by at least 0.5 cpl from the engine's - I ended up with two - White and Black - huge puzzle pgn files, approximately 500Kb each.

All this is essentially what AimChess aims to do for you, for a price!

Im getting SMASHED, lot of "closed accounts" in the zillions I've reported, f*** this ****

I thought that too at first, being hammered by 800 players with 98% accuracy when their games on average are at 65% accuracy.
I then looked at their move time and rarely could find the fated constant 3 seconds moves.
Then I looked at my own games and noticed that I too have some good games with awesome accuracy and some with terrible accuracy.
Very often I test new openings or strategies and depending on my opponent it either goes well, or terribly wrong :)

Now I take this as a challenge to improve myself instead of blaming anything and everything for a couple of lost games...