lichess.org
Donate

I investigated using ML to find cheaters

KMcGeoch

@gotwic said ^

Are you saying these bots are cheating!?

Funnily enough many LLMs are also guilty of cheating. Apparently some versions use Stockfish and others even play illegal moves. You could argue a bot using another bot is outside assistance in same way a human asking another human is also outside assistance.

@gotwic said [^](/forum/redirect/post/pRKp7Fs5) > Are you saying these bots are cheating!? Funnily enough many LLMs are also guilty of cheating. Apparently some versions use Stockfish and others even play illegal moves. You could argue a bot using another bot is outside assistance in same way a human asking another human is also outside assistance.
KnightMovesLikeJagga

@an7cplayingchess Thank you for flagging this! Are you able to share the analysis link (or DM me)? It is possible that this was missed by my system, but it's also possible that it was a false-positive by Lichess, or that Lichess used other games to deduce that white cheated here.

@Akavall I do take into account the relationship between features, but at the moment time control is ignored. The issue is that sometimes games uploaded contain rating and time control, and sometimes there are only moves and literally nothing else, I want to be able to support the full range of the spectrum. One though I had is that I could train different models, using the extra features (rating, time control, etc) when they are available, but fallback to a model that doesn't use them if they are not provided. This provides the best of both worlds, but it also increases complexity on my side!

@lizani A cool headline is that this system catches almost 80% of cheaters (where "cheater" = those who use 50% or more stockfish moves). But yeah. this is a condescend form of a much longer article -- if you want more detail, you could be interested in that! https://chesscheatdetector.com/blog/building-a-chess-cheat-detector This covers building the entire thing end-to-end, and lots of details on performance, features, ML modelling etc :)

@petri999 great idea! I'll look into this.

However problem with rating is that cheating moves rating that then influences assumptions made when performing cheating analysis.

@KMcGeoch Really interesting point. Obviously, I want to be able to determine cheating at any rating, and not have to wait until a player achieves GM status to make a good call. Therefore, perhaps a flag to do with uncertainty on the account could help? So if we know that they've played many games before and something substantial changes, then that could be a signal. But for completely fresh accounts, there isn't really much of a way to know whether they are just a super strong player or a stockfish-enabled player - unless they use stockfish consistently in a way that doesn't align with move timings. Your proposal is really interesting, I'm gonna give it some thought, but I might generate some games how you describe and see how it performs. I'll let you know!

@MagicAtom @challenger1986 Thank you for the kind words! I'll take a look into doing a blog

@an7cplayingchess Thank you for flagging this! Are you able to share the analysis link (or DM me)? It is possible that this was missed by my system, but it's also possible that it was a false-positive by Lichess, or that Lichess used other games to deduce that white cheated here. @Akavall I do take into account the relationship between features, but at the moment time control is ignored. The issue is that sometimes games uploaded contain rating and time control, and sometimes there are only moves and literally nothing else, I want to be able to support the full range of the spectrum. One though I had is that I could train different models, using the extra features (rating, time control, etc) when they are available, but fallback to a model that doesn't use them if they are not provided. This provides the best of both worlds, but it also increases complexity on my side! @lizani A cool headline is that this system catches almost 80% of cheaters (where "cheater" = those who use 50% or more stockfish moves). But yeah. this is a condescend form of a much longer article -- if you want more detail, you could be interested in that! https://chesscheatdetector.com/blog/building-a-chess-cheat-detector This covers building the entire thing end-to-end, and lots of details on performance, features, ML modelling etc :) @petri999 great idea! I'll look into this. > However problem with rating is that cheating moves rating that then influences assumptions made when performing cheating analysis. @KMcGeoch Really interesting point. Obviously, I want to be able to determine cheating at any rating, and not have to wait until a player achieves GM status to make a good call. Therefore, perhaps a flag to do with uncertainty on the account could help? So if we know that they've played many games before and something substantial changes, then that could be a signal. But for completely fresh accounts, there isn't really much of a way to know whether they are just a super strong player or a stockfish-enabled player - unless they use stockfish consistently in a way that doesn't align with move timings. Your proposal is really interesting, I'm gonna give it some thought, but I might generate some games how you describe and see how it performs. I'll let you know! @MagicAtom @challenger1986 Thank you for the kind words! I'll take a look into doing a blog
KMcGeoch

One thing definitely worth incorporating is that Leela Chess Zero (Lc0) measures sharpness by analyzing the engine's Win/Draw/Loss (WDL) probability outputs to quantify how volatile, complex, or risky a position is. So that could solve Capablanca issue I mentioned. It would also remove distortions from long endgames where there's only 1 likely outcome.

Rating is really awkward since stronger players average a lower centipawn loss but playing with a lower centipawn loss results in a higher rating.

One thing definitely worth incorporating is that Leela Chess Zero (Lc0) measures sharpness by analyzing the engine's Win/Draw/Loss (WDL) probability outputs to quantify how volatile, complex, or risky a position is. So that could solve Capablanca issue I mentioned. It would also remove distortions from long endgames where there's only 1 likely outcome. Rating is really awkward since stronger players average a lower centipawn loss but playing with a lower centipawn loss results in a higher rating.
KnightMovesLikeJagga

@KMcGeoch that sounds great thank you. I will look into that asap

@KMcGeoch that sounds great thank you. I will look into that asap
Cedur216

What does ML stand for in the first place

What does ML stand for in the first place
KnightMovesLikeJaggaedited

@Cedur216 ML is machine learning. Basically, "AI" or artificial intelligence. There are subtle differences that people in the space like to debate, but for the most part, ML can be considered the same as AI.

This is not an LLM (language model) though, it's a custom made model specifically for cheat detection :)

@Cedur216 ML is machine learning. Basically, "AI" or artificial intelligence. There are subtle differences that people in the space like to debate, but for the most part, ML can be considered the same as AI. This is not an LLM (language model) though, it's a custom made model specifically for cheat detection :)
eight_x_eightedited

This tool appears to be very poor. I tested it on games won by a significantly lower rated account that was later banned for fair play violations on chess dot com. Out of 6 such games this tool found nothing suspicious.
I see you're charging for a premium version, I think this is basically a scam.

This tool appears to be very poor. I tested it on games won by a significantly lower rated account that was later banned for fair play violations on chess dot com. Out of 6 such games this tool found nothing suspicious. I see you're charging for a premium version, I think this is basically a scam.
KnightMovesLikeJaggaedited

@eight_x_eight Very harsh words. I am not charging for anything, the site is free to use. The site allows people to enter their email if they are interested in a premium version, which is just a way of me determining how much time to spend on this. If there is interest, I may decide to build out those features, but I am not asking for anything from anyone. Signing up to be notified if I build those features is entirely voluntary and doesn't cost anything either. The free version has taken me hundreds of hours of work already. If it were a scam, it would be an extremely poor scam as I have zero revenue and a lot of costs so far.

As with any machine learning/predictive tool, there are both false positives and false negatives. A sample size of 6 from one account is not statistically significant data. You should be wary forming conclusions from such a small sample. Chess Dot Com banning it is not conclusive proof that they were cheating (overall) or that they were cheating in specifically those game either.

Also consider that chess.com has a lot more data than I am privy too - they can tell when you switch between apps, they can track move timings, they can look at the overall profile rather than just a single one-off game. This adds up to be a big difference. An individual prediction based only on one game's moves will always be less accurate than this - this is why I'm expanding out into full account-level predictions too.

The model is getting better every day and I hope to continue working on it. This is the first negative feedback I've had.

I am also unusually transparent in the quality of the model. This exact post you're replying to says the performance of the model in catching cheaters, and how I measure it. I go into even more detail on the blog.

@eight_x_eight Very harsh words. I am not charging for anything, the site is free to use. The site allows people to enter their email if they are interested in a premium version, which is just a way of me determining how much time to spend on this. If there is interest, I may decide to build out those features, but I am not asking for anything from anyone. Signing up to be notified if I build those features is entirely voluntary and doesn't cost anything either. The free version has taken me hundreds of hours of work already. If it were a scam, it would be an extremely poor scam as I have zero revenue and a lot of costs so far. As with any machine learning/predictive tool, there are both false positives and false negatives. A sample size of 6 from one account is not statistically significant data. You should be wary forming conclusions from such a small sample. Chess Dot Com banning it is not conclusive proof that they were cheating (overall) or that they were cheating in specifically those game either. Also consider that chess.com has a lot more data than I am privy too - they can tell when you switch between apps, they can track move timings, they can look at the overall profile rather than just a single one-off game. This adds up to be a big difference. An individual prediction based only on one game's moves will always be less accurate than this - this is why I'm expanding out into full account-level predictions too. The model is getting better every day and I hope to continue working on it. This is the first negative feedback I've had. I am also unusually transparent in the quality of the model. This exact post you're replying to says the performance of the model in catching cheaters, and how I measure it. I go into even more detail on the blog.
eight_x_eight

Perhaps you could demonstrate your tool working with a few actual games from confirmed cheaters.

On "harsh words", what I said was "this tool appears to be very poor". I think is a fair characterization of the evidence.

" I am not charging for anything"
Of course you are. As your site explains, 'premium' is $9.99 a month.

Perhaps you could demonstrate your tool working with a few actual games from confirmed cheaters. On "harsh words", what I said was "this tool appears to be very poor". I think is a fair characterization of the evidence. " I am not charging for anything" Of course you are. As your site explains, 'premium' is $9.99 a month.