I investigated using ML to find cheaters

@eight_x_eight As I said, that is simply gathering evidence. Premium does not exist. It does not cost to register interest. If I find that people are interested, then I may move on to a premium offering which involve additional features. But that relies on people finding it helpful and wanting to pay. There is no sin in seeing if people are interested in premium features for something that has been built over many hours of effort.

"Confirmed cheaters" do not exist. (Unless you cheat yourself over many games, or pay people to cheat to gather large datasets - and even that can never be known to be a true representation of how real cheaters behave). A platform like chess.com or lichess banning someone does not mean they are a confirmed cheater or that they cheated in any particular game. As someone who's built ML systems for some of the largest tech companies, I can tell you that there is significant error in any platform's classifiers. It's an inevitability. There is not a single confirmed cheater dataset in the world that I know of, please feel free to share if you find one - it would be very helpful!

Much better evidence than your 6 games from 1 account is the evidence that I posted that clearly explains the false positive and false negative rate over a dataset of almost 7000 games (7000 includes train, test and evaluation), how I generated it, and how cheaters are characterized in it. Calling it a scam is ridiculous and only serves to undermine your point.

Here's a performance breakdown:

Of players who use stockfish for 50–75% of moves, my ML model finds 72% of them.
Of players who use stockfish for 95+% of moves, my ML model finds 97% of them.

The test case you mentioned - a lower rated player with an engine - is actually a particularly challenging case. There is very little difference between a 1000 w/ a little stockfish and a 1800 w/ no stockfish. I have some ideas to tackle this though.

I do not believe you are debating in good faith so I'll leave it here.

@eight_x_eight said ^

Perhaps you could demonstrate your tool working with a few actual games from confirmed cheaters.

On "harsh words", what I said was "this tool appears to be very poor". I think is a fair characterization of the evidence.

" I am not charging for anything"
Of course you are. As your site explains, 'premium' is $9.99 a month.

Bruh but the site is a good site dont be jealous

@KnightMovesLikeJagga

To summarise, you don't believe there is such a thing as a confirmed cheater, which is why you can't demonstrate your tool working in real examples.

"As someone who's built ML systems for some of the largest tech companies, I can tell you that there is significant error in any platform's classifiers. "

Both sites, and especially cc for commercial reasons, are very careful to avoid false bans. I used cc accounts closed for FPV because on Lichess a ToS flag sometimes isn't related to fair play. This 'significant error' comment is not correct. False negatives, yes. But very very few false positives

"Much better evidence than your 6 games from 1 account is the evidence that I posted that clearly explains the false positive and false negative rate over a dataset of almost 7000 games (7000 includes train, test and evaluation), how I generated it, and how cheaters are characterized in it. Calling it a scam is ridiculous and only serves to undermine your point."

The 6 games were over 3 accounts, I didn't say they were from a single account. The chances of all 6 games being clean is tiny. Your 7000 games are not human games. If the hypothesis being tested is "does the tool detect humans cheating", then you need to work with human games.

"Premium does not exist."

Bruh you did cut the image and edited the text

@eight_x_eight said ^

@KnightMovesLikeJagga

To summarise, you don't believe there is such a thing as a confirmed cheater, which is why you can't demonstrate your tool working in real examples.

"As someone who's built ML systems for some of the largest tech companies, I can tell you that there is significant error in any platform's classifiers. "

Both sites, and especially cc for commercial reasons, are very careful...

If you think you are so good do it yourself from 0 and see if you can progress *Bro Chill

Just an update on this:

I have now generated almost 28,000 games, using 4 different cheating strategies.

I believe that these four groups cover the majority of cheating styles that we see real games. The streak-based methods are a common tactic, where cheaters will not cheat to start, but then consistently cheat for a few moves. Streak and Losing Streak should capture cases where humans find a position complex, get annoyed, or otherwise begin a cheating streak.

Blunder-avoidance is a different cheating style, where a cheater consults an engine frequently, but only plays the engine move in deciding positions.

An updated model trained on all these game types will be published soon.

That's it. I'm switching to tic tac toe. Especially on the weekend.

@KnightMovesLikeJagga

I want you to see this game. When I put this game into "Chess Cheat Detector"...

But obviously, see the game result and reason!

Thanks for trying it @an7cplayingchess !

I have a new model that is trained on the data I listed above. The new model takes rating into account (and has the extra data such as streaks, etc). That model flags white with 81.6% probability. :)

More detail:

So we can see that:

• If the game is uploaded with the ratings provided, the new model gets them easily now.
• If the game is uploaded without the ratings provided, the new model is still more confident than the current one, but not confident enough. Having no ratings makes it very difficult - so the model which uses the ratings will be automatically used whenever a rating is provided.

I'll deploy this version to the app soon :)