Can GhatGPT help you learn chess and prepare for serious games?

How this started!

I was interested in seeing if ChatGPT could help me prepare for a tournament game. In this case, the game in question was a ladder game. The time control was 30+30. So I asked it, and it confirmed with enthusiasm. Is this where this assessment ends? No! Of course not! I will give a full comprehensive assessment of the process I took to prepare. I started with this interaction:

I was excited to test this out so I did a full review of my opponent. Submitted 4 games for review. And this is what it spit out at me:

I did a lot more to prep for this. Like I asked how many games I can submit, and I asked if it could calculate, and I asked if it could tree analysis. And to top it all off, I asked if it could predict the type of game I would get. And I specified the move order from me of 1. d4 2. Nf3 and 3. e3. This was a positive experience. It happily accepted pretty much everything like a "yes man". Of course I was literally evaluating this system as a sound way of preparing for an actual game.

Actual analysis

So how did the machine do on actual analysis? I have to be honest. For someone in my rating range, 2000-2200, the analysis was not up to the level it should be. I even tested it's ability to accurately analyze a key position. It did not do very well. The wording from the program looked very good. And surprisingly it actually got a lot of assessments correct. But I will give an example of why I believe it is not good enough, yet:

How exciting, a live picture from my cut and paste! Anyway.. This was a critical position. Or at least it was the result of a critical position. I asked ChatGPT if it recognized it. And was confirmed. Then I asked it to perform a 2 minute calculation as to why fxe6 was incorrect and suggest a better move. This is where ChatGPT failed hard. I repeated the ideas I did and then it failed to notice that Rxe6 was an only move. Not only that, but it spit it's answer out almost immediately, and this is an indication it did not spend 2 minutes on the calculation. So raw calculation is out.

What about the rest?

Well, the chatbot did pretty good. It was able to give a concise and accurate account of things that were completed. It was able to correctly identify themes and minor tactical devices. And it recognizes openings. I think most importantly, it did actually predict how my game would go for the opening.

So.. Based on the information I gave ChatGPT, it actually did pretty good. I am willing to bet that if I introduced it to someone who plays more, it could have made better predictions. When researching my opponent, I had to try to find classical games where he was black. This presented with a problem because there was under 20 games. And about 6 of them were Semi-Slav. Which means I didn't have a very good data chunk. And further, I had no examples until my game in the classical sense where he responded to my 2. Nf3. With that in mind, you have to admire the way the program speaks. I only noticed one major error in the advice if the analysis had very little calculation involved.

Weaknesses?

Well now it is time to reveal the actual weaknesses.

First, and probably most important. If you are using the free version like I was, there is apparently a limit to its usage. When I was preparing to write this blog post, I was still using it to assist with end of game analysis. Because I wanted to be sure about certain aspects of my assessment. And it locked me out. I don't know what limit I hit. I am assuming it's a data limit. Like what I was feeding it rather than what it spits out.

Second, It cannot calculate to the level of someone who has achieved a rating of 2000+. So if you want someone to help you learn how to calculate, you are going to have to have stockfish help you. You can input your analysis and tell ChatGPT that you found errors, and it will address them and change the evaluation. But if you ask it to calculate it will not do a very good job. And I assessed it pretty hard. I even asked it to calculate a specific position and give at least 4 candidate moves with tree analysis. It's calculation was about 2-3 moves. And then 2 of the lines had easy to spot illegal moves.

Last: You as an aspiring chess player have to have the ability to work out your own critical thinking when assessing the advice that is given. So unfortunately, children might not be a prime candidate for using this method, although a persistent adult would have no problem using it. And it's not because the adult is smarter, it is having the ability to communicate with the program in such a way that you can feel interactive debate. Children tend to lack this area of expertise. The ability to work out problems with the aide of an artificial intelligence.

Conclusion

What was good and what I personally liked about the interactions:

ChatGPT in it's current state is a good idea bouncer. The reason I state that directly coincides with the last weakness, which is, the ability to critically think about the words or interactions ChatGPT is sending to you.

This is the game I prepared for. I spent about an hour on my own. And about an hour before and then about 2 hours after the game with ChatGPT, just like I would a real coach. While I came to similar conclusions as ChatGPT did later, I felt that the interactions were more engaging. And not only that, the interactions did help me overall. And the assistant did feel a little like a coach. But I had to correct it on raw calculation a lot. On basic themes, it actually did a pretty good job superficially. I have a feeling if it was not the free version it would do a little better. I didn't fully realize until writing this that I actually could have just simply paired it with stockfish and it would be like talking to a master used correctly.

An adult improver who is a low rated player can indeed use this with benefit, and it could make them feel better about chess training in general. No calculation should be involved, and they should double check all advice for accuracy of output with stockfish. Use the same rules with using stockfish as you would without ChatGPT involved. And adult improver who is higher rated, I will go ahead and say anyone over 1800, will need to concretely analyze each suggestion for accuracy. Not because it is wrong necessarily, but the fact that sometimes it gets positions incorrect, and you should be vigilant about making sure all aspects that ChatGPT spits out at you is accurate and not taken as the word. When pairing with stockfish or another good engine, just be mindful of not taking stockfish as 100% perfect. Yes, it is correct most of the time, but you still have to remember that we don't think like computers. So assessing the engines responses is even more important with this method.

In my opinion, I will likely use it again. But it won't exactly be "someone who is my superior", like I would if it had stockfish attached to it. On the contrary, it would be almost like I am bouncing a ball off a wall. Because while the raw data may have been too inaccurate (I might estimate if I gave it everything I had, it would reach about 30% accuracy. But this is thinking on the level of someone who is 2100+. The accuracy can go up to 50% on lower rated players), the interactions kept me focused on the task. You throw ideas at it, and with sufficient prompting from the user, ChatGPT can help you work things chess like out well. If you do well with ball bouncing off wall style idea hashing, this is a good method. If you feel you are good with engines, you can use stockfish as the calculator and if ChatGPT gets anything wrong let it know with the stockfish analysis and it will happily correct it.

I feel that ChatGPT with honest use of stockfish will help pretty much anyone. And I will admit, in a way, using it with stockfish kind of felt like a more comfortable style of interaction compared to Chessmaster. And since you can interact with it with voice, it makes it more human like by far. And using stockfish for the critical analysis can definitely up the level of the analysis.

If you made it this far, thank you for not falling asleep!!