Your network blocks the Lichess assets!

lichess.org
Donate

A Harness for LLM Chess: Board State and Context Handling

@SERGEY-RUS68 said ^

It would be nice to try to teach these LLM models on chess books! For example, Capablanca's book - The Manual of Chess, and so on. So that the LLM model explains each and why it is done! For example, there is another book by Andre Filidor - Analysis, that's what you need to teach.

Interesting idea! thanks, I might try and make a project on that sometime...

@SERGEY-RUS68 said [^](/forum/redirect/post/NwanI52X) > It would be nice to try to teach these LLM models on chess books! For example, Capablanca's book - The Manual of Chess, and so on. So that the LLM model explains each and why it is done! For example, there is another book by Andre Filidor - Analysis, that's what you need to teach. Interesting idea! thanks, I might try and make a project on that sometime...

LLM???!!! (LM-LIFE MASTER)
LLM-LAME LIFE MASTER

LLM???!!! (LM-LIFE MASTER) LLM-LAME LIFE MASTER

Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move.

You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far

Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move. You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far

@the39clues said ^

Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move.

You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far

Haha thanks for sharing this! Genuinely awesome! I was about to start doing this myself but I'll just bookmark yours instead :) Do you have a repo link? [EDIT: found it, thanks]

@the39clues said [^](/forum/redirect/post/2illguT5) > Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move. > > You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far Haha thanks for sharing this! Genuinely awesome! I was about to start doing this myself but I'll just bookmark yours instead :) Do you have a repo link? [EDIT: found it, thanks]

Thank you for your answer! However, in that case, I think it’s misleading to say that LLMs are bad at chess because we’re prompting them poorly. They’re bad at chess because they’re not made for chess; they’re made to make semi-randomly selected words appear next to other words. Given the real-world harm caused by misunderstanding what LLMs are, I think it’s important that we do not spread the idea that LLMs are capable of reasoning.

I appreciate that you’re approaching this as an intellectual exercise though, and I see value in that.

Thank you for your answer! However, in that case, I think it’s misleading to say that LLMs are bad at chess because we’re prompting them poorly. They’re bad at chess because they’re not made for chess; they’re made to make semi-randomly selected words appear next to other words. Given the real-world harm caused by misunderstanding what LLMs are, I think it’s important that we do not spread the idea that LLMs are capable of reasoning. I appreciate that you’re approaching this as an intellectual exercise though, and I see value in that.

I've been fiddling with leveraging an llm for a bot. There are a few chess specific llms that I can run locally. I've had success by using structure outputs AND providing a list of each LEGAL move. The prompt indicates that they MUST choose from the options provided. Then rank each move. I also provide a FEN with a PGN for added context. The results are interesting and not always a blunder.

Chess is hard.

I've been fiddling with leveraging an llm for a bot. There are a few chess specific llms that I can run locally. I've had success by using structure outputs AND providing a list of each LEGAL move. The prompt indicates that they MUST choose from the options provided. Then rank each move. I also provide a FEN with a PGN for added context. The results are interesting and not always a blunder. Chess is hard.

@Fingolfin7 said ^

Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move.

You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far

Haha thanks for sharing this! Genuinely awesome! I was about to start doing this myself but I'll just bookmark yours instead :) Do you have a repo link? [EDIT: found it, thanks]

Really appreciate the kind words, thank you! :)

@Fingolfin7 said [^](/forum/redirect/post/LPPDgG5o) > > Good ideas. For my benchmark, I also feed the model its own internal thoughts, but only from the previous move. > > > > You can see the results at https://chessbenchllm.onrender.com As you noted, Gemini is the strongest by far > > Haha thanks for sharing this! Genuinely awesome! I was about to start doing this myself but I'll just bookmark yours instead :) Do you have a repo link? [EDIT: found it, thanks] Really appreciate the kind words, thank you! :)