lichess.org
Donate

Plotting the performance of openings VS Elo with an Open Source Tool.

Hi,
thanks for your blog post, really cool research!

Do you have any especially cool insights into how opening success rate changes with elo? For example I expect most "trap" endings to score high against unprepared players but very low against grandmasters who know what they're doing. Or maybe there are openings where performance is a real rollercoaster and goes up and down with elo (kind of like King's Gambit apparently).

For rare (or unsound) openings and elo brackets, is there a problem with of not enough games played to determine elo gain reliably?

As for your questions - a website sounds really cool! And I love your charts.
I don't play enough games for repertoire analysis to be useful, and I'm not a machine - I play openings that I enjoy, not the most theoretically optimal ones ;). So that's a "no" for the other two questions, but I still find the data extremely interesting.

Hi, thanks for your blog post, really cool research! Do you have any especially cool insights into how opening success rate changes with elo? For example I expect most "trap" endings to score high against unprepared players but very low against grandmasters who know what they're doing. Or maybe there are openings where performance is a real rollercoaster and goes up and down with elo (kind of like King's Gambit apparently). For rare (or unsound) openings and elo brackets, is there a problem with of not enough games played to determine elo gain reliably? As for your questions - a website sounds really cool! And I love your charts. I don't play enough games for repertoire analysis to be useful, and I'm not a machine - I play openings that I enjoy, not the most theoretically optimal ones ;). So that's a "no" for the other two questions, but I still find the data extremely interesting.

This is a very interesting concept and something I would love to see more fleshed out. Do you think you could include the performance of certain lines as well? Like the Vienna gambit in the Vienna opening? or what about C5 in the advance caro as black?

This is a very interesting concept and something I would love to see more fleshed out. Do you think you could include the performance of certain lines as well? Like the Vienna gambit in the Vienna opening? or what about C5 in the advance caro as black?

How to install and use this?

How to install and use this?

I liked the graphic for the English.

I liked the graphic for the English.

@paren1 I have a theory on the weird graph for the King's Gambit. It goes down with elo probably because people start being prepared against it, and know how to refute the main lines, but probably at the highest levels there are some random strong players who will be aware of the common "refutations" and still know how to keep playing against them. Not only that but most people would stop playing Kings Gambit at higher elos because "it doesn't work anymore" meaning those that DO still play it, are likely only those who are extremely prepared on it, and not just "playing fun gambits".

For example, I, as a highish rated D4 player, would consider myself to be extremely prepared against the Englund gambit; I know refutations to all of what I consider playable lines. I'm talking dozens of moves deep. Most of the time I smile when an oponent plays it because I win around 80% of the time against it. Still, one day I faced a high rated guy (2700ish) that apparently knew theory well beyond what I kne. He would reach the "end positions" of my preparation (which look completely winning for white) and still blitz out maybe 3-5 more instant moves, where I had to actually start thinking, and completely outplay me. I then checked his profile and he literally only played Englund against D4, and had thousands of games on it.

@paren1 I have a theory on the weird graph for the King's Gambit. It goes down with elo probably because people start being prepared against it, and know how to refute the main lines, but probably at the highest levels there are some random strong players who will be aware of the common "refutations" and still know how to keep playing against them. Not only that but most people would stop playing Kings Gambit at higher elos because "it doesn't work anymore" meaning those that DO still play it, are likely only those who are extremely prepared on it, and not just "playing fun gambits". For example, I, as a highish rated D4 player, would consider myself to be extremely prepared against the Englund gambit; I know refutations to all of what I consider playable lines. I'm talking dozens of moves deep. Most of the time I smile when an oponent plays it because I win around 80% of the time against it. Still, one day I faced a high rated guy (2700ish) that apparently knew theory well beyond what I kne. He would reach the "end positions" of my preparation (which look completely winning for white) and still blitz out maybe 3-5 more instant moves, where I had to actually start thinking, and completely outplay me. I then checked his profile and he literally only played Englund against D4, and had thousands of games on it.

Repertoire Suggestions: Would a tool that suggests openings based on stats and personal preferences (e.g., tactical, solid) be useful?

Although I'm no longer playing rated OTB games, I assume someday players may find such a tool useful; however, making a useful tool sounds challenging since players' needs may differ.

> Repertoire Suggestions: Would a tool that suggests openings based on stats and personal preferences (e.g., tactical, solid) be useful? Although I'm no longer playing rated OTB games, I assume someday players may find such a tool useful; however, making a useful tool sounds challenging since players' needs may differ.

@paren1 said in #2:

Hi,
thanks for your blog post, really cool research!

Do you have any especially cool insights into how opening success rate changes with elo? For example I expect most "trap" endings to score high against unprepared players but very low against grandmasters who know what they're doing. Or maybe there are openings where performance is a real rollercoaster and goes up and down with elo (kind of like King's Gambit apparently).

For rare (or unsound) openings and elo brackets, is there a problem with of not enough games played to determine elo gain reliably?

As for your questions - a website sounds really cool! And I love your charts.
I don't play enough games for repertoire analysis to be useful, and I'm not a machine - I play openings that I enjoy, not the most theoretically optimal ones ;). So that's a "no" for the other two questions, but I still find the data extremely interesting.

Hello! Thanks for your feedback. Yes, I share the same analysis as you, and that’s exactly why I included the King’s Gambit. The shape of its performance is very interesting. At low Elo, it doesn’t work that well, probably because it requires some experience to convert the initiative into a real attack. But at mid Elo, it becomes an incredible weapon. If your opponent isn’t prepared, it can lead to a devastating assault. However, the higher the Elo, the lower the performance tends to be.

I’ll run simulations on other openings to see if this trend holds elsewhere.

Regarding the risk of having too few games to assess the performance of a line reliably: good question. To handle that, I use the p-value. You can read more about how it works here: https://en.wikipedia.org/wiki/P-value . When I run the program, I can choose a threshold of statistical significance, and results that don’t meet that threshold are discarded. This mechanism is implemented in the “hunt” mode of the tool, but it’s not currently used when generating the charts, so yes, it's possible that some chart data could be based on too small a sample size.

That said, for the openings I picked in the blog post, the number of games is very high, so I’m not too worried about statistical reliability, except maybe at 2500 Elo, where the sample sizes are smaller. I might double-check those, but for the rest, I’m highly confident that the numbers are meaningful.

@paren1 said in #2: > Hi, > thanks for your blog post, really cool research! > > Do you have any especially cool insights into how opening success rate changes with elo? For example I expect most "trap" endings to score high against unprepared players but very low against grandmasters who know what they're doing. Or maybe there are openings where performance is a real rollercoaster and goes up and down with elo (kind of like King's Gambit apparently). > > For rare (or unsound) openings and elo brackets, is there a problem with of not enough games played to determine elo gain reliably? > > As for your questions - a website sounds really cool! And I love your charts. > I don't play enough games for repertoire analysis to be useful, and I'm not a machine - I play openings that I enjoy, not the most theoretically optimal ones ;). So that's a "no" for the other two questions, but I still find the data extremely interesting. Hello! Thanks for your feedback. Yes, I share the same analysis as you, and that’s exactly why I included the King’s Gambit. The shape of its performance is very interesting. At low Elo, it doesn’t work that well, probably because it requires some experience to convert the initiative into a real attack. But at mid Elo, it becomes an incredible weapon. If your opponent isn’t prepared, it can lead to a devastating assault. However, the higher the Elo, the lower the performance tends to be. I’ll run simulations on other openings to see if this trend holds elsewhere. Regarding the risk of having too few games to assess the performance of a line reliably: good question. To handle that, I use the p-value. You can read more about how it works here: https://en.wikipedia.org/wiki/P-value . When I run the program, I can choose a threshold of statistical significance, and results that don’t meet that threshold are discarded. This mechanism is implemented in the “hunt” mode of the tool, but it’s not currently used when generating the charts, so yes, it's possible that some chart data could be based on too small a sample size. That said, for the openings I picked in the blog post, the number of games is very high, so I’m not too worried about statistical reliability, except maybe at 2500 Elo, where the sample sizes are smaller. I might double-check those, but for the rest, I’m highly confident that the numbers are meaningful.

@Thatrandompeashooter said in #3:

This is a very interesting concept and something I would love to see more fleshed out. Do you think you could include the performance of certain lines as well? Like the Vienna gambit in the Vienna opening? or what about C5 in the advance caro as black?

Hello,
You should check out the previous blog post I made about the tool that generates both the charts and the analysis, it lets you request stats on any line you want. I don’t think I’ve run a hunt on the Vienna Gambit yet, but that’s actually one of my hopes: that people use the tool and contribute pull requests with their own analysis. That way, the reports become a shared resource built by the community.

The tool is still quite new but I'm starting to get people interested in contributing. I know cloning the repo and running Python code isn’t ideal for everyone. I’m thinking about turning this into a website to make it more accessible.

@Thatrandompeashooter said in #3: > This is a very interesting concept and something I would love to see more fleshed out. Do you think you could include the performance of certain lines as well? Like the Vienna gambit in the Vienna opening? or what about C5 in the advance caro as black? Hello, You should check out the previous blog post I made about the tool that generates both the charts and the analysis, it lets you request stats on any line you want. I don’t think I’ve run a hunt on the Vienna Gambit yet, but that’s actually one of my hopes: that people use the tool and contribute pull requests with their own analysis. That way, the reports become a shared resource built by the community. The tool is still quite new but I'm starting to get people interested in contributing. I know cloning the repo and running Python code isn’t ideal for everyone. I’m thinking about turning this into a website to make it more accessible.