Lichess Puzzle Database (Sorted By Rating)

FritzUser · Post by **FritzUser** » 23/08/2023, 23:19

Created from the CSV obtained via the Lichess Open Database. All the product of either search and replace -- mostly with regular expressions -- or of simply opening the file in ChessBase, which then provides the proper tags and converts the movelist to standard algebraic notation.

3,307,409 puzzles • 896 MB PGN • 92.5 MB 7z
https://www.mediafire.com/file/hdwitwnm8fpefvu/lichess-puzzles-by-rating.7z/file

catalano7777 · Post by **catalano7777** » 24/08/2023, 3:17

FritzUser wrote: ↑23/08/2023, 23:19 Created from the CSV obtained via the Lichess Open Database. All the product of either search and replace -- mostly with regular expressions -- or of simply opening the file in ChessBase, which then provides the proper tags and converts the movelist to standard algebraic notation.

3,307,409 puzzles • 896 MB PGN • 92.5 MB 7z
https://www.mediafire.com/file/hdwitwnm8fpefvu/lichess-puzzles-by-rating.7z/file

The database is empty, although it's 896 Mb... All games are empty...

Thank you

FritzUser · Post by **FritzUser** » 24/08/2023, 9:14

catalano7777 wrote: ↑24/08/2023, 3:17 The database is empty, although it's 896 Mb... All games are empty...

Thank you

Well it is there. If it doesn't work in general, then I made a mistake, and I shouldn't have this up here for people to grab. It's not a problem where I can just say, "Do x, y and z to fix it." I suppose it's possible that it's supposed to start on the second move, but one problem with the non-standard puzzle tags is that they often get stripped out by software, or cause software to not know what to make of them, so presenting a simple PGN with a FEN and a movelist seems like the best idea. If it isn't, then (like I said) it shouldn't be available, and I would ask that this thread be deleted.

Stormfogel · Post by **Stormfogel** » 24/08/2023, 11:29

It works fine in Chessbase 17.

FritzUser · Post by **FritzUser** » 24/08/2023, 22:59

On further reflection, I think I underestimated how incredibly difficult it was to do these. At first it seemed impossible without knowing how to code. But I was able to do it all in about twelve hours with regular expressions and by eventually opening the file in CB 17, which switched the notation to SAN. It does say on the puzzles page on the Lichess Open DB, however, that all the puzzles start on the second move. So I'm thinking now that every single one of these is backwards. Unless I can figure out how to add in some tag that makes each one start on move two, it looks like this is beyond my capabilities. Stripping out the first move of each one might be possible, but then each of the 3.3M FEN tags would be incorrect. So this DB has to come with that caveat. My apologies.

Viktor16 · Post by **Viktor16** » 25/08/2023, 5:56

The problem of large volumes of PGN files can be solved if this file is split using the utility. And then open each part.

PgnSplit
https://pixeldrain.com/u/d4dtpaMY

FritzUser · Post by **FritzUser** » 25/08/2023, 16:59

Viktor16 wrote: ↑25/08/2023, 5:56 The problem of large volumes of PGN files can be solved if this file is split using the utility. And then open each part.

PgnSplit
https://pixeldrain.com/u/d4dtpaMY

Thank you for the suggestion. I do have that program, and had considered using it. It was just a very long process of reg exp using Notepad++, so I knew that I'd mess it up if I tried to split it into ten parts. Or lose momentum. But yes, it wouldn't have taken so long, because every single search and replace was making me wait half an hour to go through all twenty million or so lines.

catalano7777 · Post by **catalano7777** » 25/08/2023, 17:14

Stormfogel wrote: ↑24/08/2023, 11:29 It works fine in Chessbase 17.

As you can see, with Fritz 18, all games are lines..

FritzUser · Post by **FritzUser** » 26/08/2023, 1:06

Yes, these are lines. They are not games. You open it with the training tab open, and whether or not it actually plays the puzzle from the correct side, it nonetheless should allow you to guess the moves from your side.

CB, Fritz, etc. will call it a line if it's not a full game. That's all. These all start from positions and feature just the moves of the puzzle.

FritzUser · Post by **FritzUser** » 28/08/2023, 13:56

Okay. I figured it out. All you have to do if you open these puzzles in the ChessBase training tab is to untick the box that says which side plays, and tick the box for the other side! That's all you have to do. It will play the first move for you.

Lotus340 · Post by **Lotus340** » 02/01/2024, 12:05

Thanks so much to FritzUser for his work in compiling this pgn file. This is really useful!

The pre-move for each puzzle isn't really a fault. In fact, it can be educational to see what the losing move is that leads to the tactic. Chess Tempo puzzles work in the same way.

For use in Droidfish, in a text editor I changed "WhiteElo" to "White". This enables the rating to be seen directly in the Droidfish index of puzzles.

FritzUser · Post by **FritzUser** » 02/01/2024, 21:54

Thank you for the kind words. I still think of this database, however, as requiring a final step. Hopefully someone out there with knowledge of Python or R can take a look at it. The problem, as discussed, is that the FEN refers to the first move, and the puzzle starts on the second. So I need to change the FEN to reflect the position on the second move -- and I'm pretty sure you can't just make a simple change to it, like changing "w" to "b" or something. Because the position is different too, in a non-predictable way. So the second move would have to be read for each game, and then the FEN switched out. That's all far beyond me... and actually quite irritating, since Lichess knows that we can't do it easily. I also notice that they don't allow you to save the PGN for a puzzle. (Chesscom actually does.) I still can't figure out what their PGN tags look like for their puzzles, i.e. if they use extra tags.

Puzzles are a great example of a project in need of distributed effort, since the more busy work that's required, the longer the timeline necessary. Unless you can organize a group effort. Then the timeline is doable.

What boggles my mind is that Chesscom stores away about a billion new individual PGNs every day, and while we can grab those in groups of 50, or even in single PGNs for each user (if using a downloader), in the end we *do* have access to them, and we'll never be able to get them. But the more people we throw at the problem, the better the outlook gets.

Say they have half a trillion games. 500B. I know from experience that after you filter out the lower-rated games, you end up with about 5% being high-quality. And the higher-rated games only represent about 1% of the total. So just going by my own estimations, you'd end up with 250M high-quality games. Now that may be an incredibly huge amount, but then again, if you could find a way to whittle that down to ten percent, then you'd have 25M. So that's not some ungodly large number of PGNs. But it *is* 25 *million* extremely good chess games just waiting to be found.

Another good example of a project over there that could be done by a group is in switching out online games into OTB games. You would have to change the username to the name of the actual titled player, then find their FIDE rating for that month and switch it out with the Glicko2 rating that the game has. Not sure how you would categorize those, but technically if Hikaru plays Magnus on Chesscom, that's no different than if they meet in person.

My apologies for the length of the comment. Hope you're having a great new year.

EDIT: Forgot to mention. There *are* extra PGN tags for puzzles, but I've tested it out, and ChessBase strips them all out. It only allows you to use a FEN.

FritzUser · Post by **FritzUser** » 04/01/2024, 4:51

Well, I've figured it out, almost. Just make a copy of the PGN database, then strip out the first half-move of every game. Then put that database into ChessBase. *Hopefully* it would fix every single FEN automatically. Then I could make a spreadsheet from that database, copy the list of FEN positions, then take the original, convert *that* to a spreadsheet, then paste in the FENs over the ones that are in there, in the column for that tag value. Then convert the whole thing back to PGN. (This would be via turning the PGN into a CSV first. Excel can read a CSV very easily. It's super-easy to do, since there are no commas ever in a PGN file.)

I just don't know how to strip out the first half-move. I've exhausted my limited toolbox already. But if that step could be overcome, the rest would be doable. I've turned a PGN into a CSV and thus into an Excel spreadsheet many times, and then back again. So all that would be required would be to hope that CB17 recognizes that all the FENs are wrong in the second database and just fix them. This is the technique I used to convert UCI notation into SAN. Turns out that CB17 will do that automatically for you.

FritzUser · Post by **FritzUser** » 05/01/2024, 0:59

Just a note: nobody try doing that. It is absolutely not the way. :-)

Outskirts CheSS ForuM

Outskirts CheSS ForuM