Lc0 T80 Artemis Games

[FritzUser]

These are from .tar.bz2 files located at https://storage.lczero.org/files/training_pgns/. Each day ends up being about 1.5 GB in size. These are Artemis games, though. I define those as games that are between 10-250 moves, end in checkmate, are judged by the Interesting Wins Search Tool to be "very interesting" and are judged by ChessBase 17 to be both "strong" and "most beautiful" (i.e. three-medal beautiful). This reduces one-and-a-half gigabytes down to a megabyte.

These are the games for yesterday (2024-04-20). They oddly come out to exactly 800. Full tags are provided, including openings and beauty.

https://pixeldrain.com/u/dWfBo9Ks

[FritzUser]

2024.04.21 (1660 games):

https://pixeldrain.com/u/mLoN8ZE1

[FritzUser]

2024.04.22 (1638 games):

https://pixeldrain.com/u/wFU4pjz5

These are all Artemis games, filtered from 1.5 GB down to about 1 MB. 10-100 moves, 2300+, ending in checkmate, and which the Interesting Wins search tool rates "very interesting", and which ChessBase rates both "strong" and "most beautiful".

[FritzUser]

Okay. This project got off to a strange start. ChatGPT has written me a couple scripts that automate the process so well that the Interesting Wins Search Tool became a real bottleneck. So I'm combining all days from a particular month, which means I won't have April 2024 for a few more days. Here, though, is what I've been doing today.

January 2024
https://pixeldrain.com/u/cfyYL2QK

February 2024
https://pixeldrain.com/u/srnBTh8r

March 2024
https://pixeldrain.com/l/9nuoQbr8

All games end in checkmate, are between 10-100 moves, and with both players rated at least 2300. They are also considered "very interesting", "strong", and "most beautiful", according to the Interesting Wins Search Tool and ChessBase, respectively. 960 games and doubles have been removed. Includes opening tags and beauty scores.

[FritzUser]

Well... it's all a learning experience. I was using the wrong system to create the files, so having figured it out (after spending all day with ChatGPT) these are the Lc0 T80 test games from yesterday, filtered to include only those that are "very interesting", "strong", and "most beautiful".

2024.04.24 (3460 games)
https://pixeldrain.com/u/iNBr7Z3i

[FritzUser]

2024.04.25 (3361 games)
https://pixeldrain.com/u/diaf6Jm6

[FritzUser]

2024.04.26 (3321 games):
https://pixeldrain.com/u/SZUKLeg8

[FritzUser]

Okay, there's no interest here. But the Artemis collections of Lc0 Test Games are still being made. They're being archived here:

https://github.com/ianrastall/misc-chess-projects/tree/main/lczero-test-games/t80

Hope all are well.

[esamakina]

What's the difference with vanilla T80?

[FritzUser]

What's the difference with vanilla T80?

All the games from T80 would qualify in Chessbase as strong games if they're at least ten moves, since they're already always above 2300. Only about half of them would be considered beautiful at all, and only about ten percent of those would be three medal beautiful. And out of those, only maybe half would be considered very interesting.

So these are games that are "strong", "most beautiful", and "very interesting". Each original PGN is 1.5 GB (which is too much to archive day-by-day) and each Artemis PGN is 2 MB. Approximately, but still. If you plug these into ChessBase, it will read "Top Game" on every one, and will also show three medals for a beauty score. Analysis also tends to reveal that these are exceptional games.

[FritzUser]

In case anyone is interested, here is the code ChatGPT wrote for me to quickly process this stuff.

The Windows PowerShell script for taking a folder of the original .tar.bz2 files and converting them to one PGN:

Code: Select all

Get-ChildItem -Filter *.tar.bz2 | ForEach-Object {
    & tar -xOjf $_ | Out-File -FilePath raw-output.pgn -Append
}

The Bash script (for Windows) for stripping out FENs and Lc0-specific stuff.

Code: Select all

awk '!/^$/ { gsub(/\{OL: 0\}.*/, ""); print }' raw-output.pgn > cleaned_output.pgn

The Bash script (for Windows) for adding in initial headers and blank lines:

Code: Select all

awk 'BEGIN {
    header = "\n[Event \"T80\"]\n[Site \"storage.lczero.org\"]\n[Date \"2024.04.28\"]\n[Round \"?\"]\n[White \"Lc0 0.30.0\"]\n[Black \"Lc0 0.30.0\"]\n[Result \"*\"]\n[WhiteElo \"3633\"]\n[BlackElo \"3633\"]\n[EventDate \"2022.04.04\"]\n\n"
}
{
    print header $0  # Print the header followed by the current line (the moves)
}' cleaned_output.pgn > updated_output.pgn

The Windows Command Line script for running pgn-extract to further clean the PGN:

Code: Select all

pgn-extract.exe updated_output.pgn -Rroster.txt --xroster --minmoves 10 --fixresulttags --fixtagstrings --nosetuptags -D -e --plycount -w9999 -olc0-t80-filtered-240428.pgn

After running through outside programs (the Interesting Wins Search Tool and ChessBase) the output CB file is then turned into the finished product with this Command Line script for pgn-extract:

Code: Select all

pgn-extract.exe game1.pgn -Rroster.txt --xroster --minmoves 10 --fixresulttags --fixtagstrings --nosetuptags -D -e --plycount -w9999 -olc0-t80-artemis-240428.pgn

And finally this is what's in roster.txt:

Code: Select all

Event
Site
Date
Round
White
Black
Result
SetUp
WhiteElo
BlackElo
ECO
Opening
Variation
EventCountry
EventDate
TimeControl
Beauty
PlyCount

Hope all are well.

[esamakina]

What's the difference with vanilla T80?

All the games from T80 would qualify in Chessbase as strong games if they're at least ten moves, since they're already always above 2300. Only about half of them would be considered beautiful at all, and only about ten percent of those would be three medal beautiful. And out of those, only maybe half would be considered very interesting.

So these are games that are "strong", "most beautiful", and "very interesting". Each original PGN is 1.5 GB (which is too much to archive day-by-day) and each Artemis PGN is 2 MB. Approximately, but still. If you plug these into ChessBase, it will read "Top Game" on every one, and will also show three medals for a beauty score. Analysis also tends to reveal that these are exceptional games.

So, if I'm understanding correctly, "Artemis" is your code name for the selected games.

[esamakina]

In case anyone is interested, here is the code ChatGPT wrote for me to quickly process this stuff.

The Windows PowerShell script for taking a folder of the original .tar.bz2 files and converting them to one PGN:

Code: Select all

Get-ChildItem -Filter *.tar.bz2 | ForEach-Object {
    & tar -xOjf $_ | Out-File -FilePath raw-output.pgn -Append
}

The Bash script (for Windows) for stripping out FENs and Lc0-specific stuff.

Code: Select all

awk '!/^$/ { gsub(/\{OL: 0\}.*/, ""); print }' raw-output.pgn > cleaned_output.pgn

The Bash script (for Windows) for adding in initial headers and blank lines:

Code: Select all

awk 'BEGIN {
    header = "\n[Event \"T80\"]\n[Site \"storage.lczero.org\"]\n[Date \"2024.04.28\"]\n[Round \"?\"]\n[White \"Lc0 0.30.0\"]\n[Black \"Lc0 0.30.0\"]\n[Result \"*\"]\n[WhiteElo \"3633\"]\n[BlackElo \"3633\"]\n[EventDate \"2022.04.04\"]\n\n"
}
{
    print header $0  # Print the header followed by the current line (the moves)
}' cleaned_output.pgn > updated_output.pgn

The Windows Command Line script for running pgn-extract to further clean the PGN:

Code: Select all

pgn-extract.exe updated_output.pgn -Rroster.txt --xroster --minmoves 10 --fixresulttags --fixtagstrings --nosetuptags -D -e --plycount -w9999 -olc0-t80-filtered-240428.pgn

After running through outside programs (the Interesting Wins Search Tool and ChessBase) the output CB file is then turned into the finished product with this Command Line script for pgn-extract:

Code: Select all

pgn-extract.exe game1.pgn -Rroster.txt --xroster --minmoves 10 --fixresulttags --fixtagstrings --nosetuptags -D -e --plycount -w9999 -olc0-t80-artemis-240428.pgn

And finally this is what's in roster.txt:

Code: Select all

Event
Site
Date
Round
White
Black
Result
SetUp
WhiteElo
BlackElo
ECO
Opening
Variation
EventCountry
EventDate
TimeControl
Beauty
PlyCount

Hope all are well.

I've done command line and batch scripts, but PowerShell and Bash? No idea, I guess it will be similar. If I can get Lc0 training games in PGN format, it might be worth it.