jja: convert CTG books to PolyGlot format (and more!)

[sarona]

The 0 bytes bin file issue has been resolved by the author.



I am linking to the Talkchess thread where we first conversed. There are links there for Windows and Linux binaries.
https://talkchess.com/forum3/viewtopic.php?f=7&t=81702&start=50#p948813

(MZ, I hope this is not a problem mirroring to Talkchess for this. I figured this is the most direct way in getting the fixed binaries and the author does have some useful information in that thread on how to use the jja tool)

[ConRog]

Jja tool works very well converting CTG books into BIN books! All my free downloaded CTG books past year have been converted correctly with good weight for each variants.

Just launch jja.exe edit book.ctg -o book.bin.

It's quite long (10-15 minutes) to convert a 1GByte CTG book. But it worth the time spent because size is divided by at least 4 :

1,2 GByte CTG - > 300 Mbyte bin book.

Thanks to the author.

Best regards.

[catalano7777]

It worked for me, with a 250 mb CTG book, but I've obtained a BIN one with only 27 Kb...
C:\jja>jja edit Cubail.ctg -o Cubail.bin
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for positions in the CTG input file. This may take a while.
Skipping illegal move with fen:1b4k1/4rpp1/1q5p/1r1Q4/8/3Q1N1P/2P2PP1/R5K1 w - - uci:d5d3 err:illegal uci
Skipping illegal move with fen:1Q6/4P1q1/6pk/7p/4P2P/8/5PK1/2q4B b - - uci:g7g1 err:illegal uci
Skipping illegal move with fen:r5k1/2p2pp1/3q1n1p/8/1R1q4/1Q5P/4RPP1/1B4K1 b - - uci:d4d6 err:illegal uci
Skipping illegal move with fen:2Q4b/5pk1/8/4p2p/7P/6PK/4p1Q1/1q6 w - - uci:g2g8 err:illegal uci
Found 3687230 chess positions in the CTG input file.
Saving 3687230 entries from input book to the output book...
Success copying entries from the input book to the output book.
Saving done. [3687230/?] 27,217.2024/s [00:02:15]

I wonder if it's right and useful...


Thank you!

[mymoso]

Hello everybody

Where can I find documentation on all application functions?
I've looked on Github and on the author's website and haven't found it. Maybe I missed something.
Can you help?

Greetings from Brazil

[catalano7777]

It worked for me, with a 250 mb CTG book, but I've obtained a BIN one with only 27 Kb...
C:\jja>jja edit Cubail.ctg -o Cubail.bin
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for positions in the CTG input file. This may take a while.
Skipping illegal move with fen:1b4k1/4rpp1/1q5p/1r1Q4/8/3Q1N1P/2P2PP1/R5K1 w - - uci:d5d3 err:illegal uci
Skipping illegal move with fen:1Q6/4P1q1/6pk/7p/4P2P/8/5PK1/2q4B b - - uci:g7g1 err:illegal uci
Skipping illegal move with fen:r5k1/2p2pp1/3q1n1p/8/1R1q4/1Q5P/4RPP1/1B4K1 b - - uci:d4d6 err:illegal uci
Skipping illegal move with fen:2Q4b/5pk1/8/4p2p/7P/6PK/4p1Q1/1q6 w - - uci:g2g8 err:illegal uci
Found 3687230 chess positions in the CTG input file.
Saving 3687230 entries from input book to the output book...
Success copying entries from the input book to the output book.
Saving done. [3687230/?] 27,217.2024/s [00:02:15]

I wonder if it's right and useful...


Thank you!

Now, after replace jja.exe by the fied one, I've finally obtained a 30 mb BIN book from a 250 mb CTG.
The fixed file is in this link: https://talkchess.com/forum3/viewtopic.php?f=7&t=81702&start=50#p948813

Thanks to all!

[catalano7777]

It worked for me, with a 250 mb CTG book, but I've obtained a BIN one with only 27 Kb...
C:\jja>jja edit Cubail.ctg -o Cubail.bin
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for positions in the CTG input file. This may take a while.
Skipping illegal move with fen:1b4k1/4rpp1/1q5p/1r1Q4/8/3Q1N1P/2P2PP1/R5K1 w - - uci:d5d3 err:illegal uci
Skipping illegal move with fen:1Q6/4P1q1/6pk/7p/4P2P/8/5PK1/2q4B b - - uci:g7g1 err:illegal uci
Skipping illegal move with fen:r5k1/2p2pp1/3q1n1p/8/1R1q4/1Q5P/4RPP1/1B4K1 b - - uci:d4d6 err:illegal uci
Skipping illegal move with fen:2Q4b/5pk1/8/4p2p/7P/6PK/4p1Q1/1q6 w - - uci:g2g8 err:illegal uci
Found 3687230 chess positions in the CTG input file.
Saving 3687230 entries from input book to the output book...
Success copying entries from the input book to the output book.
Saving done. [3687230/?] 27,217.2024/s [00:02:15]

I wonder if it's right and useful...


Thank you!

Now, after replace jja.exe by the fied one, I've finally obtained a 30 mb BIN book from a 250 mb CTG.
The fixed file is in this link: https://talkchess.com/forum3/viewtopic.php?f=7&t=81702&start=50#p948813

Thanks to all!

Hi, guys,

Now,a new error appears when I attempt to convert a CTG +2Gb of size. The following message arises,

"Code 131 An attempt was made to move the file pointer past the start of the file"

I tried empty the TEMP folder from AppData, but nothing changed...
Any suggestion or idea to solve this?
This happen with any CTG with 2 Gb or more.

Thank you!

[sarona]

Does jja begin saving entries to the book and then triggers the Code 131 error before completion? Or does it crash before the saving process begins?

If it is happening with multiple CTGs, it is most likely not data corruption.

I have a conversion in process (the CTG is only 1.92 GB), but is not completed (around 56% completed) at the time of this writing. No trouble yet. Using a Windows executable.

[sarona]

Okay, the conversion worked.



I will try the Immortal2015v28.ctg with the Threadripper (that book is over 9 GB and will take forever with the laptop ) overnight.

Do you have a link to any of the books that failed for you? Want to try and convert it here and see if I get the error as well.

[catalano7777]

Okay, the conversion worked.



I will try the Immortal2015v28.ctg with the Threadripper (that book is over 9 GB and will take forever with the laptop ) overnight.

Do you have a link to any of the books that failed for you? Want to try and convert it here and see if I get the error as well.

Well, some things to says,
First of all, I think your jja folder is not the same as mine... I can see a batch file in your folder, that's missing in mine.
Then, these are my errors:

As you can see, I tried with Inmortal too, and there are 0 lines or positions, so, nothing is copied..

Then, this other error:

Here, you can see the code 131 error. It's in spanish, so this is the message in English
"an attempt was made to move the file pointer past the start of the file"
This happens with any file over than 2 Gb. No problem with smaller ones.

And finally, this is my jja folder,

Could you tell me if this is ok? As I told you, I have missing that batch file that I can see in yours

Tell me if further information is needed for you

Thank you!

[sarona]

I created the batch fille so that jja.exe would start automatically. Just one line in the file.

start "jja.exe"

Save the file as jja.bat and keep it in the same folder jja.exe is located. Double click jja.bat and the program starts.

It is 3 AM here and I am preparing to go to a job site. I will look at this tonight and post again this evening western Canadian time.

[sarona]

Okay, the conversion worked.



I will try the Immortal2015v28.ctg with the Threadripper (that book is over 9 GB and will take forever with the laptop ) overnight.

Do you have a link to any of the books that failed for you? Want to try and convert it here and see if I get the error as well.

Well, some things to says,
First of all, I think your jja folder is not the same as mine... I can see a batch file in your folder, that's missing in mine.
Then, these are my errors:

As you can see, I tried with Inmortal too, and there are 0 lines or positions, so, nothing is copied..

Then, this other error:

Here, you can see the code 131 error. It's in spanish, so this is the message in English
"an attempt was made to move the file pointer past the start of the file"
This happens with any file over than 2 Gb. No problem with smaller ones.

And finally, this is my jja folder,

Could you tell me if this is ok? As I told you, I have missing that batch file that I can see in yours

Tell me if further information is needed for you

Thank you!

My attempt with the Immortal v28 CTG did not succeed either.

But the author is aware of the problem. https://todo.sr.ht/~alip/jja/14

There is updated source in his repository and I am going to try and compile a binary and try again.

[sarona]

I tried a binary built from the latest source from his git
71a16fef match: use xorshift rng to pick random moves

But this executable did not produce a bin book for the ~9 GB Immortal v28 book. Same problem. I am going to PM the author at another forum.

It did convert a ~2.2 GB CTG quickly, though.


@catalano7777 if you want to try the Windows executable, it is here: https://pixeldrain.com/u/1Sn1hFjL

All credit for this excellent project goes to Ali Polatel.

[Kater Carlo]

Ali Polatel has created version 0.6 of his utility:

https://www.chesswob.org/jja/

We're excited to announce that JJA has reached another milestone with the release of version 0.6.0! Following our previous version 0.5.0, this latest update brings substantial enhancements, critical fixes, and notable new features that are sure to make your chess analysis even more powerful. Read more about it in the blog post

[sarona]

It looks like Ali has fixed the overflow issue when dealing with large CTG books. I was able to convert a ~4 GB CTG after several earlier failures and, as of the time of this writing, I am trying to convert the ~9 GB UpdateImmortal 2012a final v28. It does seem to be processing the data, but it is early. . .


ctg: fix multiplication overflow in read_page support nag value 7
https://git.sr.ht/~alip/jja/commit/ec33de9c988bf95bcfd9e97eec91e1384de04eec

jja build ##1018182
https://patchouli.sr.ht/builds.sr.ht/artifacts/~alip/1018182/8e08ff6fbc7d9cb8/jja.exe

You can access most of the builds here: https://builds.sr.ht/~alip/jja?
Click on the job number to get to an artifacts link (for example, click on #1018227).

[catalano7777]

It looks like Ali has fixed the overflow issue when dealing with large CTG books. I was able to convert a ~4 GB CTG after several earlier failures and, as of the time of this writing, I am trying to convert the ~9 GB UpdateImmortal 2012a final v28. It does seem to be processing the data, but it is early. . .


ctg: fix multiplication overflow in read_page support nag value 7
https://git.sr.ht/~alip/jja/commit/ec33de9c988bf95bcfd9e97eec91e1384de04eec

jja build ##1018182
https://patchouli.sr.ht/builds.sr.ht/artifacts/~alip/1018182/8e08ff6fbc7d9cb8/jja.exe

You can access most of the builds here: https://builds.sr.ht/~alip/jja?
Click on the job number to get to an artifacts link (for example, click on #1018227).

Excellent!! This version has definitely fixed this issue!
I've tried with two versions, 9e42cd6 and ##1018182
Both of them have created a bin book with a CTG over 4 Gb, but for any reason, jja ##1018182 has created a bit bigger bin book (914 mb) against 9e42cd6 (913 mb) from the same CTG book.

This is 9e42cd6


And this is jja ##1018182


I've not tested these 2 books yet, and I don't know if I will find any difference between then, about strengh. I also don't know how many more lines one book can have compared to the other. I'll be trying this.

Thanks a lot to the author and to the user of this post!!!