mehmet1921's books tests.

Moderators: Elijah, Igbo, timetraveller

kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

To be fair a very weak processor is the best way to test a book, errors aside, otherwise the strength of Stockfish will just end all the games as draw although the end of the lines are slightly unbalanced.

Just subtract the lost games where the engine makes a mistake in middlegame-ending and leave in the ones in which the book lines ends +{0.30~255.00} or -{0.30~255.00} (below the score 0.30 on book lines end the threshold can be considered insufficient to win because the advantage is too little).

Unfortunately the NNUE nets didn't just raise in a monstrous way the engines strength, they also brought the infamous draw death, for which many computer chess players are leaving their hobby because the games became very boring. A medium-strong processor plus the NNUE net will make the game ending as draw even if the ending books lines positions have the score
±{0.30~0.90}, which is an error in the competition chess opening books world.

Eduard wrote: 14/09/2021, 11:38 Thanks, and Congratulations to GOI book.

But I have a problem: I don't play with such slow hardware (slow 1 core). I don't know how to tune a book when moves are played in depth 20 (Stockfish)? I could do that, but it doesn't make much sense for live games on servers. So that you can really improve a book in a meaningful way, it should be played with at least 4 cores.

If I see such positions that are a draw (I play with 6-pieces), how should I improve something? Black didn't do anything wrong. I can't improve that.

Cfish 130721 G - Cfish 130721 P 1-0, My Tournament 2021


1. = (0.00): 80...Kg7
2. = (0.00): 80...Nh7
3. +- (6): 80...Ke7
4. +- (6): 80...Nd7
5. +- (3): 80...Ne6
6. +- (2): 80...Ng6
7. +- (1): 80...Ke8
8. +- (1): 80...Kg8

There are only 200 games in the database. Where are the other 300?
Eduard
I've been banned!
Points: 6 000,00 
Posts: 365
Joined: 04/11/2019, 2:05
Status: Offline (Active 10 Months, 1 Week, 3 Days, 17 Hours, 42 Minutes ago)
Topics: 32
Reputation: 579
1
Location: Stuttgart
Has thanked: 33 times
Been thanked: 635 times

Re: mehmet1921's books tests.

Post by Eduard »

I tune my books for games on the servers. Nobody there plays with such slow hardware. Depth 20 (first move after book) that's laughable, sorry. My 10 year old laptop is faster there.

I therefore no longer attach great importance to these tests here. I did a test myself on the server (same book, photo and rating is in another thread, Account "E N" unbeaten since 392 games on Playchess) and haven't lost a game after 108 games. So what should I do better? I always fix what is losing on servers. No more and no less. Personally, I do not use any tests with plies 20 here in the forum. I can't do anything with that on the server. Example: There is someone who tests my books on the server and sometimes comes out of the book with -0.50. Even so, he wins most games (!!) with this variant. It would be counterproductive if I blocked this variant just because someone loses something with 20 plies and ponder OFF and endgame errors.
kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

If a verified line is wrong (near 0.00 or negative for White and over +0.25 for Black at the beginning of middlegame) it should be corrected, no matter if the tester used a weak architecture - that's a good find anyway.

With nowadays computers and NNUE networks a game is draw even with 1.a3 or 1.h3 that doesn't mean it's good for an opening book, anyway.
Eduard wrote: 14/09/2021, 18:46 I tune my books for games on the servers. Nobody there plays with such slow hardware. Depth 20 (first move after book) that's laughable, sorry. My 10 year old laptop is faster there.

I therefore no longer attach great importance to these tests here. I did a test myself on the server (same book, photo and rating is in another thread, Account "E N" unbeaten since 392 games on Playchess) and haven't lost a game after 108 games. So what should I do better? I always fix what is losing on servers. No more and no less. Personally, I do not use any tests with plies 20 here in the forum. I can't do anything with that on the server. Example: There is someone who tests my books on the server and sometimes comes out of the book with -0.50. Even so, he wins most games (!!) with this variant. It would be counterproductive if I blocked this variant just because someone loses something with 20 plies and ponder OFF and endgame errors.
Eduard
I've been banned!
Points: 6 000,00 
Posts: 365
Joined: 04/11/2019, 2:05
Status: Offline (Active 10 Months, 1 Week, 3 Days, 17 Hours, 42 Minutes ago)
Topics: 32
Reputation: 579
1
Location: Stuttgart
Has thanked: 33 times
Been thanked: 635 times

Re: mehmet1921's books tests.

Post by Eduard »

I could post many games where White wins more than Black in special variants (-0.50 after the last book move). Opening theory looks different, especially with fast 64 cores. Even with blocked positions, the faster hardware usually wins, even if the opponent sees himself +0.50 ahead. That's why I stopped relying on slow hardware games. You are welcome to do it differently, but with fast hardware you will only play a draw because everything is blurred in the draw zone.

I can always optimize a book for weak hardware, where your book will no longer win. But that doesn't make much sense, because with fast hardware I would only play a draw. You do your books commercially, I think. Such book tests are therefore important to you. But I want to play successfully on the servers, and what wins there is of course more important to me than tests like this one.

Oh, that's all the more true for Lichess, because I only play with BIN books there. I have a great rating there! What wins in online chess is good. That is my motto.
kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

The latest statistics I've made to the new Playchess games (60.000 or so) give 98% draws smile26 In 7 years of Playchess computer playing I've never seen such a thing. Playing to win looks now almost meaningless, unless they'll find something in the programs to win them, but it will be harder time after time.

We were on almost the same situation before the NNUE arrival, where the best standard was a great book and asmFish: mostly draws, even on weak hardwares. Then NNUE arrived and found the holes in the once playing, getting a score enhancement of 100 Elo points. Maybe the next generation programs will find holes over the NNUE ones and gain other 100 points, but that doesn't means they will be effectively 100 poins stronger; it's like a match between two books in which one exploits a line error of the other and gets 23 wins on 100: a big Elo up in the table but that's not the effective score because on the other openings maybe both the books are pretty the same.

Eduard wrote: 14/09/2021, 21:29 I could post many games where White wins more than Black in special variants (-0.50 after the last book move). Opening theory looks different, especially with fast 64 cores. Even with blocked positions, the faster hardware usually wins, even if the opponent sees himself +0.50 ahead. That's why I stopped relying on slow hardware games. You are welcome to do it differently, but with fast hardware you will only play a draw because everything is blurred in the draw zone.

I can always optimize a book for weak hardware, where your book will no longer win. But that doesn't make much sense, because with fast hardware I would only play a draw. You do your books commercially, I think. Such book tests are therefore important to you. But I want to play successfully on the servers, and what wins there is of course more important to me than tests like this one.

Oh, that's all the more true for Lichess, because I only play with BIN books there. I have a great rating there! What wins in online chess is good. That is my motto.
Eduard
I've been banned!
Points: 6 000,00 
Posts: 365
Joined: 04/11/2019, 2:05
Status: Offline (Active 10 Months, 1 Week, 3 Days, 17 Hours, 42 Minutes ago)
Topics: 32
Reputation: 579
1
Location: Stuttgart
Has thanked: 33 times
Been thanked: 635 times

Re: mehmet1921's books tests.

Post by Eduard »

There are still variants that you can win with. However, you need strong hardware (32 or 64 cores) for this. With weak hardware you will also lose these variants. So some things are also a matter of hardware.

I am developing such a book in CTG format for Playchess.

I actually developed my BIN book for Lichess. There are a lot of weaker engines playing there, because Lichess is not the place where only Stockfish is present. Many of own developments can also be seen there. A lame book that shows an eval in the range of + - 0.25 for both sides does not make sense. I want to win. My games on Lichess are right, I am at the top in all disciplines.

Book duels here in the forum are irrelevant, although the book achieves very good results. Better than I expected.

I could make the book even better for these duels here, but then it would only play a draw, and it would be worse for Lichess.

I will therefore not publish any more books here. My BIN books are made for Lichess, and if you want to play with them on Lichess, you will find them on my homepage in the future.

For playing on PlayChess I recommend CTG books, these are much better than the BIN format.
mehmet1921

Top contribute Forum
Forum Contributions
Points: 30 679,00 
Posts: 1002
Joined: 04/11/2019, 16:00
Status: Online (Active 2 Hours, 16 Minutes ago)
Medals: 1
Topics: 36
Reputation: 3534
Location: Turkey
Has thanked: 2645 times
Been thanked: 3860 times

Re: mehmet1921's books tests.

Post by mehmet1921 »

Sf14-090921.bin vs Cevdet.bin:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 130721 S : 2400 8 8 300 50.0 % 2400 95.3 %
2 Cfish 130721 C : 2400 8 8 300 50.0 % 2400 95.3 %

Individual statistics:

1 Cfish 130721 S : 2400 300 (+ 7,=286,- 7), 50.0 %

Cfish 130721 C : 300 (+ 7,=286,- 7), 50.0 %

2 Cfish 130721 C : 2400 300 (+ 7,=286,- 7), 50.0 %

Cfish 130721 S : 300 (+ 7,=286,- 7), 50.0 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 3 min TC, 512 Mb Hash, Ponder Off, BestMove=Off / BookDepth=Max
Cfish 130721 S (Sf14-090921 "Skynet"), Cfish 130721 C (Cevdet: Cevdet Sarı)
https://www.mediafire.com/file/fmv5huubsl82xyv/r130.pgn/file
mehmet1921

Top contribute Forum
Forum Contributions
Points: 30 679,00 
Posts: 1002
Joined: 04/11/2019, 16:00
Status: Online (Active 2 Hours, 16 Minutes ago)
Medals: 1
Topics: 36
Reputation: 3534
Location: Turkey
Has thanked: 2645 times
Been thanked: 3860 times

Re: mehmet1921's books tests.

Post by mehmet1921 »

Goi X.bin vs 1337chess PRO-NN 25082021.ctg:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 2.30 bmi2 G : 2401 9 2 200 50.2 % 2399 99.5 %
2 SugaR AI 2.30 bmi2 1 : 2399 2 9 200 49.8 % 2401 99.5 %

Individual statistics:

1 SugaR AI 2.30 bmi2 G : 2401 200 (+ 1,=199,- 0), 50.2 %

SugaR AI 2.30 bmi2 1 : 200 (+ 1,=199,- 0), 50.2 %

2 SugaR AI 2.30 bmi2 1 : 2399 200 (+ 0,=199,- 1), 49.8 %

SugaR AI 2.30 bmi2 G : 200 (+ 0,=199,- 1), 49.8 %


Game Conditions: Fritz 17 Gui, 6 Cores (Core-i7 9750h), 2 min TC, 1024 Mb Hash, Ponder Off,Experience File Off
Book Settings: Both books played with author's recommended settings.
SugaR AI 2.30 bmi2 G (Goi X: Massimiliano Goi ), SugaR AI 2.30 bmi2 1 (1337chess PRO-NN 25082021: Michael Wagner)
https://www.mediafire.com/file/ywil42hldvwzz2z/r131.pgn/file

Goi managed to win the match of these two paid books, which are two of the most strong opening books in the world, albeit with difficulty.
1337chess PRO opening book is not just a paid book, it is the most expensive opening book on the market. It would not be right to make a decision about such books based only on book-book matches. It cannot be denied that this book, which has a serious clientele among standart and correspondence chess players, makes a serious contribution to the players who buy it.
mehmet1921

Top contribute Forum
Forum Contributions
Points: 30 679,00 
Posts: 1002
Joined: 04/11/2019, 16:00
Status: Online (Active 2 Hours, 16 Minutes ago)
Medals: 1
Topics: 36
Reputation: 3534
Location: Turkey
Has thanked: 2645 times
Been thanked: 3860 times

Re: mehmet1921's books tests.

Post by mehmet1921 »

NimaTiv 4.1.ctg vs 1337chess PRO-NN -25082021.ctg:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 2.30 bmi2 1 : 2400 5 5 200 50.0 % 2400 99.0 %
2 SugaR AI 2.30 bmi2 N : 2400 5 5 200 50.0 % 2400 99.0 %

Individual statistics:

1 SugaR AI 2.30 bmi2 1 : 2400 200 (+ 1,=198,- 1), 50.0 %

SugaR AI 2.30 bmi2 N : 200 (+ 1,=198,- 1), 50.0 %

2 SugaR AI 2.30 bmi2 N : 2400 200 (+ 1,=198,- 1), 50.0 %

SugaR AI 2.30 bmi2 1 : 200 (+ 1,=198,- 1), 50.0 %


Game Conditions: Fritz 17 Gui, 6 Cores (Core-i7 9750h), 2 min TC, 1024 Mb Hash, Ponder Off,Experience File Off
Book Settings: Both books played with author's recommended settings.
SugaR AI 2.30 bmi2 1 (1337chess PRO-NN 25082021: Michael Wagner), SugaR AI 2.30 bmi2 N (NimaTiv 4.1: Ian Hurnavich)
https://www.mediafire.com/file/w18ztwulbqvios2/r132.pgn/file

Nimativ 4.1 made a very strong defense against the world's most expensive opening book and came out of this match in a draw.
mehmet1921

Top contribute Forum
Forum Contributions
Points: 30 679,00 
Posts: 1002
Joined: 04/11/2019, 16:00
Status: Online (Active 2 Hours, 16 Minutes ago)
Medals: 1
Topics: 36
Reputation: 3534
Location: Turkey
Has thanked: 2645 times
Been thanked: 3860 times

Re: mehmet1921's books tests.

Post by mehmet1921 »

Big 3 Tournament:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 2.30 bmi2 G : 2401 4 1 400 50.1 % 2400 99.8 %
2 SugaR AI 2.30 bmi2 N : 2400 2 2 400 50.0 % 2400 99.5 %
3 SugaR AI 2.30 bmi2 1 : 2399 3 3 400 49.9 % 2400 99.2 %

Individual statistics:

1 SugaR AI 2.30 bmi2 G : 2401 400 (+ 1,=399,- 0), 50.1 %

SugaR AI 2.30 bmi2 N : 200 (+ 0,=200,- 0), 50.0 %
SugaR AI 2.30 bmi2 1 : 200 (+ 1,=199,- 0), 50.2 %

2 SugaR AI 2.30 bmi2 N : 2400 400 (+ 1,=398,- 1), 50.0 %

SugaR AI 2.30 bmi2 G : 200 (+ 0,=200,- 0), 50.0 %
SugaR AI 2.30 bmi2 1 : 200 (+ 1,=198,- 1), 50.0 %

3 SugaR AI 2.30 bmi2 1 : 2399 400 (+ 1,=397,- 2), 49.9 %

SugaR AI 2.30 bmi2 N : 200 (+ 1,=198,- 1), 50.0 %
SugaR AI 2.30 bmi2 G : 200 (+ 0,=199,- 1), 49.8 %


Game Conditions: Fritz 17 Gui, 6 Cores (Core-i7 9750h), 2 min TC, 1024 Mb Hash, Ponder Off,Experience File Off
Book Settings: Both books played with author's recommended settings.
SugaR AI 2.30 bmi2 G (Goi X: Massimiliano Goi ),
SugaR AI 2.30 bmi2 N (NimaTiv 4.1: Ian Hurnavich)
SugaR AI 2.30 bmi2 1 (1337chess PRO-NN 25082021: Michael Wagner)
https://www.mediafire.com/file/jo2fl3gl2yfkh5i/r133.pgn/file

There is no need to comment too much on these three books. It's obvious how strong they are.
As a result of the whole tests, Goi X turned out to be much stronger than I expected. As a free book, Nimativ 4.1 has shown incredible success with only one defeat in 400 games against these two paid opening books.
kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

Thank you for your testings, Mehmet, your work is well appreciated. thankyou

With the outcoming of Nimativ 5, I ask you if you could try my new Goi beta ctg.
mehmet1921

Top contribute Forum
Forum Contributions
Points: 30 679,00 
Posts: 1002
Joined: 04/11/2019, 16:00
Status: Online (Active 2 Hours, 16 Minutes ago)
Medals: 1
Topics: 36
Reputation: 3534
Location: Turkey
Has thanked: 2645 times
Been thanked: 3860 times

Re: mehmet1921's books tests.

Post by mehmet1921 »

Manifest 070921.ctg v s Goi X.bin:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 2.30 bmi2 G : 2403 10 8 200 50.7 % 2397 96.5 %
2 SugaR AI 2.30 bmi2 M : 2397 8 10 200 49.2 % 2403 96.5 %

Individual statistics:

1 SugaR AI 2.30 bmi2 G : 2403 200 (+ 5,=193,- 2), 50.8 %

SugaR AI 2.30 bmi2 M : 200 (+ 5,=193,- 2), 50.7 %

2 SugaR AI 2.30 bmi2 M : 2397 200 (+ 2,=193,- 5), 49.2 %

SugaR AI 2.30 bmi2 G : 200 (+ 2,=193,- 5), 49.2 %


Game Conditions: Fritz 17 Gui, 6 Cores (Core-i7 9750h), 2 min TC, 1024 Mb Hash, Ponder Off,Experience File Off
Book Settings: Both books played with author's recommended settings.
SugaR AI 2.30 bmi2 G (Goi X: Massimiliano Goi ), SugaR AI 2.30 bmi2 M (Manifest 070921: Peter Grayson)
https://www.mediafire.com/file/9h729s93ozn9vgs/r134.pgn/file
kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

Hrllo. Did you see the lost above?
Thank you for your testings, Mehmet, your work is well appreciated. thankyou

With the outcoming of Nimativ 5, I ask you if you could try my new Goi beta ctg.

mehmet1921 wrote: 20/09/2021, 14:34 Manifest 070921.ctg v s Goi X.bin:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 2.30 bmi2 G : 2403 10 8 200 50.7 % 2397 96.5 %
2 SugaR AI 2.30 bmi2 M : 2397 8 10 200 49.2 % 2403 96.5 %

Individual statistics:

1 SugaR AI 2.30 bmi2 G : 2403 200 (+ 5,=193,- 2), 50.8 %

SugaR AI 2.30 bmi2 M : 200 (+ 5,=193,- 2), 50.7 %

2 SugaR AI 2.30 bmi2 M : 2397 200 (+ 2,=193,- 5), 49.2 %

SugaR AI 2.30 bmi2 G : 200 (+ 2,=193,- 5), 49.2 %


Game Conditions: Fritz 17 Gui, 6 Cores (Core-i7 9750h), 2 min TC, 1024 Mb Hash, Ponder Off,Experience File Off
Book Settings: Both books played with author's recommended settings.
SugaR AI 2.30 bmi2 G (Goi X: Massimiliano Goi ), SugaR AI 2.30 bmi2 M (Manifest 070921: Peter Grayson)
https://www.mediafire.com/file/9h729s93ozn9vgs/r134.pgn/file
Hurnavich

Top contribute Forum Book Maker Top Active Users
Administrators
Points: 37 607,00 
Forum Contributions
Posts: 2870
Joined: 01/11/2019, 14:33
Status: Online (Active 2 Hours ago)
Medals: 3
Topics: 112
Reputation: 10421
Has thanked: 3811 times
Been thanked: 8972 times

Re: mehmet1921's books tests.

Post by Hurnavich »

NimaTiv 5.1 tonight all been well some corrections additions about 15 elo gains in my tests ill send you link to test against goi beta.

Hand shake Goi
"May your next game be your best"
kramnik

Top contribute Forum
Forum Contributions
Points: 40 305,00 
Posts: 1924
Joined: 04/11/2019, 14:45
Status: Offline (Active 3 Months, 2 Days, 17 Hours, 43 Minutes ago)
Medals: 1
Topics: 71
Reputation: 2388
Location: North-Italy
Has thanked: 1185 times
Been thanked: 2949 times

Re: mehmet1921's books tests.

Post by kramnik »

mehmet: please use the book I gave you through the private message against NimaTiv 5.1 and not the free version I've just published here. (old ones)

I expect the ctg version to underperform a bit since the compiler does not truncate the losing moves as Polyglot does instead.
Post Reply

Return to “Books Tests and Tournaments”