Oliprog's Engines and Book Tests

Post by **Hurnavich** » 03/04/2020, 21:53

Greetings,

There are many good quality engine test suites available created for the purpose of really getting a good idea of an engines ability.

When I run engine vs engine they are what I use
When I experiment I will have a set opening book say on the C54 and have it balanced and set to exit in the gui at as equal as possible.

And have the engine to play repeat from both sides
This then helps to identify which engine is better suited to certain openings this then helps to pick the best engine later to tune my books in favoured openings.

Further to this running many private matches of 16 min pluse 4 sec with engine pruning changes can unearth many treasures to add to books.

kramnik · Post by **kramnik** » 03/04/2020, 23:20

Boring but effective and no, books should not be used especially in the case of such small number of games: book variants can alter the balance of the match if the opening randomization is high. Furthermore, if it's a small book with few lines ending with 0.00 there is a great possibility (over 98%) that the games will end as draws.

Peter Grayson wrote:With the way Oliprog runs the engine matches with a book, both engines play both sides of the same opening and it is a good exercise to see how the engines' handling of the opening may differ. It comes down to personal choice when engine versus engine matches playing without books are, from my perspective, usually no spectacle and very boring affairs. Engines should be tested using contemporary openings to maintain interest in their development. Certainly at the fast time controls and relatively modest hardware most people use, the engine's standard of opening play leaves much to be desired. The main purpose of an engine is surely for what it offers in the middle game and beyond?

Peter

kramnik · Post by **kramnik** » 03/04/2020, 23:23

Hurnavich wrote:When I run engine vs engine they are what I use
When I experiment I will have a set opening book say on the C54 and have it balanced and set to exit in the gui at as equal as possible.

This messes up the test, especially if the variants are longer than 5-7 moves (10-14 plies): the book will bring many draws.

Peter Grayson · Post by **Peter Grayson** » 04/04/2020, 2:09

kramnik wrote: book variants can alter the balance of the match

As does the random variability caused by the use of SMP engines playing from move 1. The use of deterministic single core engines should produce repetitive games that reveal nothing about the engine. If they produce variability from move 1 using the same hash setting and time control there is something wrong with the set up.

For testing engines, ideally the opening line/book assistance should stop at a point where there are move choices whilst retaining one or more thematic continuations from those moves. For example closed or open openings, opposite side castling for rapid attack for the side with the initiative or control of a wing or the centre. Engines reliant on deep look ahead with heavy pruning produce little in the way of themed openings that is the artistic work from players over the centuries.

Perhaps the discussion should be moved elsewhere rather than take up space on Oliprog' s Engine and Book Tests.

Peter

Oliprog · Post by **Oliprog** » 04/04/2020, 14:57

Scorpio 200331 vs. Fres

Processor AMD A10 – 9700 R7,10 Compute Cores 4C + 6G - 3.50 GHz - RAM 8.00 GB - 64-bit operating system - x64 processor - Windows 10 - 2048 mb hash - Komodo 13 GUI – opening book: Scorpio 200331 vs. Fres - final tablebases: Syzygy 3-4-5 pieces - both engines play with 4 threads and large pages

Book Settings:

Scorpio 200331

1.Tournament ON
2.Min games 1-100
3.variety of play 50%
4.influence of learn 100%
5.Learning strength 500%

Fres

1. Tournament ON
2. Min games 0-100
3. Variety of play 50%
4. Influence of learn 100%
5. Learning strength 100%

DESKTOP-TORM2PL, Blitz 5min 0

1 Brainfish 220320 64 POPCNT Fr +14 +15/=204/-6 52.00% 117.0/225
2 Brainfish 220320 64 POPCNT Sc -14 +6/=204/-15 48.00% 108.0/225

Games in PGN: http://www.mediafire.com/folder/k9t66wzappjj3/Scorpio_200331_vs._Fres

Congratulations Cubail!

Scorpio · Post by **Scorpio** » 04/04/2020, 18:44

I've returned to school and got teachead by... cubail.

I'm also glad to see you back!

I hope that the new GUI keeps you safe of headaches, Oliprog.
Thanks for including me at your test routine.

tacos40 · Post by **tacos40** » 05/04/2020, 18:51

Hello Oliprog

: I hope you are well / healthy, take care of yourself and family

New Vitamin published (dual: BIN+CTG within same engine)
=> http://outskirts.altervista.org/forum/viewtopic.php?p=6885&view=single_post#p6885

- If possible / if you want ... test Vitamin (dual) against any other book of your choice. Thank you!
- Readme+image inside compressed file, but ask whatever you need, in case of doubts

Oliprog · Post by **Oliprog** » 05/04/2020, 19:39

tacos40 wrote:Hello Oliprog : I hope you are well / healthy, take care of yourself and family

New Vitamin published (dual: BIN+CTG within same engine)
=> http://outskirts.altervista.org/forum/viewtopic.php?p=6885&view=single_post#p6885

- If possible / if you want ... test Vitamin (dual) against any other book of your choice. Thank you!
- Readme+image inside compressed file, but ask whatever you need, in case of doubts

Hello tacos40! Thank you for your concern, for here all my family well I hope it is the same for yours. Never try a dual book and it will be a good opportunity to do so.

A big greeting

Oliprog · Post by **Oliprog** » 06/04/2020, 18:42

Honey XI-r1 vs. Stockfish 220320 64 BMI1

Processor AMD A10 – 9700 R7,10 Compute Cores 4C + 6G - 3.50 GHz - RAM 8.00 GB - 64-bit operating system - x64 processor - Windows 10 - 2048 mb hash – Fritz 16 GUI – opening book: SALC V5.03 fc 10 moves 5000 Opening Test Suite - final tablebases: Syzygy 3-4-5 pieces - both engines play with 4 threads and no large pages

DESKTOP-TORM2PL, Blitz 5min 0

1 Honey XI-r1 +25 +25/=84/-16 53.60% 67.0/125
2 Stockfish 220320 64 BMI1 -25 +16/=84/-25 46.40% 58.0/125

Games in PGN: http://www.mediafire.com/folder/tfoe2zyd0c2e6/Engmatch_-_9_-_1055

Congratulations Honey Team!

Oliprog · Post by **Oliprog** » 07/04/2020, 16:59

Vitamin 20 Dual vs. Zeus fixed bin

Processor AMD A10 – 9700 R7,10 Compute Cores 4C + 6G - 3.50 GHz - RAM 8.00 GB - 64-bit operating system - x64 processor - Windows 10 - 2048 mb hash - Fritz 16 GUI – opening book: Vitamin 20 Dual vs. Zeus fixed bin - final tablebases: Syzygy 3-4-5 pieces - both engines play with 4 threads and large pages

Book Settings:

Vitamin 20 Dual

Bin book:

Book Depth 40
Best Book Line OFF

Ctg book:

1.Tournament ON
2.Min games 0-40
3.variety of play 120
4.influence of learn 45
5.Learning strength 0

Zeuz fixed

Best Book Line OFF

DESKTOP-TORM2PL, Blitz 5min 0

1 Brainfish 220320 64 POPCNT Ze +11 +10/=171/-4 51.62% 95.5/185
2 Brainfish 220320 64 POPCNT Vi -11 +4/=171/-10 48.38% 89.5/185

Games in PGN: http://www.mediafire.com/folder/t7iu3scu2gwue/Vitamin_20_Dual_vs._Zeus_fixed_bin

Congratulations Kerveros!

tacos40 · Post by **tacos40** » 07/04/2020, 17:28

Hello Oliprog:
Thank you for the test

Tech note: afaik.. Large page should be set OFF in both engines to make match more reliable / fair for BOTH engines
In low RAM systems, memory is fragmented and an engine will enjoy unfragmented/large pages, the other will not
I dont think this affects too much the result in this case anyway, but who knows...

Oliprog · Post by **Oliprog** » 07/04/2020, 17:37

tacos40 wrote:Hello Oliprog:
Thank you for the test

Tech note: afaik.. Large page should be set OFF in both engines to make match more reliable / fair for BOTH engines
In low RAM systems, memory is fragmented and an engine will enjoy unfragmented/large pages, the other will not
I dont think this affects too much the result in this case anyway, but who knows...

Hello tacos40! Your suggestions are always welcome, but do you mean the dual book tests or all the book tests in which it is convenient to disable long pages?
Regards

tacos40 · Post by **tacos40** » 07/04/2020, 17:47

Oliprog wrote:
tacos40 wrote:Hello Oliprog:
Thank you for the test

Tech note: afaik.. Large page should be set OFF in both engines to make match more reliable / fair for BOTH engines
In low RAM systems, memory is fragmented and an engine will enjoy unfragmented/large pages, the other will not
I dont think this affects too much the result in this case anyway, but who knows...

Hello tacos40! Your suggestions are always welcome, but do you mean the dual book tests or all the book tests in which it is convenient to disable long pages?
Regards

- No: this recommendation is in general, any test of books using same engine: CTG vs BIN or BIN vs BIN or DUAL vs etc
- e.g: Brainfish_AAA_book.exe versus Branfish_CCC_book.exe => UNCHECK "Large Pages" in both engines

However, in very exceptional cases, "Large Pages" must be set ON
- For example engine versus engine match, Brainfish versus SF TS (?) or some other engines where Large Pages were enabled but HIDDEN

Peter Grayson · Post by **Peter Grayson** » 08/04/2020, 13:09

tacos40 wrote:
Tech note: afaik.. Large page should be set OFF in both engines to make match more reliable / fair for BOTH engines
In low RAM systems, memory is fragmented and an engine will enjoy unfragmented/large pages, the other will not
I dont think this affects too much the result in this case anyway, but who knows...

very valid point. On my 8Gb RAM machines, since updating to Windows 10 Pro I have never been able to get an engine to initialise with 4 Gb hash when the best is 2Gb and if that engine is unloaded and other work is done and then the GUI reloaded the RAM has fragmented further to limit LP hash to 1Gb.

In a match using the Tournament mode of the Chessbase GUI the sequence of engine handling is engines are loaded, initialised, game completed and then engines unloaded for each game. With time the amount of unfragmented RAM available may reduce dependant on the amount of RAM on the Machine. With 2Gb LP hash setting for two engines on an 8Gb RAM system it is unlikely both engines would start with LP hash and most likely finish with just one or neither using LP hash. The benfit of LP hash is approximately a 10% speed improvement for the engine (s) using it.

However if "Lock pages in memory" has not been set in the Windows permissions configuration it will not matter because Large Pages memory will not be available.

Peter

Oliprog · Post by **Oliprog** » 09/04/2020, 18:17

Bluefish Lc0 v0.24.1+git.0404 vs. Honey XI-r1

Processor AMD A10 – 9700 R7,10 Compute Cores 4C + 6G - 3.50 GHz - RAM 8.00 GB - 64-bit operating system - x64 processor - Windows 10 - 2048 mb hash – Fritz 16 GUI – opening book: TWIC Books 200301 Games ctg - final tablebases: Syzygy 3-4-5 pieces - both engines play with 4 threads and no large pages

Book Settings:

Tournament Book UNCHECKED
Minimum Games 3
Maximum Moves 40
Variety of play setting 150
Influence of Learn Value 25
Learning Strength 125

DESKTOP-TORM2PL, Blitz 5min 0

1 Lc0 v0.24.1+git.0404 +2 +9/=143/-8 50.31% 80.5/160
2 Honey XI-r1 -2 +8/=143/-9 49.69% 79.5/160

Games in PGN: http://www.mediafire.com/folder/htgxoebmsv5el/Engmatch_-_9_-_1056

Congratulations to Bluefish author Darin Pann!

Outskirts CheSS ForuM

Outskirts CheSS ForuM