experience file vs opening book : the right way !

[chris]

Before the competition, training !

It seems so logical yet i have already seen few tournaments where the organizers did not train their engine before throwing them into the arena...

The right way :

- if you plan a tourney where there will be a list of openings, train your engine on each opening (500 to 1000 games per opening, several sparring-partners)

- if you plan a tourney against opponents which will use opening books, train your engine against each opening book (see below)

- if we plan to participate in a tournament where the opponent will have longer TC or more threads, reinforce your experience data

[chris]

hardware : dual xeon e5-2660v3
software : cutechess-gui 1.2.0
settings : TC 1m+1s, 40 threads, hash 4096 MB, partial syzygy 7men

engines :
- Eman 8.10 played with last green net and experience data from 6 learning sessions
- Eman 8.30 played with last green net and reinforced experience data from 8 learning sessions

opening books :
- Hubble 3.1 by mehmet1921 (31 MB, 2 037 401 moves)
- Mantis-DH1 by ConRog (30.2 MB, 1 984 383 moves)
- Solista-010722 by Eduard (28.1 MB, 1 846 197 moves)

opening lists :
- 100 games from the start position
- 30 games from the GHPPN's 15 most popular openings
- 40 games from the 20 first legal moves
- 50 games from the Albert Silver's 25 gambit suit
- 100 games from the Jeroen Noomen's 50 tcec season 22 superfinal openings

individual results :

Code: Select all

   # PLAYER                            :  RATING  ERROR  POINTS  PLAYED   (%)    W    D    L  D(%)  OppAvg  OppN
   1 Depth4_050522.exp (Eman 8.10)     :      39     27   177.5     320  55.5   42  271    7  84.7       0     1
   2 Solista-010722.bin (Eman 8.10)    :       0   ----   142.5     320  44.5    7  271   42  84.7      39     1

White advantage = 23.31 +/- 13.36
Draw rate (equal opponents) = 50.00 % +/- 0.00

Code: Select all

   # PLAYER                           :  RATING  ERROR  POINTS  PLAYED   (%)    W    D    L  D(%)  OppAvg  OppN
   1 Depth4_031222.exp (Eman 8.30)    :      39     27   177.5     320  55.5   43  269    8  84.1       0     1
   2 Hubble 3.1.bin (Eman 8.30)       :       0   ----   142.5     320  44.5    8  269   43  84.1      39     1

White advantage = 21.09 +/- 13.34
Draw rate (equal opponents) = 50.00 % +/- 0.00

Code: Select all

   # PLAYER                           :  RATING  ERROR  POINTS  PLAYED   (%)    W    D    L  D(%)  OppAvg  OppN
   1 Depth4_031222.exp (Eman 8.30)    :      36     27   176.5     320  55.2   40  273    7  85.3       0     1
   2 Mantis-DH1.bin (Eman 8.30)       :       0   ----   143.5     320  44.8    7  273   40  85.3      36     1

White advantage = 25.51 +/- 13.39
Draw rate (equal opponents) = 50.00 % +/- 0.00

link : games
pass : 59Ef9tOGT1RCBGYgpMTuzw

[chris]

By grouping the engines with experience files VS the engines with opening book, we obtain this overall result :

Code: Select all

   # PLAYER                         :  RATING  ERROR  POINTS  PLAYED   (%)    W    D    L  D(%)  OppAvg  OppN
   1 experience_file (Eman 8.xx)    :      38     16   531.5     960  55.4  125  813   22  84.7       0     1
   2 opening_books (Eman 8.xx)      :       0   ----   428.5     960  44.6   22  813  125  84.7      38     1

White advantage = 23.30 +/- 8.00
Draw rate (equal opponents) = 50.00 % +/- 0.00

Some stats :

Code: Select all

experience_file (eman 8.xx) : avg. D32, 1 979 ms/move, 960 games, 50 394 moves
opening_books (eman 8.xx)   : avg. D34, 2 414 ms/move, 960 games, 40 917 moves

Averages :
207 sec/game (3 min/game)
113 plies/game (95 played plies/game)

[chris]

How to read the moves of the games of the PGN files ?

Code: Select all

[Event "EXP VS BOOK"]
[Site "dual xeon e5-2660v3"]
[Date "2022.12.13"]
[Round "1"]
[White "Mantis-DH1 (Eman 8.30)"]
[Black "Depth4_031222 (Eman 8.30)"]
[Result "1/2-1/2"]
[ECO "B40"]
[GameDuration "00:01:54"]
[Opening "Sicilian"]
[PlyCount "44"]
[TimeControl "60+1"]
[Variation "Anderssen Variation"]

1. e4 {book} c5 {book} 2. Nf3 {book} e6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} Nf6 {book} 5. Nc3 {book} d6 {book} 6. Be3 {book} Be7 {book}
7. g4 {book} h6 {book} 8. Qe2 Nc6 {+1.91/30 7.7s} 9. O-O-O Nxd4 {+1.89/23 0.58s}
10. Bxd4 {+2.04/33 9.6s} Bd7 {+1.87/28 1.5s} 11. f3 {+2.08/32 7.1s} Rc8 {+1.95/30 9.7s}
12. Qd2 {+2.10/32 6.2s} e5 {+1.94/27 3.9s} 13. Be3 {+2.00/32 6.3s} Qa5 {+1.87/29 5.1s}
14. a3 {+1.94/26 1.6s} a6 {+1.91/33 7.1s} 15. h4 {+2.05/29 7.4s} Be6 {+1.78/26 1.8s}
16. Be2 {+2.09/24 1.2s} b5 {+2.11/28 8.8s} 17. Kb1 g6 {+2.22/23 0.86s}
18. Rhe1 Rc6 {+1.90/26 4.7s} 19. Qd3 {+2.16/29 13s} Bc4 {+1.93/22 0.74s}
20. Qd2 {+2.19/27 1.1s} Be6 {+2.12/26 1.2s}
21. Rh1 {+2.19/29 2.2s} Rc8 {+1.90/25 1.2s}
22. Rhe1 Rc6 {0.00/45 1.1s, Draw by 3-fold repetition} 1/2-1/2

CUTECHESS-GUI adds the {book} mention after each move of the current opening.
For example : 1. e4 {book}...

When the engine plays a book's move, there is no evaluation.
For example : 8. Qe2 ...

When the engine is out-of-book, CUTECHESS-GUI adds the engine's evaluation.
For example : 10. Bxd4 {+2.04/33 9.6s} ...

CUTECHESS-GUI always adds the evaluation of the engine with experience data.
For example : ... Nc6 {+1.91/30 7.7s} ...

[chris]

Why did you choose these 3 opening books (or these 3 book's authors) ?

As i knew nothing about the opening books nor their performance nor their rating, at the time i was inspired by this Hurnavich's tourney.
Then, i searched for the last available releases of the books from this rating list.
Starting from the best, at the time, i had found these ones.

[chris]

Why did you choose a TC 1m+1s ?

It favors better the engine with the opening book, examples :
- with a TC 1m+1s, if the engine plays 30 book's moves, it increments its clock by 30s => +50% more time.
- with a TC 4m+2s, if the engine plays 30 book's moves, it increments its clock by 60s => +25% more time.

You should know that :
- in chess, the perfect game is a draw so the more time you give the engines the less difference there will be between them.
- even if the engine was trained with a TC 1m+1s on 7 threads, its learned experience data is really very resilient (odd tc).

[chris]

How was the experience file trained ?

Initially, i trained Eman on the openings where he had lost during my reference tournament (100 openings from salc_v3_10moves).
Then i trained Eman on the openings where the other engines did better (ex : eman drew when others won, eman lost when others drew, etc).

After each tournament, Eman analyzed its defeats at 10min/move then also the defeats of the other engines.

It's no longer a secret but learned data is better when an engine trains against several opponents because it faces different styles of play.
Since 2 years at 24/7 on few hosts, Eman trains until 4 sparring-partners who also have a learning feature so they don't repeat the same bad lines and try new ones.

[chris]

What does reinforced experience data mean ?

reinforced = analyzed with greater depth / more time

When the engines train, they try several moves and depending on their efficiency, they forget them more or less quickly.
It generates some 'secondary' experience data which contains moves with shallow depth => imprecise evaluation.

In addition, depending on the TC and the threads number, the average depth is not very high so the resistance of the experience data is less important.

So it is necessary to reinforce this 'secondary' experience data by analyzing the moves which have a shallow depth.
For example, i analyzed at D40 all the moves of a hundred lines for each trained opening.
It improved the resistance of experience data.