Stockfish-NNUE
Moderators: Elijah, Igbo, timetraveller
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Wed Jan 19 , 2022 .
Thanks to Author: ppigazzini
Download : https://abrok.eu/stockfish/
Timestamp: 1642616470
Add msys2 Clang x86_64 to GitHub Action matrix
Also use Windows Server 2022 virtual environment for msys2 builds.
closes https://github.com/official-stockfish/Stockfish/pull/3893
No functional change
see source
Thanks to Author: ppigazzini
Download : https://abrok.eu/stockfish/
Timestamp: 1642616470
Add msys2 Clang x86_64 to GitHub Action matrix
Also use Windows Server 2022 virtual environment for msys2 builds.
closes https://github.com/official-stockfish/Stockfish/pull/3893
No functional change
see source
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Thanks to the Author: ppigazzini
Date: Wed Jan 19 22:26:20 2022 +0100
Timestamp: 1642627580
Improve Makefile for Windows native builds
A Windows Native Build (WNB) can be done:
- on Windows, using a recent mingw-w64 g++/clang compiler
distributed by msys2, cygwin and others
- on Linux, using mingw-w64 g++ to cross compile
Improvements:
- check for a WNB in a proper way and set a variable to simplify the code
- set the proper EXE for a WNB
- use the proper name for the mingw-w64 clang compiler
- use the static linking for a WNB
- use wine to make a PGO cross compile on Linux (also with Intel SDE)
- enable the LTO build for mingw-w64 g++ compiler
- set `lto=auto` to use the make's job server, if available, or otherwise
to fall back to autodetection of the number of CPU threads
- clean up all the temporary LTO files saved in the local directory
Tested on:
- msys2 MINGW64 (g++), UCRT64 (g++), MINGW32 (g++), CLANG64 (clang)
environments
- cygwin mingw-w64 g++
- Ubuntu 18.04 & 21.10 mingw-w64 PGO cross compile (also with Intel SDE)
closes #3891
No functional change
see source
DOWNLOAD : https://abrok.eu/stockfish/
Date: Wed Jan 19 22:26:20 2022 +0100
Timestamp: 1642627580
Improve Makefile for Windows native builds
A Windows Native Build (WNB) can be done:
- on Windows, using a recent mingw-w64 g++/clang compiler
distributed by msys2, cygwin and others
- on Linux, using mingw-w64 g++ to cross compile
Improvements:
- check for a WNB in a proper way and set a variable to simplify the code
- set the proper EXE for a WNB
- use the proper name for the mingw-w64 clang compiler
- use the static linking for a WNB
- use wine to make a PGO cross compile on Linux (also with Intel SDE)
- enable the LTO build for mingw-w64 g++ compiler
- set `lto=auto` to use the make's job server, if available, or otherwise
to fall back to autodetection of the number of CPU threads
- clean up all the temporary LTO files saved in the local directory
Tested on:
- msys2 MINGW64 (g++), UCRT64 (g++), MINGW32 (g++), CLANG64 (clang)
environments
- cygwin mingw-w64 g++
- Ubuntu 18.04 & 21.10 mingw-w64 PGO cross compile (also with Intel SDE)
closes #3891
No functional change
see source
DOWNLOAD : https://abrok.eu/stockfish/
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Thu Jan 20 , 2022 .
Thanks to the Author: Joost VandeVondele
Timestamp: 1642700056
Revert -flto=auto on mingw
causes issues on some installations (glinscott/fishtest#1255).
closes https://github.com/official-stockfish/Stockfish/pull/3898
No functional change
see source
Download : https://abrok.eu/stockfish/
Thanks to the Author: Joost VandeVondele
Timestamp: 1642700056
Revert -flto=auto on mingw
causes issues on some installations (glinscott/fishtest#1255).
closes https://github.com/official-stockfish/Stockfish/pull/3898
No functional change
see source
Download : https://abrok.eu/stockfish/
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Sat Jan 22 , 2022 .
Thanks to the Author: J. Oster
Download : https://abrok.eu/stockfish/
Timestamp: 1642844904
Simplify limiting extensions.
Replace the current method for limiting extensions to avoid search getting stuck
with a much simpler method.
the test position in https://github.com/official-stockfish/Stockfish/commit/73018a03375b4b72ee482eb5a4a2152d7e4f0aac
can still be searched without stuck search.
fixes #3815 where the search now makes progress with rootDepth
shows robust behavior in a d10 search for 1M positions.
passed STC
https://tests.stockfishchess.org/tests/view/61e303e3babab931824dfb18
LLR: 2.94 (-2.94,2.94) <-2.25,0.25>
Total: 57568 W: 15449 L: 15327 D: 26792 Elo +0.74
Ptnml(0-2): 243, 6211, 15779, 6283, 268
passed LTC
https://tests.stockfishchess.org/tests/view/61e3586cbabab931824e091c
LLR: 2.96 (-2.94,2.94) <-2.25,0.25>
Total: 128200 W: 34632 L: 34613 D: 58955 Elo +0.05
Ptnml(0-2): 124, 12559, 38710, 12588, 119
closes https://github.com/official-stockfish/Stockfish/pull/3899
Bench: 4550528
see source
Thanks to the Author: J. Oster
Download : https://abrok.eu/stockfish/
Timestamp: 1642844904
Simplify limiting extensions.
Replace the current method for limiting extensions to avoid search getting stuck
with a much simpler method.
the test position in https://github.com/official-stockfish/Stockfish/commit/73018a03375b4b72ee482eb5a4a2152d7e4f0aac
can still be searched without stuck search.
fixes #3815 where the search now makes progress with rootDepth
shows robust behavior in a d10 search for 1M positions.
passed STC
https://tests.stockfishchess.org/tests/view/61e303e3babab931824dfb18
LLR: 2.94 (-2.94,2.94) <-2.25,0.25>
Total: 57568 W: 15449 L: 15327 D: 26792 Elo +0.74
Ptnml(0-2): 243, 6211, 15779, 6283, 268
passed LTC
https://tests.stockfishchess.org/tests/view/61e3586cbabab931824e091c
LLR: 2.96 (-2.94,2.94) <-2.25,0.25>
Total: 128200 W: 34632 L: 34613 D: 58955 Elo +0.05
Ptnml(0-2): 124, 12559, 38710, 12588, 119
closes https://github.com/official-stockfish/Stockfish/pull/3899
Bench: 4550528
see source
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Tue Jan 25 , 2022 .
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1643092072
Scale child node futility pruning with previous move history.
Idea is to do more futility pruning if previous move has bad histories and less if it has good histories.
passed STC
https://tests.stockfishchess.org/tests/view/61e3757fbabab931824e0db7
LLR: 2.96 (-2.94,2.94) <0.00,2.50>
Total: 156816 W: 42282 L: 41777 D: 72757 Elo +1.12
Ptnml(0-2): 737, 17775, 40913, 18212, 771
passed LTC
https://tests.stockfishchess.org/tests/view/61e43496928632f7813a5535
LLR: 2.95 (-2.94,2.94) <0.50,3.00>
Total: 349968 W: 94612 L: 93604 D: 161752 Elo +1.00
Ptnml(0-2): 300, 35934, 101550, 36858, 342
closes https://github.com/official-stockfish/Stockfish/pull/3903
bench 4720954
see source
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1643092072
Scale child node futility pruning with previous move history.
Idea is to do more futility pruning if previous move has bad histories and less if it has good histories.
passed STC
https://tests.stockfishchess.org/tests/view/61e3757fbabab931824e0db7
LLR: 2.96 (-2.94,2.94) <0.00,2.50>
Total: 156816 W: 42282 L: 41777 D: 72757 Elo +1.12
Ptnml(0-2): 737, 17775, 40913, 18212, 771
passed LTC
https://tests.stockfishchess.org/tests/view/61e43496928632f7813a5535
LLR: 2.95 (-2.94,2.94) <0.50,3.00>
Total: 349968 W: 94612 L: 93604 D: 161752 Elo +1.00
Ptnml(0-2): 300, 35934, 101550, 36858, 342
closes https://github.com/official-stockfish/Stockfish/pull/3903
bench 4720954
see source
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Sat Jan 29 , 2022 .
Thank to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1643443092
Do stats updates after LMR for captures
Since captures that are in LMR use continuation histories of corresponding quiet moves it makes sense to update this histories if this capture passes LMR by analogy to existing logic for quiet moves.
Passed STC
https://tests.stockfishchess.org/tests/view/61f367eef7fba9f1a4f1318b
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 208464 W: 56006 L: 55407 D: 97051 Elo +1.00
Ptnml(0-2): 964, 23588, 54655, 23935, 1090
Passed LTC
https://tests.stockfishchess.org/tests/view/61f41e34f7fba9f1a4f15241
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 69144 W: 18793 L: 18441 D: 31910 Elo +1.77
Ptnml(0-2): 65, 6982, 20142, 7302, 81
closes https://github.com/official-stockfish/Stockfish/pull/3910
bench 4637392
see source
Thank to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1643443092
Do stats updates after LMR for captures
Since captures that are in LMR use continuation histories of corresponding quiet moves it makes sense to update this histories if this capture passes LMR by analogy to existing logic for quiet moves.
Passed STC
https://tests.stockfishchess.org/tests/view/61f367eef7fba9f1a4f1318b
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 208464 W: 56006 L: 55407 D: 97051 Elo +1.00
Ptnml(0-2): 964, 23588, 54655, 23935, 1090
Passed LTC
https://tests.stockfishchess.org/tests/view/61f41e34f7fba9f1a4f15241
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 69144 W: 18793 L: 18441 D: 31910 Elo +1.77
Ptnml(0-2): 65, 6982, 20142, 7302, 81
closes https://github.com/official-stockfish/Stockfish/pull/3910
bench 4637392
see source
-
- Points: 0,00
- Status: Offline (Active 2 Hours, 15 Minutes ago)
- Topics:
Re: Stockfish-NNUE
SF-PB-290122_x64, Updated to "Do stats updates after LMR for captures".
Engines https://pixeldrain.com/u/rXGg8PZ3
Network https://tests.stockfishchess.org/api/nn/nn-ac07bd334b62.nnue
Engines https://pixeldrain.com/u/rXGg8PZ3
Network https://tests.stockfishchess.org/api/nn/nn-ac07bd334b62.nnue
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Sat Feb 5 , 2022 .
Thanks to the Author Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644043221
Reintroduce razoring
Razoring was simplified away some years ago, this patch reintroduces it in a slightly different form.
Now for low depths if eval is far below alpha we check if qsearch can push it above alpha - and if it can't we return a fail low.
passed STC
https://tests.stockfishchess.org/tests/view/61fbf968d508ec6a1c9f3274
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 226120 W: 61106 L: 60472 D: 104542 Elo +0.97
Ptnml(0-2): 1118, 25592, 59080, 26078, 1192
passed LTC
https://tests.stockfishchess.org/tests/view/61fcc569d508ec6a1c9f5617
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 113128 W: 30851 L: 30397 D: 51880 Elo +1.39
Ptnml(0-2): 114, 11483, 32926, 11917, 124
closes https://github.com/official-stockfish/Stockfish/pull/3921
bench 4684080
see source
Thanks to the Author Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644043221
Reintroduce razoring
Razoring was simplified away some years ago, this patch reintroduces it in a slightly different form.
Now for low depths if eval is far below alpha we check if qsearch can push it above alpha - and if it can't we return a fail low.
passed STC
https://tests.stockfishchess.org/tests/view/61fbf968d508ec6a1c9f3274
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 226120 W: 61106 L: 60472 D: 104542 Elo +0.97
Ptnml(0-2): 1118, 25592, 59080, 26078, 1192
passed LTC
https://tests.stockfishchess.org/tests/view/61fcc569d508ec6a1c9f5617
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 113128 W: 30851 L: 30397 D: 51880 Elo +1.39
Ptnml(0-2): 114, 11483, 32926, 11917, 124
closes https://github.com/official-stockfish/Stockfish/pull/3921
bench 4684080
see source
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Mon Feb 7 , 2022 .
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644251435
Do less depth reduction in null move pruning for complex positions
This patch makes us reduce less depth in null move pruning if complexity is high enough.
Thus, null move pruning now depends in two distinct ways on complexity,
while being the only search heuristic that exploits complexity so far.
passed STC
https://tests.stockfishchess.org/tests/view/61fde60fd508ec6a1c9f7754
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 170000 W: 45555 L: 45027 D: 79418 Elo +1.08
Ptnml(0-2): 760, 19352, 44359, 19658, 871
passed LTC
https://tests.stockfishchess.org/tests/view/61fe91febf46cb834cbd5c90
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 145272 W: 39182 L: 38651 D: 67439 Elo +1.27
Ptnml(0-2): 127, 14864, 42157, 15327, 161
closes https://github.com/official-stockfish/Stockfish/pull/3923
bench 4461945
see source
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644251435
Do less depth reduction in null move pruning for complex positions
This patch makes us reduce less depth in null move pruning if complexity is high enough.
Thus, null move pruning now depends in two distinct ways on complexity,
while being the only search heuristic that exploits complexity so far.
passed STC
https://tests.stockfishchess.org/tests/view/61fde60fd508ec6a1c9f7754
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 170000 W: 45555 L: 45027 D: 79418 Elo +1.08
Ptnml(0-2): 760, 19352, 44359, 19658, 871
passed LTC
https://tests.stockfishchess.org/tests/view/61fe91febf46cb834cbd5c90
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 145272 W: 39182 L: 38651 D: 67439 Elo +1.27
Ptnml(0-2): 127, 14864, 42157, 15327, 161
closes https://github.com/official-stockfish/Stockfish/pull/3923
bench 4461945
see source
-
- Points: 0,00
- Status: Offline (Active 2 Hours, 15 Minutes ago)
- Topics:
Re: Stockfish-NNUE
SF-PB-070222_x64, Updated to "Do less depth reduction in null move pruning for complex positions".
Engines https://pixeldrain.com/u/Cr5HHuaV
Network https://tests.stockfishchess.org/api/nn/nn-ac07bd334b62.nnue
bench 4461945
Engines https://pixeldrain.com/u/Cr5HHuaV
Network https://tests.stockfishchess.org/api/nn/nn-ac07bd334b62.nnue
bench 4461945
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Wed Feb 9 , 2022 .
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644423420
Big search tuning
Most credits for this patch should go to @candirufish.
Based on his big search tuning (1M games at 20+0.1s)
https://tests.stockfishchess.org/tests/view/61fc7a6ed508ec6a1c9f4b7d
with some hand polishing on top of it, which includes :
a) correcting trend sigmoid - for some reason original tuning resulted in it being negative. This heuristic was proven to be worth some elo for years so reversing it sign is probably some random artefact;
b) remove changes to continuation history based pruning - this heuristic historically was really good at providing green STCs and then failing at LTC miserably if we tried to make it more strict, original tuning was done at short time control and thus it became more strict - which doesn't scale to longer time controls;
c) remove changes to improvement - not really indended :).
passed STC
https://tests.stockfishchess.org/tests/view/6203526e88ae2c84271c2ee2
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 16840 W: 4604 L: 4363 D: 7873 Elo +4.97
Ptnml(0-2): 82, 1780, 4449, 2033, 76
passed LTC
https://tests.stockfishchess.org/tests/view/620376e888ae2c84271c35d4
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 17232 W: 4771 L: 4542 D: 7919 Elo +4.62
Ptnml(0-2): 14, 1655, 5048, 1886, 13
closes https://github.com/official-stockfish/Stockfish/pull/3926
bench 5030992
see source
Thanks to the Author: Michael Chaly
Download : https://abrok.eu/stockfish/
Timestamp: 1644423420
Big search tuning
Most credits for this patch should go to @candirufish.
Based on his big search tuning (1M games at 20+0.1s)
https://tests.stockfishchess.org/tests/view/61fc7a6ed508ec6a1c9f4b7d
with some hand polishing on top of it, which includes :
a) correcting trend sigmoid - for some reason original tuning resulted in it being negative. This heuristic was proven to be worth some elo for years so reversing it sign is probably some random artefact;
b) remove changes to continuation history based pruning - this heuristic historically was really good at providing green STCs and then failing at LTC miserably if we tried to make it more strict, original tuning was done at short time control and thus it became more strict - which doesn't scale to longer time controls;
c) remove changes to improvement - not really indended :).
passed STC
https://tests.stockfishchess.org/tests/view/6203526e88ae2c84271c2ee2
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 16840 W: 4604 L: 4363 D: 7873 Elo +4.97
Ptnml(0-2): 82, 1780, 4449, 2033, 76
passed LTC
https://tests.stockfishchess.org/tests/view/620376e888ae2c84271c35d4
LLR: 2.96 (-2.94,2.94) <0.50,3.00>
Total: 17232 W: 4771 L: 4542 D: 7919 Elo +4.62
Ptnml(0-2): 14, 1655, 5048, 1886, 13
closes https://github.com/official-stockfish/Stockfish/pull/3926
bench 5030992
see source
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Thu Feb 10 , 2022 .
Thanks to the Author: Tomasz Sobczyk
Download : https://abrok.eu/stockfish/
Update architecture to "SFNNv4". Update network to nn-6877cd24400e.nnue.
Architecture:
The diagram of the "SFNNv4" architecture:
https://user-images.githubusercontent.com/8037982/153455685-cbe3a038-e158-4481-844d-9d5fccf5c33a.png
The most important architectural changes are the following:
* 1024x2 [activated] neurons are pairwise, elementwise multiplied (not quite pairwise due to implementation details, see diagram), which introduces a non-linearity that exhibits similar benefits to previously tested sigmoid activation (quantmoid4), while being slightly faster.
* The following layer has therefore 2x less inputs, which we compensate by having 2 more outputs. It is possible that reducing the number of outputs might be beneficial (as we had it as low as 8 before). The layer is now 1024->16.
* The 16 outputs are split into 15 and 1. The 1-wide output is added to the network output (after some necessary scaling due to quantization differences). The 15-wide is activated and follows the usual path through a set of linear layers. The additional 1-wide output is at least neutral, but has shown a slightly positive trend in training compared to networks without it (all 16 outputs through the usual path), and allows possibly an additional stage of lazy evaluation to be introduced in the future.
Additionally, the inference code was rewritten and no longer uses a recursive implementation. This was necessitated by the splitting of the 16-wide intermediate result into two, which was impossible to do with the old implementation with ugly hacks. This is hopefully overall for the better.
First session:
The first session was training a network from scratch (random initialization). The exact trainer used was slightly different (older) from the one used in the second session, but it should not have a measurable effect. The purpose of this session is to establish a strong network base for the second session. Small deviations in strength do not harm the learnability in the second session.
The training was done using the following command:
python3 train.py \
/home/sopel/nnue/nnue-pytorch-training/data/nodes5000pv2_UHO.binpack \
/home/sopel/nnue/nnue-pytorch-training/data/nodes5000pv2_UHO.binpack \
--gpus "$3," \
--threads 4 \
--num-workers 4 \
--batch-size 16384 \
--progress_bar_refresh_rate 20 \
--random-fen-skipping 3 \
--features=HalfKAv2_hm^ \
--lambda=1.0 \
--gamma=0.992 \
--lr=8.75e-4 \
--max_epochs=400 \
--default_root_dir ../nnue-pytorch-training/experiment_$1/run_$2
Every 20th net was saved and its playing strength measured against some baseline at 25k nodes per move with pure NNUE evaluation (modified binary). The exact setup is not important as long as it's consistent. The purpose is to sift good candidates from bad ones.
The dataset can be found https://drive.google.com/file/d/1UQdZN_LWQ265spwTBwDKo0t1WjSJKvWY/view
Second session:
The second training session was done starting from the best network (as determined by strength testing) from the first session. It is important that it's resumed from a .pt model and NOT a .ckpt model. The conversion can be performed directly using serialize.py
The LR schedule was modified to use gamma=0.995 instead of gamma=0.992 and LR=4.375e-4 instead of LR=8.75e-4 to flatten the LR curve and allow for longer training. The training was then running for 800 epochs instead of 400 (though it's possibly mostly noise after around epoch 600).
The training was done using the following command:
The training was done using the following command:
python3 train.py \
/data/sopel/nnue/nnue-pytorch-training/data/T60T70wIsRightFarseerT60T74T75T76.binpack \
/data/sopel/nnue/nnue-pytorch-training/data/T60T70wIsRightFarseerT60T74T75T76.binpack \
--gpus "$3," \
--threads 4 \
--num-workers 4 \
--batch-size 16384 \
--progress_bar_refresh_rate 20 \
--random-fen-skipping 3 \
--features=HalfKAv2_hm^ \
--lambda=1.0 \
--gamma=0.995 \
--lr=4.375e-4 \
--max_epochs=800 \
--resume-from-model /data/sopel/nnue/nnue-pytorch-training/data/exp295/nn-epoch399.pt \
--default_root_dir ../nnue-pytorch-training/experiment_$1/run_$run_id
In particular note that we now use lambda=1.0 instead of lambda=0.8 (previous nets), because tests show that WDL-skipping introduced by vondele performs better with lambda=1.0. Nets were being saved every 20th epoch. In total 16 runs were made with these settings and the best nets chosen according to playing strength at 25k nodes per move with pure NNUE evaluation - these are the 4 nets that have been put on fishtest.
The dataset can be found either at ftp://ftp.chessdb.cn/pub/sopel/data_sf/T60T70wIsRightFarseerT60T74T75T76.binpack in its entirety (download might be painfully slow because hosted in China) or can be assembled in the following way:
Get the https://github.com/official-stockfish/Stockfish/blob/5640ad48ae5881223b868362c1cbeb042947f7b4/script/interleave_binpacks.py script.
Download T60T70wIsRightFarseer.binpack https://drive.google.com/file/d/1_sQoWBl31WAxNXma2v45004CIVltytP8/view
Download farseerT74.binpack http://trainingdata.farseer.org/T74-May13-End.7z
Download farseerT75.binpack http://trainingdata.farseer.org/T75-June3rd-End.7z
Download farseerT76.binpack http://trainingdata.farseer.org/T76-Nov10th-End.7z
Run python3 interleave_binpacks.py T60T70wIsRightFarseer.binpack farseerT74.binpack farseerT75.binpack farseerT76.binpack T60T70wIsRightFarseerT60T74T75T76.binpack
Tests:
STC: https://tests.stockfishchess.org/tests/view/6203fb85d71106ed12a407b7
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 16952 W: 4775 L: 4521 D: 7656 Elo +5.21
Ptnml(0-2): 133, 1818, 4318, 2076, 131
LTC: https://tests.stockfishchess.org/tests/view/62041e68d71106ed12a40e85
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 14944 W: 4138 L: 3907 D: 6899 Elo +5.37
Ptnml(0-2): 21, 1499, 4202, 1728, 22
closes https://github.com/official-stockfish/Stockfish/pull/3927
Bench: 4919707
Thanks to the Author: Tomasz Sobczyk
Download : https://abrok.eu/stockfish/
Update architecture to "SFNNv4". Update network to nn-6877cd24400e.nnue.
Architecture:
The diagram of the "SFNNv4" architecture:
https://user-images.githubusercontent.com/8037982/153455685-cbe3a038-e158-4481-844d-9d5fccf5c33a.png
The most important architectural changes are the following:
* 1024x2 [activated] neurons are pairwise, elementwise multiplied (not quite pairwise due to implementation details, see diagram), which introduces a non-linearity that exhibits similar benefits to previously tested sigmoid activation (quantmoid4), while being slightly faster.
* The following layer has therefore 2x less inputs, which we compensate by having 2 more outputs. It is possible that reducing the number of outputs might be beneficial (as we had it as low as 8 before). The layer is now 1024->16.
* The 16 outputs are split into 15 and 1. The 1-wide output is added to the network output (after some necessary scaling due to quantization differences). The 15-wide is activated and follows the usual path through a set of linear layers. The additional 1-wide output is at least neutral, but has shown a slightly positive trend in training compared to networks without it (all 16 outputs through the usual path), and allows possibly an additional stage of lazy evaluation to be introduced in the future.
Additionally, the inference code was rewritten and no longer uses a recursive implementation. This was necessitated by the splitting of the 16-wide intermediate result into two, which was impossible to do with the old implementation with ugly hacks. This is hopefully overall for the better.
First session:
The first session was training a network from scratch (random initialization). The exact trainer used was slightly different (older) from the one used in the second session, but it should not have a measurable effect. The purpose of this session is to establish a strong network base for the second session. Small deviations in strength do not harm the learnability in the second session.
The training was done using the following command:
python3 train.py \
/home/sopel/nnue/nnue-pytorch-training/data/nodes5000pv2_UHO.binpack \
/home/sopel/nnue/nnue-pytorch-training/data/nodes5000pv2_UHO.binpack \
--gpus "$3," \
--threads 4 \
--num-workers 4 \
--batch-size 16384 \
--progress_bar_refresh_rate 20 \
--random-fen-skipping 3 \
--features=HalfKAv2_hm^ \
--lambda=1.0 \
--gamma=0.992 \
--lr=8.75e-4 \
--max_epochs=400 \
--default_root_dir ../nnue-pytorch-training/experiment_$1/run_$2
Every 20th net was saved and its playing strength measured against some baseline at 25k nodes per move with pure NNUE evaluation (modified binary). The exact setup is not important as long as it's consistent. The purpose is to sift good candidates from bad ones.
The dataset can be found https://drive.google.com/file/d/1UQdZN_LWQ265spwTBwDKo0t1WjSJKvWY/view
Second session:
The second training session was done starting from the best network (as determined by strength testing) from the first session. It is important that it's resumed from a .pt model and NOT a .ckpt model. The conversion can be performed directly using serialize.py
The LR schedule was modified to use gamma=0.995 instead of gamma=0.992 and LR=4.375e-4 instead of LR=8.75e-4 to flatten the LR curve and allow for longer training. The training was then running for 800 epochs instead of 400 (though it's possibly mostly noise after around epoch 600).
The training was done using the following command:
The training was done using the following command:
python3 train.py \
/data/sopel/nnue/nnue-pytorch-training/data/T60T70wIsRightFarseerT60T74T75T76.binpack \
/data/sopel/nnue/nnue-pytorch-training/data/T60T70wIsRightFarseerT60T74T75T76.binpack \
--gpus "$3," \
--threads 4 \
--num-workers 4 \
--batch-size 16384 \
--progress_bar_refresh_rate 20 \
--random-fen-skipping 3 \
--features=HalfKAv2_hm^ \
--lambda=1.0 \
--gamma=0.995 \
--lr=4.375e-4 \
--max_epochs=800 \
--resume-from-model /data/sopel/nnue/nnue-pytorch-training/data/exp295/nn-epoch399.pt \
--default_root_dir ../nnue-pytorch-training/experiment_$1/run_$run_id
In particular note that we now use lambda=1.0 instead of lambda=0.8 (previous nets), because tests show that WDL-skipping introduced by vondele performs better with lambda=1.0. Nets were being saved every 20th epoch. In total 16 runs were made with these settings and the best nets chosen according to playing strength at 25k nodes per move with pure NNUE evaluation - these are the 4 nets that have been put on fishtest.
The dataset can be found either at ftp://ftp.chessdb.cn/pub/sopel/data_sf/T60T70wIsRightFarseerT60T74T75T76.binpack in its entirety (download might be painfully slow because hosted in China) or can be assembled in the following way:
Get the https://github.com/official-stockfish/Stockfish/blob/5640ad48ae5881223b868362c1cbeb042947f7b4/script/interleave_binpacks.py script.
Download T60T70wIsRightFarseer.binpack https://drive.google.com/file/d/1_sQoWBl31WAxNXma2v45004CIVltytP8/view
Download farseerT74.binpack http://trainingdata.farseer.org/T74-May13-End.7z
Download farseerT75.binpack http://trainingdata.farseer.org/T75-June3rd-End.7z
Download farseerT76.binpack http://trainingdata.farseer.org/T76-Nov10th-End.7z
Run python3 interleave_binpacks.py T60T70wIsRightFarseer.binpack farseerT74.binpack farseerT75.binpack farseerT76.binpack T60T70wIsRightFarseerT60T74T75T76.binpack
Tests:
STC: https://tests.stockfishchess.org/tests/view/6203fb85d71106ed12a407b7
LLR: 2.94 (-2.94,2.94) <0.00,2.50>
Total: 16952 W: 4775 L: 4521 D: 7656 Elo +5.21
Ptnml(0-2): 133, 1818, 4318, 2076, 131
LTC: https://tests.stockfishchess.org/tests/view/62041e68d71106ed12a40e85
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 14944 W: 4138 L: 3907 D: 6899 Elo +5.37
Ptnml(0-2): 21, 1499, 4202, 1728, 22
closes https://github.com/official-stockfish/Stockfish/pull/3927
Bench: 4919707
-
- Points: 0,00
- Status: Offline (Active 2 Hours, 15 Minutes ago)
- Topics:
Re: Stockfish-NNUE
SF-PB-100222_x64, Updated to "Update architecture to "SFNNv4". Update network to nn-6877cd24400e.nnue."
Engines https://pixeldrain.com/u/WsV4FB6N
Network https://tests.stockfishchess.org/api/nn/nn-6877cd24400e.nnue
Bench: 4919707
Engines https://pixeldrain.com/u/WsV4FB6N
Network https://tests.stockfishchess.org/api/nn/nn-6877cd24400e.nnue
Bench: 4919707
-
- Forum Contributions
- Points: 35 133,00
- Posts: 3274
- Joined: 01/01/2020, 8:40
- Status: Offline (Active 8 Hours, 7 Minutes ago)
- Medals: 2
- Topics: 405
- Reputation: 18714
- Location: Somewhere in time .
- Has thanked: 10493 times
- Been thanked: 18691 times
Re: Stockfish-NNUE
Date: Sun Feb 13 , 2022 .
Thanks to the Author: Michael Chaly ssfgmfgsoke
Download : https://abrok.eu/stockfish/
Timestamp: 1644710727
Big search tuning (version 2)
One more tuning - this one includes newly introduced heuristics and
some other parameters that were not included in previous one. Result
of 400k games at 20+0.2 "as is". Tuning is continuing since there is
probably a lot more elo to gain.
STC:
https://tests.stockfishchess.org/tests/view/620782edd71106ed12a497d1
LLR: 2.99 (-2.94,2.94) <0.00,2.50>
Total: 38504 W: 10260 L: 9978 D: 18266 Elo +2.54
Ptnml(0-2): 142, 4249, 10230, 4447, 184
LTC:
https://tests.stockfishchess.org/tests/view/6207a243d71106ed12a49d07
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 25176 W: 6793 L: 6546 D: 11837 Elo +3.41
Ptnml(0-2): 20, 2472, 7360, 2713, 23
closes https://github.com/official-stockfish/Stockfish/pull/3931
Bench: 4784796
see source
Thanks to the Author: Michael Chaly ssfgmfgsoke
Download : https://abrok.eu/stockfish/
Timestamp: 1644710727
Big search tuning (version 2)
One more tuning - this one includes newly introduced heuristics and
some other parameters that were not included in previous one. Result
of 400k games at 20+0.2 "as is". Tuning is continuing since there is
probably a lot more elo to gain.
STC:
https://tests.stockfishchess.org/tests/view/620782edd71106ed12a497d1
LLR: 2.99 (-2.94,2.94) <0.00,2.50>
Total: 38504 W: 10260 L: 9978 D: 18266 Elo +2.54
Ptnml(0-2): 142, 4249, 10230, 4447, 184
LTC:
https://tests.stockfishchess.org/tests/view/6207a243d71106ed12a49d07
LLR: 2.94 (-2.94,2.94) <0.50,3.00>
Total: 25176 W: 6793 L: 6546 D: 11837 Elo +3.41
Ptnml(0-2): 20, 2472, 7360, 2713, 23
closes https://github.com/official-stockfish/Stockfish/pull/3931
Bench: 4784796
see source
-
- Points: 0,00
- Status: Offline (Active 2 Hours, 15 Minutes ago)
- Topics:
Re: Stockfish-NNUE
SF-PB-170222_x64, Updated to "Tune search at very long time control." https://github.com/official-stockfish/Stockfish/commit/84b1940fcae95bb0a641dda9e85cb96f8c21cd22
Not suggested for fast time controls (less than 1 minute).
Engines https://pixeldrain.com/u/2vYYVXnX
Network https://tests.stockfishchess.org/api/nn/nn-6877cd24400e.nnue
Not suggested for fast time controls (less than 1 minute).
Engines https://pixeldrain.com/u/2vYYVXnX
Network https://tests.stockfishchess.org/api/nn/nn-6877cd24400e.nnue