Android engines tests.

[Archimedes]

And finally a tournament between the version with Clang and Android NDK.

Tournament was played on Motorola Moto G7 Power (Android 10, Snapdragon 632) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Stockfish_clang \
  -engine cmd=../uci/Stockfish_ndk \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn 1

c-chess-cli.txt:

Code: Select all

3-fold repetition: 228
50 moves rule: 41
Adjudication: 0
Checkmate: 147
Insufficient material: 78
Rules infraction: 0
Stalemate: 6
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name                    Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Stockfish 211021-clang   3102   0.0    8    8   500  252.5  50.5   76   71  353  15.2  70.6  3098 
   2 Stockfish 211021-ndk     3098   3.3    8    8   500  247.5  49.5   71   76  353  14.2  70.6  3102 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/obg2cfle2f0rqbsutaxc1kn8cifhmj6u

[Evo]

Thank's for the Tour Archie..

Seems GCC still managed to be the strongest compile after all right ? Even when it's not as big as we thought before..

Hopely u would like to compile ur next builds all in the GCC format.. if that's okay for u..

Regards

[Archimedes]

Hopely u would like to compile ur next builds all in the GCC format.. if that's okay for u..

I will do that with the CECSA script. Currently CorChess, Crystal, Stockfish and Demolito are supported. GCC is only used for arm64-v8a builds, the rest will be compiled with Android NDK (wich is fast enough for that purpose).

By the way, you can do it yourself (with the next update of CECSA).

[Archimedes]

Tournament was played on Samsung Galaxy Tab A7 (Android 11, Snapdragon 662) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Berserk \
  -engine cmd=../uci/Fire \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn

c-chess-cli.txt:

Code: Select all

3-fold repetition: 166
50 moves rule: 24
Adjudication: 1
Checkmate: 270
Insufficient material: 37
Rules infraction: 0
Stalemate: 2
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name            Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Fire 8.NN.MC.3   3137   0.0   11   11   500  302.0  60.4  187   83  230  37.4  46.0  3063 
   2 Berserk 7        3063  73.3   11   11   500  198.0  39.6   83  187  230  16.6  46.0  3137 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/p02tku1k3xqueitranemuv063hlhxxg6

[Archimedes]

A good performance from Koivisto against Fire. Keep in mind, Koivisto has no native NEON support (only through sse2neon header file).

Tournament was played on Samsung Galaxy Tab A7 (Android 11, Snapdragon 662) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Fire \
  -engine cmd=../uci/Koivisto \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn

c-chess-cli.txt:

Code: Select all

3-fold repetition: 120
50 moves rule: 52
Adjudication: 1
Checkmate: 247
Insufficient material: 68
Rules infraction: 0
Stalemate: 12
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name            Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Fire 8.NN.MC.3   3133   0.0   11   11   500  297.5  59.5  171   76  253  34.2  50.6  3067 
   2 Koivisto 7.5     3067  66.3   11   11   500  202.5  40.5   76  171  253  15.2  50.6  3133 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/81edu8yd81185j5x8zn413xx1h3ocat6

[mibaskom]

dear chess-mate,
Archimedes,

would you like provided another links please.
many thanks,

best regards,
mi.

[Archimedes]

Temporary download link for the last tournament. Old link will work again next month.

https://pixeldrain.com/u/jYCMRvWU

[Skynet]

Even after update, Chiron is miles away.
Tablet - Lenovo android 4.4.2 - 32 bit - Quad Core A7 - 1GB RAM - 1,3 GHz
GUI=C4a - Hash 64 mb - 2min+1sec - Book=8move_v3 - 4 cores per engine - ponder=NO
syzygy=NO, Adjudication Rule=Resign - Move 25, Move Count 3, Score (in cp) 600
Draw=Move Number 30, Move count 4, Score (in cp) 10 - Calculator - Ordo 1.2.6

Code: Select all

# PLAYER                 :  RATING  ERROR  PLAYED   (%)    W    D    L  D(%)  CFS(%)
1 Komodo 13.1 32-bit     :    3120     21     200  79.0  124   68    8  34.0     100
2 Chiron 5               :    2880     21     200  21.0    8   68  124  34.0     ---

games https://pixeldrain.com/u/udxokERA

[Archimedes]

Tournament was played on Motorola Moto G7 Power (Android 10, Snapdragon 632) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Berserk \
  -engine cmd=../uci/Koivisto \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn 1

c-chess-cli.txt:

Code: Select all

3-fold repetition: 105
50 moves rule: 74
Adjudication: 1
Checkmate: 264
Insufficient material: 49
Rules infraction: 0
Stalemate: 7
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name          Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Berserk 8      3113   0.0   11   11   500  269.0  53.8  151  113  236  30.2  47.2  3087 
   2 Koivisto 7.9   3087  25.4   11   11   500  231.0  46.2  113  151  236  22.6  47.2  3113 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/gm8zy0fnf5kss09e0iyi70fdhlo6byif

[Archimedes]

g++ versus clang++. g++ is ahead by a small amount.

Tournament was played on Samsung Galaxy Tab A7 (Android 11, Snapdragon 662) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Stockfish_clang++ \
  -engine cmd=../uci/Stockfish_g++ \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn

c-chess-cli.txt:

Code: Select all

3-fold repetition: 211
50 moves rule: 25
Adjudication: 1
Checkmate: 132
Insufficient material: 126
Rules infraction: 0
Stalemate: 5
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name                      Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Stockfish g++-211214       3103   0.0    8    8   500  255.0  51.0   71   61  368  14.2  73.6  3097 
   2 Stockfish clang++-211214   3097   7.0    8    8   500  245.0  49.0   61   71  368  12.2  73.6  3103 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/qg9d7phvt71o2r60y4bji2isel1bpch0

[LittleJoe]

g++ versus clang++. g++ is ahead by a small amount.

Tournament was played on Samsung Galaxy Tab A7 (Android 11, Snapdragon 662) with Termux and the CETSA script.
Offset for rating of one of the engines was not set in Bayeselo.

c-chess-cli.conf:

Code: Select all

../bin/c-chess-cli/$CETSA_ABI/c-chess-cli \
  -each tc=10+0.1 option.Hash=16 option.Threads=1 \
  -engine cmd=../uci/Stockfish_clang++ \
  -engine cmd=../uci/Stockfish_g++ \
  -games 500 \
  -concurrency 2 \
  -openings file=../epd/IM_4mvs.epd order=random \
  -repeat \
  -resign number=200 count=5 score=900 \
  -draw number=200 count=5 score=5 \
  -pgn games.pgn

c-chess-cli.txt:

Code: Select all

3-fold repetition: 211
50 moves rule: 25
Adjudication: 1
Checkmate: 132
Insufficient material: 126
Rules infraction: 0
Stalemate: 5
Time forfeit: 0
Unterminated: 0

Bayeselo.txt:

Code: Select all

Rank Name                      Rating   Δ     +    -     #     Σ    Σ%     W    L    D   W%    =%   OppR 
---------------------------------------------------------------------------------------------------------
   1 Stockfish g++-211214       3103   0.0    8    8   500  255.0  51.0   71   61  368  14.2  73.6  3097 
   2 Stockfish clang++-211214   3097   7.0    8    8   500  245.0  49.0   61   71  368  12.2  73.6  3103 
---------------------------------------------------------------------------------------------------------
  Δ = delta from the next higher rated opponent
  # = number of games played
  Σ = total score, 1 point for win, 1/2 point for draw

Games:
https://app.box.com/s/qg9d7phvt71o2r60y4bji2isel1bpch0

How can a speed difference of 1% lead to a ELO-difference of 7 or 8 ELO? This is impossible. There are values of +8 and -8 given in the results. Is this the estimated error (standard deviation) of the ELO difference after 500 games? If yes, there would be about a 1/3 change that deltaELO of the two builds is about zero, which would make sense.
Or does +/- 8 apply to each ELO value so that the two values could be separated by even 16 ELO with 32% chance?

[Archimedes]

How can a speed difference of 1% lead to a ELO-difference of 7 or 8 ELO? This is impossible. There are values of +8 and -8 given in the results. Is this the estimated error (standard deviation) of the ELO difference after 500 games? If yes, there would be about a 1/3 change that deltaELO of the two builds is about zero, which would make sense.
Or does +/- 8 apply to each ELO value so that the two values could be separated by even 16 ELO with 32% chance?

1 % was estimated by me based on the start position only. Now we are diving into the Bayeselo world now. I'm not the specialist here for all of the parameters which can be used. I only used some of the parameters and commands for a basic output. Regarding the + and - signs, i would say, this tells how much the elo rating can differ. The elo rating for Stockfish g++-211214, for example, is something between 3095 and 3111 (as the offset for rating of one of the engines was not set in Bayeselo, the default rating was set to 3100).

[LittleJoe]

How can a speed difference of 1% lead to a ELO-difference of 7 or 8 ELO? This is impossible. There are values of +8 and -8 given in the results. Is this the estimated error (standard deviation) of the ELO difference after 500 games? If yes, there would be about a 1/3 change that deltaELO of the two builds is about zero, which would make sense.
Or does +/- 8 apply to each ELO value so that the two values could be separated by even 16 ELO with 32% chance?

1 % was estimated by me based on the start position only. Now we are diving into the Bayeselo world now. I'm not the specialist here for all of the parameters which can be used. I only used some of the parameters and commands for a basic output. Regarding the + and - signs, i would say, this tells how much the elo rating can differ. The elo rating for Stockfish g++-211214, for example, is something between 3095 and 3111 (as the offset for rating of one of the engines was not set in Bayeselo, the default rating was set to 3100).

Chess statistics is quite difficult, no question. I think that +/-8 is the standard deviation but I'm not sure if this error has to be applied to the ELO difference only or to both ELO values. Anyway, in both cases the error is larger than the measured ELO difference of 6 ELO. A very crude rule of thumb is that doubling the speed (100% faster) gives 100 ELO and thus this 6 ELO would correspond to 6% nps difference. The nps difference of the gcc and clang builds is definitely much smaller.

[n0c]

CF-030121_vs_SF-100222 in a current smartphone: 26.4 ELO

Tested on a Samsung A12 phone: SF-100222 has 26.2 points more ELO than CF-030121

~/Engines $ ./CF-030121-arm8 bench 2>&1 | tail -n 4
===========================
Total time (ms) : 15285
Nodes searched : 4109336
Nodes/second : 268847

~/Engines $ ./SF_100222 bench 2>&1 | tail -n 4
===========================
Total time (ms) : 32935
Nodes searched : 4919707
Nodes/second : 149376

c-chess-cli command:
$ ./c-chess-cli -each tc=10+0.1 option.Hash=16 option.Threads=1 -engine cmd=../Engines/SF_100222 name=SF_100222 -engine cmd=../Engines/CF-030121-arm8 name=CF-030121 -games 20000 -concurrency 7 -openings file=../epd/UHO_XXL_+0.90_+1.19.epd order=random -repeat -sprt elo0=0 elo1=2.5 -resign count=3 score=300 -draw number=30 count=6 score=10 -pgn out.pgn 0

In zip file: https://pixeldrain.com/l/18giF4ou
CF-030121-arm8
4284games_SF-100222_vs_CF-030121.pgn
Bayeselo_4284games_SF-100222_vs_CF-030121.txt <--- dif. ELO: 26.4

[Skynet]

Tablet - Lenovo android 4.4.2 - 32 bit - Quad Core (4x Cortex A7) - 1GB RAM - 1,3 GHz
GUI=Cfa - Hash 64 mb - 4 cores per engine - TC: 2min+1sec - Book=8moves.pgn (60 pos) - Resign 600 cp
ELO rating has been set to 3498 (Houdini 6.03 with 4 cores and TC: more then 2 min). Engines armv7.

Code: Select all

# PLAYER           :  RATING  ERROR  PLAYED   (%)    W    D    L  D(%)  CFS(%)
1 Berserk 8.5.1    :    3601     22     360  71.3  183  147   30  40.8     100
2 Houdini 6.03     :    3498     22     360  53.8  115  157   88  43.6      91
3 Fire 8.2         :    3477     20     360  50.0  102  156  102  43.3     100
4 Tucano 10.00     :    3328     24     360  25.0   23  134  203  37.2     ---

Games https://pixeldrain.com/u/Kpa9xEJK
======================================
P.s. i also wanted to test last Koivisto and Ethereal, but unfortunately both engines resigns after one or two moves.