Chess Engine Compiler Script for Android

[Archimedes]

If you have already installed a working environment of gcc on Termux and want to compile Stockfish (and may be other chess engines too) from time to time, then, may be, this script makes your life easier.

Download CECSA (Chess Engine Compiler Script for Android), start Termux and copy the tar archive from the download directory of your smarthphone to the home directory of Termux (change the command line for your needs).

Code: Select all

cp /storage/emulated/0/Download/cecsa_2021.10.18.tar.gz ./

Extract the tar archive.

Code: Select all

tar -xf ./cecsa_2021.10.18.tar.gz

Now you have a directory cecsa in your home directory. Go to the cecsa directory and start the script.

Code: Select all

cd ./cecsa
./start.sh

What does the script do? When you start the script without any parameter, it download the sources from Stockfish (via "git clone"), copy a working makefile (etc/Stockfish/Makefile) to the source directory of Stockfish and start the compiliation process. When finished, the executable will be copied to the CECSA_HOME directory (/storage/emulated/0/Download). The CECSA_HOME directory can be defined as an environment variable at the bottom of the script. With the CECSA_COMPILER environment variable can you define your preferred compiler (clang or gcc).

At the moment, the script supports the following parameters:

Code: Select all

./start.sh
./start.sh all
./start.sh corchess
./start.sh crystal
./start.sh stockfish

As you can see, currently CorChess, Crystal and Stockfish are supported, but can be extended to other chess engines too, for sure. With the parameter "all" all supported chess engines will be compiled (this is mainly for testing purposes). The changed makefiles (necessary for proper compiling) are in the etc directory. Changes to the makefiles were only made, when it was absolute necessary for proper compiling. May be, it is a better idea, to create new makefiles for the CECSA script, may be only for arm64-v8a based CPUs, so people can better see, what optimization settings are used and what can be optimized for better performance.

There is only one downside at the moment. Compilation with gcc seems not to be very stable. Sometimes, during the compilation process, gcc crashes and with it, Termux too. However, the clang compiler works without any issues.

CECSA 2021.10.18:
https://app.box.com/s/uajbge2qhp3zj3oglitnww8of0r8sz5t

[MichaelM]

Wow! It worked fine without parameters e.g. for the SF build. Congrats for this script!
One downside is that I got several warnings concerning the PGO optimization. However, in Droidfish it's about as fast as your high speed compile from yesterday. A testing with the SF bench may follow. Again very much!

[MichaelM]

My SF build (made with CECSA):
Total time (ms) : 11179
Nodes searched : 2437232
Nodes/second : 218018

SF 151021 gcc, "Smooth improving":
Total time (ms) : 10930
Nodes searched : 2340839
Nodes/second : 214166

Stockfish 211017:
Total time (ms) : 9336
Nodes searched : 2156378
Nodes/second : 230974

Not so bad 'my' new SF build (with embedded net ) but is it PGO optimized? Have to check a lot of warnings.

[Archimedes]

Not so bad 'my' new SF build (with embedded net ) but is it PGO optimized? Have to check a lot of warnings.

With clang (CECSA_COMPILER=clang) there are no warnings and no crashes. I could compile all three supported chess engines in one batch without any issues.

With gcc i'm also getting warnings (need to be proofed, but i didn't encounter any slowdown with it). On my Smartphone gcc crashes all the time at a specific compilation point. On ny Tablet gcc works, but often i need a second attempt.

Regarding the warnings, we have to deal with a not correct makefile from Stockfish. An own makefile for the CECSA script would be a good idea.

PGO is allways on.

[Archimedes]

If you want a non embedded version of Stockfish, edit etc/Stockfish/Makefile and add -DNNUE_EMBEDDING_OFF to the compiler flags (CXXFLAGS), may be in line 318 of the makefile (works for clang and gcc).

Code: Select all

CXXFLAGS += -Wall -Wcast-qual -fno-exceptions -std=c++17 -DNNUE_EMBEDDING_OFF $(EXTRACXXFLAGS)

[Archimedes]

I've uploaded the package again. There was a compiler flag -fno-gcse in the gcc section of the makefiles which i removed, when i prepared for armv64-v8 based CPUs. I added the compiler flag again, because it has a (very) small impact on the performance. There was no changes to the script, only to the makefiles. Makefiles should now a good base (they are nearly identical with the original makefiles) for further improvements.

[MichaelM]

If you want a non embedded version of Stockfish, edit etc/Stockfish/Makefile and add -DNNUE_EMBEDDING_OFF to the compiler flags (CXXFLAGS), may be in line 318 of the makefile (works for clang and gcc).

Code: Select all

CXXFLAGS += -Wall -Wcast-qual -fno-exceptions -std=c++17 -DNNUE_EMBEDDING_OFF $(EXTRACXXFLAGS)

By chance, I was then live on GitHub, reading their comments, when vondele and snicolet were finishing this 'net embedding PR'. And since that time I try to convince people NOT to embed the net. Certainly, I know CXXFLAGS += -DNNUE_EMBEDDING_OFF since more than one year:

http://outskirts.altervista.org/forum/viewtopic.php?f=40&t=473&start=31

Anyway, thanks for your hint!

[Archimedes]

Problems with crashes seems to be solved now. It seems, that gcc doesn't work well with parallel execution on Android. If i execute the make command without the -j parameter, make will do a serial execution (one thing at a time) and gcc has no more crashes so far.

CECSA 2021.10.19:
https://app.box.com/s/2i7hcwadwq1ibz0n5ojqervla798vubu

[MichaelM]

Problems with crashes seems to be solved now. It seems, that gcc doesn't work well with parallel execution on Android. If i execute the make command without the -j parameter, make will do a serial execution (one thing at a time) and gcc has no more crashes so far.

CECSA 2021.10.19:
https://app.box.com/s/2i7hcwadwq1ibz0n5ojqervla798vubu

Do you have 3 or 4 GB ram in your smartphone? The crashes could -maybe- due to low free ram.

[Archimedes]

Do you have 3 or 4 GB ram in your smartphone? The crashes could -maybe- due to low free ram.

With clang it was working without any problems, only gcc was making trouble. May be, it's a problem with the ram management. I don't know. However, with serial execution, it works.

[LittleJoe]

I have now also installed Termux, Clang, GCC, and CECSA and have used the script several times. It worked excellent, no problems.

[Archimedes]

On my devices -march=native and -march=armv8-a produces identical executables. Therefore i must not care about it and can distribute the executables with the default settings (-march=native).

[Archimedes]

I have added Demolito as new chess engine (builds made with Android NDK have poor performance).

When starting the script you will see now, what chess engine will be compiled. For example, when you start the script without any parameters, then "Compile Stockfish for Android?" will be appeared. If you answered this question with yes, then there will be a new question "With embedded network?". No more need to change the makefiles for that purpose. Just enter the information just in time.

The correct version number is always a part of the compilation tasks which i take care about. If you are not pleased with the automatic version numbering of Stockfish (when version string is empty, version number is the date of the compile), you can now enter your own version number (normally i'm using the date of the last commit in reverse order as version number). This can be done by adding an additional parameter when starting the script.

Example:

Code: Select all

./start.sh stockfish 211021

Stockfish now shows "Stockfish 211021" when starting with DroidFish.

This works with all included chess engines.

Example:

Code: Select all

./start.sh demolito 2021-10-21

Demolito, for example, shows "Demolito dev" in DroidFish when you start the script only with "./start.sh demolito".

And, last but not least, there are new environment variables related to the chess engines.

Code: Select all

CECSA_CORCHESS_MAKEFILE=Makefile
CECSA_CRYSTAL_MAKEFILE=Makefile
CECSA_DEMOLITO_MAKEFILE=makefile
CECSA_STOCKFISH_MAKEFILE=Makefile

With this new environment variables you can choose, what makefile you want to use for a specific chess engine. When you want to do some experiments with settings or want to use your own makefiles, just make a copy of the original makefile in the etc directory and work with this copy. No need to make backups of the original makefile when doing some tests or something like that. You can create several makefiles and choose them in the script.

For normal compilation tasks there is no need to edit the script (changing environment variables) or the makefiles, all tasks can be done in the command line, that is, choosing, whether to use an embedded network or not and entering a version number (which is optional). Things should be kept as simple as possible.

CECSA 2021.10.21:
https://app.box.com/s/0zr9zf8vipuvafz6gh53beimgd77j8tv

[Archimedes]

Support for Android NDK added. Now, you are able to compile although for armeabi-v7a, x86 and x86_64 based CPUs (needs an installed Android NDK).

But, before we begin, install zip in your Termux environment. It is needed for making the zip archive with all the executables at the end of compiling.

Code: Select all

apt-get install zip

Let's have a look in the environment variables in the script.

Code: Select all

CECSA_ARM64_V8A_ONLY=true
CECSA_COMPILER=gcc
CECSA_HOME=/storage/emulated/0/Download
CECSA_VERSION=2021.10.24
CECSA_WAKE_LOCK=true

CECSA_CORCHESS_MAKEFILE=Makefile
CECSA_CRYSTAL_MAKEFILE=Makefile
CECSA_DEMOLITO_MAKEFILE=makefile
CECSA_STOCKFISH_MAKEFILE=Makefile

CECSA now supports three different compilers: clang, gcc and ndk.

The environment variable CECSA_ARM64_V8A_ONLY is new and everybody should know, what it means. If true, only builds for arm64-v8a based CPUs will be generated. If false, then there will also be generated builds for armeabi-v7a, x86 and x86_64 based CPUs. This is the job for Android NDK. Android NDK should be properly installed (extracted to a directory of your choice) and the root directory should be in the path environment variable (the command "ndk-build" should work in the command line). With CECSA_ARM64_V8A_ONLY=false you enable Android NDK.

Example: If you start a compile for Stockfish with the following settings, then the build for arm64-v8a will be done with gcc and the rest with the help of Android NDK.

Code: Select all

CECSA_ARM64_V8A_ONLY=false
CECSA_COMPILER=gcc

Same, when you start a compilation with Clang.

Code: Select all

CECSA_ARM64_V8A_ONLY=false
CECSA_COMPILER=clang

When you start a compilation with Android NDK, then all builds will be made with Android NDK (although that for arm64-v8a).

Code: Select all

CECSA_ARM64_V8A_ONLY=false
CECSA_COMPILER=ndk

Example: If i want to distribute the current Stockfish with all available builds, then yo need the following settings in the script:

Code: Select all

CECSA_ARM64_V8A_ONLY=false
CECSA_COMPILER=gcc

After that, start the compiliation process with the following command line (which includes a version number).

Code: Select all

./start.sh stockfish 211023

CECSA 2021.10.24:
https://app.box.com/s/z9wxi4cjledxbas0wktuvedata2tk66v

[Archimedes]

Installing Android NDK on Termux is simple. Just donwload the latest version from https://github.com/Lzhiyong/termux-ndk (currently this is the file android-ndk-r23.tar.xz). Extract the file to a directory in Termux of your choice, e. g. /data/data/com.termux/files/usr/lib/android-ndk, and put that directory to your path environment variable. I used a hidden file named .bashrc in the home directory for that.

.bashrc:

Code: Select all

export PATH=/data/data/com.termux/files/cctools-toolchain/bin:$PATH
export PATH=$PATH:/data/data/com.termux/files/usr/lib/android-ndk

That's all.