Permanent Brain

[Sedat Canbaz]

By the way,
When I say practice is more important than knowledge

I have to add also that,
My machine 2x 2686 via HT enabled supports 72 threads

Ok..then I have a question to all,
Is it a good idea testing the overall engines with 72 Threads via Ponder OFF ?
with 100% CPU load ?? when summer time: 40+ c degrees ???

Forget everything..many of the engines are buggy even via 8 Threads...

Especially after 16 threads...these bugs are more frequent...

And one of the most important issues,
Cutechess GUI crashed when 64 threads per engine as Ponder OFF
- Some GUIs or Engines does not support 72 threads.. etc and etc

That's why...for best way...I usually prefer:
Less cores..via ponder on...

That's why please...once more,
I have a little experience, not much but I have...

Greetings

[Peter Grayson]

To be more clear, the practice is more important than knowledge!

I generally try to avoid a potential flame discussion especially when people are satisfied with the way they run their tests but for those sufficiently interested to ask a question in the matter of ponder on or ponder off there is sufficient information available on the Internet to determine which arrangement gives best performance whether by science or consideration of how established Test Houses run their matches.

Trying to keep it relatively simple as much for my own benefit as anyone else's, for each instance of an engine running at the same time the CPU has to access CPU cache, RAM and the disk drive. For single CPUs each engine is using the same data bus to access the same memory and especially in end games the same hard drive. Therefore when more than one engine is running it is also battling against the other engine(s) for data bus access to the memory and disk drive compared to a single instance of the engine that is only battling with the operating system demands. Minimising competition for resources must therefore give the strongest singular instance performance that includes preventing more than one engine running at the same time. It becomes even more complex when more than one CPU and NUMA come into consideration.

If the science is too heady for a layperson that is understandable but alternatively there is a good guide to identify if the argument to run with ponder = off gives best performance and integrity in results when it is worth checking how established Engine Test Houses run their games.

TCEC run with ponder = off, including CPU and GPU engines.

CCRL run with ponder = off

CEGT generally run with ponder = off but do run some matches with ponder = on. However, in the latter case the engines are allocated just one CPU core for those particular test groups probably to obtain as close as possible to deterministic engine results but unclear.

FGRL run with Ponder = off

POHL runs with Ponder = off

Peter

[Sedat Canbaz]

To be more clear, the practice is more important than knowledge!

I generally try to avoid a potential flame discussion especially when people are satisfied with the way they run their tests but for those sufficiently interested to ask a question in the matter of ponder on or ponder off there is sufficient information available on the Internet to determine which arrangement gives best performance whether by science or consideration of how established Test Houses run their matches.

Trying to keep it relatively simple as much for my own benefit as anyone else's, for each instance of an engine running at the same time the CPU has to access CPU cache, RAM and the disk drive. For single CPUs each engine is using the same data bus to access the same memory and especially in end games the same hard drive. Therefore when more than one engine is running it is also battling against the other engine(s) for data bus access to the memory and disk drive compared to a single instance of the engine that is only battling with the operating system demands. Minimising competition for resources must therefore give the strongest singular instance performance that includes preventing more than one engine running at the same time. It becomes even more complex when more than one CPU and NUMA come into consideration.

If the science is too heady for a layperson that is understandable but alternatively there is a good guide to identify if the argument to run with ponder = off gives best performance and integrity in results when it is worth checking how established Engine Test Houses run their games.

TCEC run with ponder = off, including CPU and GPU engines.

CCRL run with ponder = off

CEGT generally run with ponder = off but do run some matches with ponder = on. However, in the latter case the engines are allocated just one CPU core for those particular test groups probably to obtain as close as possible to deterministic engine results but unclear.

FGRL run with Ponder = off

POHL runs with Ponder = off

Peter

You forget to mention,
SSDF (one of the best rating list out there) which is the father of all your mentioned lists
SSDF team is based on testers , which run via Ponder ON since a long time...plus via 40/120 at max cores..

SCCT is probably brother of SSDF...) which mainly prefer as MP and Ponder ON

IPON..not active for nowadays, but in the past played as Ponder ON

Plus many others individual testers...I can count more... if you wish...

Btw, many other online engine testings prefer Ponder ON..such as Playchess, Infinitychess etc.
Actually I can count more...but will change your mind ? I think no...

A small note,
Many of your mentioned rating list started during 2000 years.. at these old times ...
Mostly processors were 1 cpu or 2 cpu...so they started and still continuing as Ponder OFF

I believe in that,
Many of these testers (you mentioned) prefer parallel matches+ via ponder off...to produce more games...
No more...no less...and hopes helps...

And I am talking about best and for max engine performance
Sure..then I strongly recommend: MP+Ponder ON

As last,
Anyone is free to run at any conditions...no problem from my side )

Good luck,
Sedat

[Nemesis]

Tomato - Tomahto
Potato - Potahto

Personal preference therefore any point in continuing this discussion ?
An exercise in futilty.

[Sedat Canbaz]

My final words over this issue,

Once believed that the sun returned, not the earth!

Or

The one who travels far knows much than who read...

Still not so clear ? then:

Practice is better than Knowledge..

That's all..

[Peter Grayson]

You forget to mention,
SSDF (one of the best rating list out there) which is the father of all your mentioned lists
SSDF team is based on testers , which run via Ponder ON since a long time...plus via 40/120 at max cores..

...and you forgot to mention the test criteria ...
From the SSDF site
All games have been played on the tournament level, 40 moves/2 hours followed by 20 moves/each following hour. In matches between PC-programs, two separate PCs have been used, connected with an auto232-cable.

Btw, many other online engine testings prefer Ponder ON..such as Playchess, Infinitychess etc.

Again a misrepresentation because two machines are involved that run independantly of eachother with no demand on the other machine's resources with connection by means of Internet server.

I believe in that,
Many of these testers (you mentioned) prefer parallel matches+ via ponder off...to produce more games...
No more...no less...and hopes helps...

Which Test Houses? Given the indicated hardware that is used then running concurrent games with the indicated number of cores would not be possible.

I have tested dedicated chess computers and chess engines since 1982. I ran with two PCs to test engine vs engine matches until purchasing the dual Xeon machine when provided the engine core allocation can be set correctly the CPU and RAM bank allocation reduces any issues for NUMA enabled machines but there still arises the issue of resources for the hard drive. However there were other issues when less than full resources were allocated and cores could not be allocated to non NUMA supporting engines therefore for integrity I reverted to ponder = off.

When two machines are used there may be some benefit in using ponder = on especially when the engines under test use the same base code, today mainly Stockfish, because the ponder hit rate is likely to be higher than when two independent source code engines are used. There was never a definitive answer to that question because there always remained the disadvantage of hash being overwritten upon a ponder miss.

When conducting games between two engines on a single PC, ponder = off will give better overall engine performance and integrity.

When a player ponders on opponent's time they do not interfere wih the operation of the other player's mind therefore there is a similar analogy to two computers.

The original question related to running engine matches on a single CPU PC.

Peter

[unofficial]

I mean especially about SCCT - DEEP BRAIN MP RATING:
https://sites.google.com/site/computerschess/scct-deep-brain-rating

Code: Select all

# PLAYER             : RATING    POINTS    GAMES   (%)
3 Ponder ON 4CPU     : 3670.6     326.5     600   54.4%
6 Ponder OFF 4CPU    : 3569.5     385.5     836   46.1%

Do you see the Elo difference too..?)
More than 100 Elo...if you can not...sorry...

Don't you see that "4cpu ponder on" vs "4cpu ponder off" is like "4cpu ponder off with longer tc" vs "4cpu ponder off with default tc" ?
The advantage of "ponder off" is to use more threads !

A valuable test between "4cpu ponder on" vs "8cpu ponder off" tell us a lot more than this one, sorry.

If we told about serious competition, please don't hide the +/-20 elo as error margin with only 600-800 games...

Once more,
Practice and knowledge are two different things...
I mean, usually Practice is more important than Knowledge!

If after 2 million games with "ponder on", your knowledge about "ponder off" stays so poor, this pratice loses some importance.
The best human chess players use supercomputer in order to prepare their tourneys, they want the most credible and accurate system, all the opposite as "ponder on" and this half power effect.

[EzioMagnifico]

I believe that the truth will not have it in your pocket no one ever and Mr. "sure" died, indeed he was never born. As for the phrase: "practice is better than knowledge", here too I half agree, as it is true that a strong and calloused hand of a worker tightens a bolt more easily than the neat and "soft" one of an engineer, who has designed the bolts and will never tighten them, but at the same time the worker also believes he is tightening the bolt well without knowing the physics thoroughly mechanics which, on the other hand, explains the forces involved and the correct tightening torque well.
I am not an expert in chess at your level, indeed not at all, but if I think about my work, since I come "from the street" but I have also done in-depth studies in my field, I can say from life experience that the two things do not go in contrast, on the contrary they complement each other in a harmonious and symbiotic way. Clearly it takes respect of the arm for the mind and of the mind for the arm, otherwise only contrasts are created that never lead to personal and temperamental improvements.
Sorry for the non-technical intrusion.

[Sedat Canbaz]

Just a moment

I am talking about my hardware conditions, not about yours..
No any idea about 1982...I am talking about 2022 year

Nowadays everything is changed...everything...

I ran many testings but many...not many can not imagine..
Actually many of them available at SedatChess site
And I wonder, where is your testings, prove that I am wrong ?
In short, can you publish..but next time if possible via games
Because without games..really this thread started to be boring..

Btw, about SCCT:
I run all the matches as NUMA disabled...
And as far I noticed, no any problem via Ponder ON on same PC

I think that,
You are missing a lot of things.. you stated close to 65% ponder hit rate..wrong..
It all depends..we can get even close to 10%, sure vs very weak chess engines
Once more, over than 90% is expecting...in case of Top NNUE engines each other
Or in case if use same Top NON-NNUE engine in book tours...
Then ponder hits is much higher...much more than 65%..just saying...
At least under SCCT conditions...

On other hand,
I agree with you that if our target will be to test all of the available engines, but all
Then...for all engines Ponder OFF would be better, because some can not ponder...
But nowadays almost all Top engines runs perfectly via MP + Ponder ON on same PC
Tested..guaranteed..

Btw, just another comparison:
https://sites.google.com/site/computerschess/scct-3m-1s-mp-rating-2022

SCCT (3m+1s) Ponder ON

Houdini 6.03 3489.7
Fritz 16 3168.8
Elo Difference: 321

If you check other blitz ratings...who run Blitz as Ponder OFF
I mean SCCT and the other as Ponder OFF,
Then you will notice in some, more than 50 Elo difference than SCCT

And now,
What does it mean ? a lot... at least I can say for my test conditions
I am individual tester... played on same hardware, use same book etc
For each player is installed a separate folder of engine..
In my active tours: No any internet connection etc and etc...
Against of any kind of risk...on other hand, you may know..
There are a lot of issues we can talk...but not required I think...

In other words,
If these engines will be played by various testers, various books..etc..
We can see a lot of strange results...really..this is already proved...
And now the most important question is coming:
Which rating list will help us to see the real Elo between both engines ?
I mean, on which source we have to relay for more accurate standings ?
Probably in your eyes: NON-SCCT results, right ?
Because I use Ponder ON on same hardware... ))
And probably Fritz 16 suffers on my hardwares.. hehe...
Think about these issues please...!

Greetings,
Sedat

[unofficial]

in sum, best strength and power usage but ponder on => half thread by engine and in addition NUMA disabled on dual socket hardware => ok i'm out, sorry guys loool

[Sedat Canbaz]

I mean especially about SCCT - DEEP BRAIN MP RATING:
https://sites.google.com/site/computerschess/scct-deep-brain-rating

Code: Select all

# PLAYER             : RATING    POINTS    GAMES   (%)
3 Ponder ON 4CPU     : 3670.6     326.5     600   54.4%
6 Ponder OFF 4CPU    : 3569.5     385.5     836   46.1%

Do you see the Elo difference too..?)
More than 100 Elo...if you can not...sorry...

Don't you see that "4cpu ponder on" vs "4cpu ponder off" is like "4cpu ponder off with longer tc" vs "4cpu ponder off with default tc" ?
The advantage of "ponder off" is to use more threads !

A valuable test between "4cpu ponder on" vs "8cpu ponder off" tell us a lot more than this one, sorry.

If we told about serious competition, please don't hide the +/-20 elo as error margin with only 600-800 games...

Once more,
Practice and knowledge are two different things...
I mean, usually Practice is more important than Knowledge!

If after 2 million games with "ponder on", your knowledge about "ponder off" stays so poor, this pratice loses some importance.
The best human chess players use supercomputer in order to prepare their tourneys, they want the most credible and accurate system, all the opposite as "ponder on" and this half power effect.

Ok...no words to say, exception once more:
Chess is thinking in opponent's turn!

And I am very happy with current used conditions, really... )

Good bye )

[Sedat Canbaz]

in sum, best strength and power usage but ponder on => half thread by engine and in addition NUMA disabled on dual socket hardware => ok i'm out, sorry guys loool

You have diffulties to see the reality.. can you publish your games ? only comments I see...
Too boring...enough..

Really I tested many many engines..I did not notice big difference in performance as NUMA ON
Even some engines started to loose on time when NUMA was enabled, but when NUMA OFF, all stable...

Once more,
Practice is more important than comments ))

Hopes helps

[Sedat Canbaz]

Meanwhile,

For anyone interested.. I run a new Ponder OFF test:
http://outskirts.altervista.org/forum/viewtopic.php?f=95&t=11&p=41581#p41581

Greetings

[Peter Grayson]

You have diffulties to see the reality.. can you publish your games ? only comments I see...

You have only conducted two matches relevant to the discussion in the link to the Deep Brain Rating match where you played Ponder ON 2CPU versus Ponder OFF 4CPU
The result of that match was

SCCT - Deep Brain Rating blitz 2020
1 Ponder ON 2CPU +2 +21/=159/-20 50.25% 100.5/200
2 Ponder OFF 4CPU -2 +20/=159/-21 49.75% 99.5/200

but according to your introductory data the match was played using two PCs not one and the result of just 1 point confirms my statement that historically it has been too close to call.

It was also interesting that the quickest wins were by the Ponder OFF 4CPU engine when the Ponder ON 2CPU engine benefitted least from time gain by ponder hit.

There was also a shorter match of
Ponder ON 1CPU versus Ponder OFF 1CPU that produced the result

SCCT - Deep Brain Rating blitz 2020

1 Ponder OFF 1CPU +5 +3/=69/-2 50.68% 37.5/74
2 Ponder ON 1CPU -5 +2/=69/-3 49.32% 36.5/74

Indicating Ponder = off produced the better result that is opposite of what you argue! However in reality it is too close to call.

Therefore the game evidence you presented to support your arguments actually supports my argument and conflicts with your argument! Thank you :-)

Peter

[Sedat Canbaz]

SCCT - Deep Brain Rating blitz 2020

1 Ponder OFF 1CPU +5 +3/=69/-2 50.68% 37.5/74
2 Ponder ON 1CPU -5 +2/=69/-3 49.32% 36.5/74

Indicating Ponder = off produced the better result that is opposite of what you argue! However in reality it is too close to call.


Peter

I do not know what do you mean, from where did you get this data ?
Or simply Is this as joke ? according to my official ranking e.g
SCCT - Deep Brain Rating, here the right standings:

Code: Select all

 # PLAYER             : RATING    POINTS    GAMES   (%)
 7 Ponder ON 1CPU     : 3500.0     295.5     700   42.2%
 8 Ponder OFF 1CPU    : 3435.3     187.5     516   36.3%

Code: Select all

1   Ponder ON 1CPU   +52/-12/=186 58.00%  145.0/250
2   Ponder OFF 1CPU  +12/-52/=186 42.00%  105.0/250

More than 50 Elo difference is recorded...via direct matches
Via Ordo..vs overall, the Elo difference is close to 60 Elo difference
Please check more carefully next time..

Btw, what do you think about my latest test via Ponder OFF ?
http://outskirts.altervista.org/forum/viewtopic.php?f=95&t=11&start=1730
2 games are recorded as lost on time...only in 12 games...

Greetings