These are some benchmark results put/get 500K entries on three node partitions.
- ONHEAP_TIERED enable off-heap eviction
Put: 107111 ms ~ 4668 msg/s
Get: 524107 ms ~ 954 msg/s
Near cache: 24917 ms ~ 20067 msg/s
Put: 104923 ms ~ 4765 msg/s
Get: 519992 ms ~ 962 msg/s
Near cache: 22889 ms ~ 21845 msg/s
Put: 104104 ms ~ 4803 msg/s
Get: 442893 ms ~ 1129 msg/s
Near cache: 6107 ms ~ 81873 msg/s
I don't know reason why OFFHEAP_TIERED is faster than ONHEAP_TIERED (with config data is not eviction).
I run benchmark same testcase on Hazelcast (on heap), CPU is less than ignite( only 2-3% total).
The results you are getting look strange to me. I tried running your code and configuration on a cluster and for OFFHEAP_TIERED I got results similar to yours, for ONHEAP_TIERED I got results twice as fast as off-heap. I would recommend adding at least a 30-seconds warmup to your code to get more stable and reliable results.
As for the CPU load, I also noticed about 20% load on servers (using top command), however this looks reasonable to me - it means that only single core out of 24 is loaded to about 1/5 of it's capacity.