graceful shutdown for C++ applications

[rakshita04]

Hi Team,

We are using apache-ignite for our applications running on 2 machines and
connected over network.
We are facing some issue where if kill is performed on running application,
it somehow corrupts the node and then node never comes up and keep on
rebooting.
Is there a way to handle this shutdown gracefully? so that there is no data
loss and node corruption.

regards,
Rakshita Chaudhary



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[Wesley Peng-3]

Issuing a command like "kill process_id"  doesn't work?

regards.

On Thu, Jan 7, 2021 at 4:14 PM rakshita04 <[hidden email]> wrote:

Hi Team,

We are using apache-ignite for our applications running on 2 machines and
connected over network.
We are facing some issue where if kill is performed on running application,
it somehow corrupts the node and then node never comes up and keep on
rebooting.
Is there a way to handle this shutdown gracefully? so that there is no data
loss and node corruption.

regards,
Rakshita Chaudhary



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[rakshita04]

it works, the process is stopped but when application is started using the
same Database node, it crashes with below logs on terminal-
Ignite node stopped OK [uptime=00:00:55.197]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x76c9f208, pid=26716, tid=26716
#
# JRE version: OpenJDK Runtime Environment (11.0.6+10) (build
11.0.6+10-post-Debian-1deb10u1)
# Java VM: OpenJDK Server VM (11.0.6+10-post-Debian-1deb10u1, mixed mode, g1
gc, linux-)
# Problematic frame:
# C  [libignite-2.8.0.44294.so.0+0x11208]
ignite::Ignite::SetActive(bool)+0xb

Our application is calling SetActive method after node::start().

The application is recovered only when Databse is deleted and node is
started again.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[stephendarlington]

Not that it excuses the crash, but why are you calling activate every time the node starts? It should be called once, the first time all the nodes are present. The cluster will auto-activate every time after that.

Regards,
Stephen

> On 7 Jan 2021, at 08:56, rakshita04 <[hidden email]> wrote:
>
> it works, the process is stopped but when application is started using the
> same Database node, it crashes with below logs on terminal-
> Ignite node stopped OK [uptime=00:00:55.197]
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x76c9f208, pid=26716, tid=26716
> #
> # JRE version: OpenJDK Runtime Environment (11.0.6+10) (build
> 11.0.6+10-post-Debian-1deb10u1)
> # Java VM: OpenJDK Server VM (11.0.6+10-post-Debian-1deb10u1, mixed mode, g1
> gc, linux-)
> # Problematic frame:
> # C  [libignite-2.8.0.44294.so.0+0x11208]
> ignite::Ignite::SetActive(bool)+0xb
>
> Our application is calling SetActive method after node::start().
>
> The application is recovered only when Databse is deleted and node is
> started again.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[rakshita04]

can SetActive() cause the crash?
is this way okay to terminate the process by kill or there is some better
way?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[stephendarlington]

It shouldn’t cause a crash, but since you don’t need to activate an already active cluster maybe it’s not well tested.

Sending the node a TERM signal (press ^C) is good way to stop a node. 

On 7 Jan 2021, at 09:26, rakshita04 <[hidden email]> wrote:

can SetActive() cause the crash?
is this way okay to terminate the process by kill or there is some better
way?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[rakshita04]

I am also getting below error on my ignite logs-
[20:00:50,515][SEVERE][db-checkpoint-thread-#54][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385, len=21409],
cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b, type=START]]]
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385,
len=21409], cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b,
type=START]

what can cause this?
And how to avoid this problem?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[stephendarlington]

We’d need to see more of the exception. It says it couldn’t write a file, but your snippet doesn’t say why.

> On 7 Jan 2021, at 10:51, rakshita04 <[hidden email]> wrote:
>
> I am also getting below error on my ignite logs-
> [20:00:50,515][SEVERE][db-checkpoint-thread-#54][] Critical system error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.processors.cache.persistence.StorageException: Failed to write
> checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385, len=21409],
> cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b, type=START]]]
> class
> org.apache.ignite.internal.processors.cache.persistence.StorageException:
> Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385,
> len=21409], cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b,
> type=START]
>
> what can cause this?
> And how to avoid this problem?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[rakshita04]

These are the full set of logs, if it helps-
[10:10:56,860][WARNING][main][G] Ignite work directory is not provided,
automatically resolved to: /home/dsudev/ignite-master/work
[10:10:56,873][WARNING][main][G] Consistent ID is not set, it is recommended
to set consistent ID for production clusters (use
IgniteConfiguration.setConsistentId property)
[10:10:57,103][INFO][main][IgniteKernal]

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/  
>>>
>>> ver. 2.8.0#20200226-sha1:341b01df
>>> 2020 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org

[10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a
[10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration
[igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8,
stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1,
dataStreamerPoolSize=8, utilityCachePoolSize=8,
utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8,
sqlQryHistSize=1000, dfltQryTimeout=0,
igniteHome=/home/dsudev/ignite-master,
igniteWorkDir=/home/dsudev/ignite-master/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb,
nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [],
marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000,
netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3,
metricsHistSize=10000, metricsUpdateFreq=2000,
metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi
[addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10,
reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false,
clientReconnectDisabled=false, internalLsnr=null,
skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
commSpi=TcpCommunicationSpi [connectGate=null,
connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459,
chConnPlc=null, enableForcibleNodeKill=false,
enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100,
locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false,
idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000,
reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0,
slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null,
usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true,
filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0,
sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1,
selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1],
stopping=false, metricsLsnr=null],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe,
addrRslvr=null,
encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7,
clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=10000,
rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0,
rebalanceBatchSize=524288, txCfg=TransactionConfiguration
[txSerEnabled=false, dfltIsolation=REPEATABLE_READ,
dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0,
txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000,
pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null,
txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true,
discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100,
locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100,
failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211,
noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768,
idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0,
selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false,
sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8,
msgInterceptor=null], odbcCfg=null, warmupClos=null,
atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED,
backups=1, aff=null, grpName=null], classLdr=null,
sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS,
keyStoreFile=/home/dsudev/config/keystore.jks,
trustStoreFile=/home/dsudev/config/truststore.jks],
platformCfg=PlatformConfiguration [], binaryCfg=BinaryConfiguration
[idMapper=BinaryBaseIdMapper [isLowerCase=true],
nameMapper=BinaryBaseNameMapper [isSimpleName=true], serializer=null,
compactFooter=false], memCfg=null, pstCfg=null,
dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040,
sysRegionMaxSize=104857600, pageSize=4096, concLvl=0,
dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=419430400,
initSize=104857600, swapPath=null, pageEvictionMode=DISABLED,
evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false,
metricsSubIntervalCount=5, metricsRateTimeInterval=60000,
persistenceEnabled=true, checkpointPageBufSize=0,
lazyMemoryAllocation=true], dataRegions=null,
storagePath=/media/linux_hdd/keymanagementdata, checkpointFreq=180000,
lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL,
walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10,
walSegmentSize=67108864, walPath=/media/linux_hdd/keymanagementdata/db/wal,
walArchivePath=/media/linux_hdd/keymanagementdata/db/wal/archive,
metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0,
walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864,
alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1459c75,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
walCompactionEnabled=false, walCompactionLevel=1,
checkpointReadLockTimeout=null, walPageCompression=DISABLED,
walPageCompressionLevel=null], activeOnStart=true, autoActivation=true,
longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true,
thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true,
sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration
[maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000,
authEnabled=false, failureHnd=null, commFailureRslvr=null]
[10:10:57,191][INFO][main][IgniteKernal] Daemon mode: off
[10:10:57,193][INFO][main][IgniteKernal] OS: Linux 4.19.0-8-686 i386
[10:10:57,193][INFO][main][IgniteKernal] OS user: root
[10:10:57,211][INFO][main][IgniteKernal] PID: 1461
[10:10:57,222][INFO][main][IgniteKernal] Language runtime: Java Platform API
Specification ver. 11
[10:10:57,222][INFO][main][IgniteKernal] VM information: OpenJDK Runtime
Environment 11.0.6+10-post-Debian-1deb10u1 Debian OpenJDK Server VM
11.0.6+10-post-Debian-1deb10u1
[10:10:57,223][INFO][main][IgniteKernal] VM total memory: 0.48GB
[10:10:57,223][INFO][main][IgniteKernal] Remote Management [restart: off,
REST: on, JMX (remote: off)]
[10:10:57,223][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=true,
config=null]
[10:10:57,223][INFO][main][IgniteKernal]
IGNITE_HOME=/home/dsudev/ignite-master
[10:10:57,223][INFO][main][IgniteKernal] VM arguments:
[-DIGNITE_HOME=/home/dsudev/ignite-master, -Xms256m, -Xmx512m,
-Dfile.encoding=UTF-8,
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED,
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED,
--add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED,
--add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED,
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED,
--illegal-access=permit]
[10:10:57,224][INFO][main][IgniteKernal] System cache's DataRegion size is
configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize
property to change the setting.
[10:10:57,224][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc'
dataRegion: ['ignite-sys-cache'], in 'default' dataRegion: ['DSU_DataBase']]
[10:10:57,224][WARNING][main][IgniteKernal] Peer class loading is enabled
(disable it in production for performance and deployment consistency
reasons)
[10:10:57,224][WARNING][main][IgniteKernal] Please set system property
'-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed
environments.
[10:10:57,232][INFO][main][IgniteKernal] 3-rd party licenses can be found
at: /home/dsudev/ignite-master/libs/licenses
[10:10:57,240][WARNING][pub-#22][GridDiagnostic] Initial heap size is 256MB
(should be no less than 512MB, use -Xms512m -Xmx512m).
[10:10:57,441][INFO][main][IgnitePluginProcessor] Configured plugins:
[10:10:57,441][INFO][main][IgnitePluginProcessor]   ^-- None
[10:10:57,442][INFO][main][IgnitePluginProcessor]
[10:10:57,451][INFO][main][FailureProcessor] Configured failure handler:
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[10:10:58,881][INFO][main][TcpCommunicationSpi] Successfully bound
communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0,
selectorsCnt=4, selectorSpins=0, pairedConn=false]
[10:10:58,903][WARNING][main][TcpCommunicationSpi] Message queue limit is
set to 0 which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[10:10:58,975][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled
(to enable configure any GridCheckpointSpi implementation)
[10:10:58,999][WARNING][main][GridCollisionManager] Collision resolution is
disabled (all jobs will be activated upon arrival).
[10:10:59,005][INFO][main][IgniteKernal] Security status
[authentication=off, tls/ssl=on]
[10:10:59,263][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639]
[10:10:59,276][INFO][main][PdsFoldersResolver] Successfully created new
persistent storage folder
[/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256]
[10:10:59,276][INFO][main][PdsFoldersResolver] Consistent ID used for local
node is [9aae0abd-bec2-4d53-93c2-cdf34201d256] according to persistence data
storage folders
[10:10:59,277][INFO][main][CacheObjectBinaryProcessorImpl] Resolved
directory for serialized binary metadata:
/home/dsudev/ignite-master/work/binary_meta/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,655][INFO][main][FilePageStoreManager] Resolved page store work
directory:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead
log work directory:
/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead
log archive directory:
/media/linux_hdd/keymanagementdata/db/wal/archive/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,786][INFO][main][FileWriteAheadLogManager] Reached logical end of
the segment for file
/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal
[10:10:59,791][INFO][main][FileHandleManagerImpl] Initialized write-ahead
log manager [mode=LOG_ONLY]
[10:10:59,796][WARNING][main][GridCacheDatabaseSharedManager]
DataRegionConfiguration.maxWalArchiveSize instead
DataRegionConfiguration.walHistorySize would be used for removing old
archive wal files
[10:10:59,830][INFO][main][GridCacheDatabaseSharedManager] Configured data
regions initialized successfully [total=4]
[10:10:59,863][INFO][main][PartitionsEvictManager] Evict partition permits=2
[10:10:59,950][WARNING][main][IgniteH2Indexing] Serialization of Java
objects in H2 was enabled.
[10:11:00,422][INFO][main][ClientListenerProcessor] Client connector
processor has started on TCP port 10800
[10:11:00,530][INFO][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[10:11:00,592][WARNING][main][PlatformProcessorImpl] Binary configuration is
automatically initiated, note that binary name mapper is set to
BinaryBaseNameMapper [isSimpleName=true] and binary ID mapper is set to
BinaryBaseIdMapper [isLowerCase=true] (other nodes must have the same binary
name and ID mapper types).
[10:11:00,632][INFO][main][IgniteKernal] Non-loopback local IPs:
192.168.10.5, 192.168.100.10, 192.168.111.111, 192.168.12.112,
192.168.15.111, fe80:0:0:0:a00:27ff:fe72:e439%enp0s9
[10:11:00,632][INFO][main][IgniteKernal] Enabled local MACs: 08002706A1F2,
0800270A9679, 08002772E439, 080027BD35B7
[10:11:00,654][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:00,679][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,683][INFO][main][GridCacheDatabaseSharedManager] Checking memory
state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0],
lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:00,759][INFO][main][GridCacheDatabaseSharedManager] Applying lost
cache updates since last checkpoint record [lastMarked=FileWALPointer
[idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:00,767][INFO][main][GridCacheDatabaseSharedManager] Finished applying
WAL changes [updatesApplied=0, time=10 ms]
[10:11:00,767][INFO][main][GridCacheProcessor] Restoring partition state for
local groups.
[10:11:00,768][INFO][main][GridCacheProcessor] Finished restoring partition
state for local groups [groupsProcessed=0, partitionsProcessed=0, time=0ms]
[10:11:00,866][INFO][main][FilePageStoreManager] Cleanup cache stores
[total=1, left=0, cleanFiles=false]
[10:11:00,898][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,898][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,910][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,929][INFO][main][GridCacheDatabaseSharedManager] Configured data
regions started successfully [total=4]
[10:11:00,930][INFO][main][GridCacheDatabaseSharedManager] Starting binary
memory restore for: [-2100569601, -510489548]
[10:11:01,167][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:01,169][INFO][main][GridCacheDatabaseSharedManager] Checking memory
state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0],
lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:01,182][INFO][main][FileWriteAheadLogManager] Resuming logging to WAL
segment
[file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal,
offset=0, ver=2]
[10:11:01,613][INFO][main][GridCacheProcessor] Started cache in recovery
mode [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc,
mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false]
[10:11:01,632][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=400.0 MiB, pages=99259, tableSize=7.7 MiB,
checkpointBuffer=256.0 MiB]
[10:11:01,740][INFO][main][GridCacheProcessor] Started cache in recovery
mode [name=DSU_DataBase, id=-510489548, dataRegionName=default,
mode=PARTITIONED, atomicity=ATOMIC, backups=1, mvcc=false]
[10:11:01,753][INFO][main][GridCacheDatabaseSharedManager] Binary recovery
performed in 823 ms.
[10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Applying lost
cache updates since last checkpoint record [lastMarked=FileWALPointer
[idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:01,888][INFO][main][GridCacheDatabaseSharedManager] Finished applying
WAL changes [updatesApplied=0, time=127 ms]
[10:11:01,888][INFO][main][GridCacheProcessor] Restoring partition state for
local groups.
[10:11:01,976][INFO][main][GridCacheProcessor] Finished restoring partition
state for local groups [groupsProcessed=2, partitionsProcessed=0, time=93ms]
[10:11:02,176][INFO][main][TcpDiscoverySpi] Connection check threshold is
calculated: 10000
[10:11:02,244][INFO][disco-notifier-worker-#38][MvccProcessorImpl] Assigned
mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=0], nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
ver=1599885662229, local=true, initialized=false]]
[10:11:02,320][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'null' to
'false'
[10:11:02,328][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustTimeout' was changed from 'null' to
'300000'
[10:11:02,604][INFO][main][IgniteKernal] Performance suggestions for grid
(fix if possible)
[10:11:02,605][INFO][main][IgniteKernal] To disable, set
-DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[10:11:02,605][INFO][main][IgniteKernal]   ^-- Decrease number of backups
(set 'backups' to 0)
[10:11:02,606][INFO][main][IgniteKernal]   ^-- Disable fully synchronous
writes (set 'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC)
[10:11:02,607][INFO][main][IgniteKernal]   ^-- Switch to the most recent 1.8
JVM version
[10:11:02,608][INFO][main][IgniteKernal]   ^-- Set max direct memory size if
getting 'OOME: Direct buffer memory' (add
'-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[10:11:02,608][INFO][main][IgniteKernal]   ^-- Disable processing of calls
to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
[10:11:02,610][INFO][main][IgniteKernal] Refer to this page for more
performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[10:11:02,610][INFO][main][IgniteKernal]
[10:11:02,611][INFO][main][IgniteKernal] To start Console Management &
Monitoring run ignitevisorcmd.{sh|bat}
[10:11:02,615][INFO][main][IgniteKernal] Data Regions Configured:
[10:11:02,616][INFO][main][IgniteKernal]   ^-- default [initSize=100.0 MiB,
maxSize=400.0 MiB, persistence=true, lazyMemoryAllocation=true]
[10:11:02,616][INFO][main][IgniteKernal]

>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.8.0#20200226-sha1:341b01dfd8abf2d9b01d468ad1bb26dfe84ac4f6
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 4.19.0-8-686 i386
>>> CPU(s): 1
>>> Heap: 0.48GB
>>> VM name: 1461@dsudev
>>> Local node [ID=0AAD560A-FAAD-4F86-B65C-A7F161BF2639, order=1,
>>> clientMode=false]
>>> Local node addresses: [192.168.15.111/0:0:0:0:0:0:0:1%lo,
>>> 192.168.111.111/127.0.0.1, 192.168.100.10/192.168.10.5,
>>> 192.168.10.5/192.168.100.10, 192.168.12.112/192.168.111.111,
>>> /192.168.12.112, /192.168.15.111]
>>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500

[10:11:02,623][INFO][main][IgniteKernal] >>> Ignite cluster is not active
(limited functionality available). Use control.(sh|bat) script or
IgniteCluster interface to activate.
[10:11:02,630][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1,
locNode=0aad560a, servers=1, clients=0, state=INACTIVE, CPUs=1,
offheap=0.39GB, heap=0.48GB]
[10:11:02,631][INFO][main][G] Node started : [stage="Configure system pool"
(158 ms),stage="Start managers" (1940 ms),stage="Configure binary metadata"
(324 ms),stage="Start processors" (1268 ms),stage="Init metastore" (241
ms),stage="Init and start regions" (63 ms),stage="Restore binary memory"
(824 ms),stage="Restore logical state" (222 ms),stage="Finish recovery" (183
ms),stage="Join topology" (81 ms),stage="Await transition" (17
ms),stage="Await exchange" (372 ms),stage="Total time" (5693 ms)]
[10:11:02,666][INFO][main][GridClusterStateProcessor] Sending activate
request with BaselineTopology BaselineTopology [id=0,
branchingHash=-173057973, branchingType='New BaselineTopology',
baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]]
[10:11:02,711][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Received activate request with BaselineTopology[id=0]
[10:11:02,713][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Started state transition: true
[10:11:02,724][INFO][exchange-worker-#40][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true,
evt=DISCOVERY_CUSTOM_EVT, evtNode=0aad560a-faad-4f86-b65c-a7f161bf2639,
customEvt=ChangeGlobalStateMessage
[id=86bbd908471-259db0fa-9d36-40c3-ad9f-b8359727dc28,
reqId=e5a2e4ff-c7aa-4924-bc83-7f86c3f16ea1,
initiatingNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, activate=true,
baselineTopology=BaselineTopology [id=0, branchingHash=-173057973,
branchingType='New BaselineTopology',
baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]],
forceChangeBaselineTopology=false, timestamp=1599885662666],
allowMerge=false, exchangeFreeSwitch=false]
[10:11:02,725][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Start activation process [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,726][INFO][exchange-worker-#40][FilePageStoreManager] Resolved
page store work directory:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:11:02,730][INFO][exchange-worker-#40][FileWriteAheadLogManager] Resuming
logging to WAL segment
[file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal,
offset=13069, ver=2]
[10:11:02,732][INFO][exchange-worker-#40][GridClusterStateProcessor] Writing
BaselineTopology[id=0]
[10:11:02,768][INFO][exchange-worker-#40][GridCacheDatabaseSharedManager]
Finish recovery performed in 42 ms.
[10:11:02,770][INFO][exchange-worker-#40][msg] Components activation
performed in 44 ms.
[10:11:02,840][INFO][exchange-worker-#40][GridCacheProcessor] Finished
recovery for cache [cache=DSU_DataBase, grp=DSU_DataBase,
startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,840][INFO][sys-#49][GridCacheProcessor] Finished recovery for
cache [cache=ignite-sys-cache, grp=ignite-sys-cache,
startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,886][INFO][exchange-worker-#40][GridAffinityAssignmentCache] Local
node affinity assignment distribution is not ideal [cache=DSU_DataBase,
expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00,
actualBackups=0, warningThreshold=50.00%]
[10:11:02,909][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Successfully activated caches [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,922][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms,
futInfo=NA, mode=LOCAL]
[10:11:03,485][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:03,513][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=d6caae21-b222-4a61-9956-671e24030c13,
startPtr=FileWALPointer [idx=0, fileOff=142444, len=21425],
checkpointBeforeLockTime=87ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=56ms, checkpointLockHoldTime=80ms,
walCpRecordFsyncDuration=9ms, writeCheckpointEntryDuration=10ms,
splitAndSortCpPagesDuration=0ms,  pages=20, reason='node started']
[10:11:03,545][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager]
Checkpoint finished [cpId=d6caae21-b222-4a61-9956-671e24030c13, pages=20,
markPos=FileWALPointer [idx=0, fileOff=142444, len=21425],
walSegmentsCleared=0, walSegmentsCovered=[], markDuration=109ms,
pagesWrite=5ms, fsync=26ms, total=227ms]
[10:11:03,667][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Received state change finish message: true
[10:11:03,669][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
err=null, rebalanced=true, wasRebalanced=false]
[10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy
initialization, cacheName=ignite-sys-cache,
localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639
[10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy
initialization, cacheName=DSU_DataBase,
localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639
[10:11:03,766][INFO][sys-#53][DistributedBaselineConfiguration] Baseline
auto-adjust is 'disabled' with timeout='300000' ms
[10:11:03,767][INFO][sys-#53][GridClusterStateProcessor] Successfully
performed final activation steps
[nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:03,848][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Completed partition exchange
[localNode=0aad560a-faad-4f86-b65c-a7f161bf2639,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
evtNode=TcpDiscoveryNode [id=0aad560a-faad-4f86-b65c-a7f161bf2639,
consistentId=9aae0abd-bec2-4d53-93c2-cdf34201d256, addrs=ArrayList
[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.10.5, 192.168.100.10,
192.168.111.111, 192.168.12.112, 192.168.15.111], sockAddrs=HashSet
[/192.168.111.111:47500, /192.168.15.111:47500, /192.168.12.112:47500,
/192.168.10.5:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
/192.168.100.10:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1599885662175, loc=true, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=1]]
[10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
stage="Waiting in exchange queue" (1 ms), stage="Exchange parameters
initialization" (2 ms), stage="Update caches registry" (57 ms), stage="Start
caches" (67 ms), stage="Affinity initialization on cache group start" (59
ms), stage="Determine exchange type" (9 ms), stage="Preloading notification"
(0 ms), stage="WAL history reservation" (1 ms), stage="Wait partitions
release" (5 ms), stage="Restore partition states" (22 ms), stage="After
states restored callback" (536 ms), stage="Waiting for all single messages"
(0 ms), stage="Affinity recalculation (crd)" (0 ms), stage="Collect update
counters and create affinity messages" (0 ms), stage="Validate partitions
states" (8 ms), stage="Assign partitions states" (54 ms), stage="Ideal
affinity diff calculation (enforced)" (41 ms), stage="Apply update counters"
(4 ms), stage="Full message preparing" (70 ms), stage="Full message sending"
(0 ms), stage="State finish message sending" (2 ms), stage="Exchange done"
(180 ms), stage="Total time" (1118 ms)]
[10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
stage="Affinity initialization on cache group start [grp=DSU_DataBase]" (59
ms) (parent=Affinity initialization on cache group start), stage="Affinity
initialization on cache group start [grp=ignite-sys-cache]" (37 ms)
(parent=Affinity initialization on cache group start), stage="Affinity
centralized initialization (crd) [grp=ignite-sys-cache, crd=true]" (5 ms)
(parent=Determine exchange type), stage="Affinity centralized initialization
(crd) [grp=DSU_DataBase, crd=true]" (5 ms) (parent=Determine exchange type),
stage="Restore partition states [grp=DSU_DataBase]" (22 ms) (parent=Restore
partition states), stage="Restore partition states [grp=ignite-sys-cache]"
(14 ms) (parent=Restore partition states), stage="Affinity recalculation
(partitions availability) [grp=DSU_DataBase]" (40 ms) (parent=Ideal affinity
diff calculation (enforced)), stage="Affinity recalculation (partitions
availability) [grp=ignite-sys-cache]" (14 ms) (parent=Ideal affinity diff
calculation (enforced))]
[10:11:03,849][INFO][exchange-worker-#40][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true]
[10:11:03,918][INFO][exchange-worker-#40][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT,
node=0aad560a-faad-4f86-b65c-a7f161bf2639]
[10:11:05,808][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustTimeout' was changed from '300000' to
'5'
[10:11:05,817][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'false' to
'true'
[10:12:02,608][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:01:00.002]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-95.83%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=32MB, free=93.48%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=8, qSize=0]
[10:12:12,305][INFO][ignite-update-notifier-timer][GridUpdateNotifier]
Update status is not available.
[10:13:02,616][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:02:00.020]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-97.78%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=35MB, free=92.91%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[10:14:02,624][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:03:00.025]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-98.63%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=36MB, free=92.58%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[10:14:03,465][SEVERE][db-checkpoint-thread-#54][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409],
cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]]
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931,
len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8,
type=START]
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3248)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:4220)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3738)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3623)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.NoSuchFileException:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/cp/1599885843404-78332869-fedc-4687-9202-180b8de1c2b8-START.bin.tmp
        at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at
java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201)
        at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253)
        at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:66)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:44)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3231)
        ... 5 more
[10:14:03,499][WARNING][db-checkpoint-thread-#54][CacheDiagnosticManager]
Page locks dump:

Thread=[name=db-checkpoint-thread-#54, id=84], state=RUNNABLE
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#54 time=(1599885843482,
2020-09-12 10:14:03.482)


Thread=[name=dms-writer-thread-#45, id=75], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#45 time=(1599885843482, 2020-09-12
10:14:03.482)


Thread=[name=exchange-worker-#40, id=69], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#40 time=(1599885843482, 2020-09-12
10:14:03.482)


Thread=[name=main, id=1], state=RUNNABLE
Locked pages = []
Locked pages log: name=main time=(1599885843482, 2020-09-12 10:14:03.482)



[10:14:03,499][SEVERE][db-checkpoint-thread-#54][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409],
cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]]




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

[ilya.kasnacheev]

Hello!

This will happen when this file is deleted while the instance is running. Not sure who deleted it. Maybe you tried to start another node with the same consistent id in the background?

You should avoid calling setActive() every time since it will lead to data loss.

Regards,

чт, 7 янв. 2021 г. в 14:04, rakshita04 <[hidden email]>:

These are the full set of logs, if it helps-
[10:10:56,860][WARNING][main][G] Ignite work directory is not provided,
automatically resolved to: /home/dsudev/ignite-master/work
[10:10:56,873][WARNING][main][G] Consistent ID is not set, it is recommended
to set consistent ID for production clusters (use
IgniteConfiguration.setConsistentId property)
[10:10:57,103][INFO][main][IgniteKernal]

>>>    __________  ________________ 
>>>   /  _/ ___/ |/ /  _/_  __/ __/ 
>>>  _/ // (7 7    // /  / / / _/   
>>> /___/\___/_/|_/___/ /_/ /___/   
>>>
>>> ver. 2.8.0#20200226-sha1:341b01df
>>> 2020 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org

[10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a
[10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration
[igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8,
stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1,
dataStreamerPoolSize=8, utilityCachePoolSize=8,
utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8,
sqlQryHistSize=1000, dfltQryTimeout=0,
igniteHome=/home/dsudev/ignite-master,
igniteWorkDir=/home/dsudev/ignite-master/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb,
nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [],
marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000,
netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3,
metricsHistSize=10000, metricsUpdateFreq=2000,
metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi
[addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10,
reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false,
clientReconnectDisabled=false, internalLsnr=null,
skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
commSpi=TcpCommunicationSpi [connectGate=null,
connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459,
chConnPlc=null, enableForcibleNodeKill=false,
enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100,
locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false,
idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000,
reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0,
slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null,
usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true,
filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0,
sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1,
selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1],
stopping=false, metricsLsnr=null],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe,
addrRslvr=null,
encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7,
clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=10000,
rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0,
rebalanceBatchSize=524288, txCfg=TransactionConfiguration
[txSerEnabled=false, dfltIsolation=REPEATABLE_READ,
dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0,
txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000,
pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null,
txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true,
discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100,
locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100,
failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211,
noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768,
idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0,
selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false,
sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8,
msgInterceptor=null], odbcCfg=null, warmupClos=null,
atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED,
backups=1, aff=null, grpName=null], classLdr=null,
sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS,
keyStoreFile=/home/dsudev/config/keystore.jks,
trustStoreFile=/home/dsudev/config/truststore.jks],
platformCfg=PlatformConfiguration [], binaryCfg=BinaryConfiguration
[idMapper=BinaryBaseIdMapper [isLowerCase=true],
nameMapper=BinaryBaseNameMapper [isSimpleName=true], serializer=null,
compactFooter=false], memCfg=null, pstCfg=null,
dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040,
sysRegionMaxSize=104857600, pageSize=4096, concLvl=0,
dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=419430400,
initSize=104857600, swapPath=null, pageEvictionMode=DISABLED,
evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false,
metricsSubIntervalCount=5, metricsRateTimeInterval=60000,
persistenceEnabled=true, checkpointPageBufSize=0,
lazyMemoryAllocation=true], dataRegions=null,
storagePath=/media/linux_hdd/keymanagementdata, checkpointFreq=180000,
lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL,
walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10,
walSegmentSize=67108864, walPath=/media/linux_hdd/keymanagementdata/db/wal,
walArchivePath=/media/linux_hdd/keymanagementdata/db/wal/archive,
metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0,
walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864,
alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1459c75,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
walCompactionEnabled=false, walCompactionLevel=1,
checkpointReadLockTimeout=null, walPageCompression=DISABLED,
walPageCompressionLevel=null], activeOnStart=true, autoActivation=true,
longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true,
thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true,
sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration
[maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000,
authEnabled=false, failureHnd=null, commFailureRslvr=null]
[10:10:57,191][INFO][main][IgniteKernal] Daemon mode: off
[10:10:57,193][INFO][main][IgniteKernal] OS: Linux 4.19.0-8-686 i386
[10:10:57,193][INFO][main][IgniteKernal] OS user: root
[10:10:57,211][INFO][main][IgniteKernal] PID: 1461
[10:10:57,222][INFO][main][IgniteKernal] Language runtime: Java Platform API
Specification ver. 11
[10:10:57,222][INFO][main][IgniteKernal] VM information: OpenJDK Runtime
Environment 11.0.6+10-post-Debian-1deb10u1 Debian OpenJDK Server VM
11.0.6+10-post-Debian-1deb10u1
[10:10:57,223][INFO][main][IgniteKernal] VM total memory: 0.48GB
[10:10:57,223][INFO][main][IgniteKernal] Remote Management [restart: off,
REST: on, JMX (remote: off)]
[10:10:57,223][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=true,
config=null]
[10:10:57,223][INFO][main][IgniteKernal]
IGNITE_HOME=/home/dsudev/ignite-master
[10:10:57,223][INFO][main][IgniteKernal] VM arguments:
[-DIGNITE_HOME=/home/dsudev/ignite-master, -Xms256m, -Xmx512m,
-Dfile.encoding=UTF-8,
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED,
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED,
--add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED,
--add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED,
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED,
--illegal-access=permit]
[10:10:57,224][INFO][main][IgniteKernal] System cache's DataRegion size is
configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize
property to change the setting.
[10:10:57,224][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc'
dataRegion: ['ignite-sys-cache'], in 'default' dataRegion: ['DSU_DataBase']]
[10:10:57,224][WARNING][main][IgniteKernal] Peer class loading is enabled
(disable it in production for performance and deployment consistency
reasons)
[10:10:57,224][WARNING][main][IgniteKernal] Please set system property
'-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed
environments.
[10:10:57,232][INFO][main][IgniteKernal] 3-rd party licenses can be found
at: /home/dsudev/ignite-master/libs/licenses
[10:10:57,240][WARNING][pub-#22][GridDiagnostic] Initial heap size is 256MB
(should be no less than 512MB, use -Xms512m -Xmx512m).
[10:10:57,441][INFO][main][IgnitePluginProcessor] Configured plugins:
[10:10:57,441][INFO][main][IgnitePluginProcessor]   ^-- None
[10:10:57,442][INFO][main][IgnitePluginProcessor]
[10:10:57,451][INFO][main][FailureProcessor] Configured failure handler:
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
[10:10:58,881][INFO][main][TcpCommunicationSpi] Successfully bound
communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0,
selectorsCnt=4, selectorSpins=0, pairedConn=false]
[10:10:58,903][WARNING][main][TcpCommunicationSpi] Message queue limit is
set to 0 which may lead to potential OOMEs when running cache operations in
FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and
receiver sides.
[10:10:58,975][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled
(to enable configure any GridCheckpointSpi implementation)
[10:10:58,999][WARNING][main][GridCollisionManager] Collision resolution is
disabled (all jobs will be activated upon arrival).
[10:10:59,005][INFO][main][IgniteKernal] Security status
[authentication=off, tls/ssl=on]
[10:10:59,263][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639]
[10:10:59,276][INFO][main][PdsFoldersResolver] Successfully created new
persistent storage folder
[/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256]
[10:10:59,276][INFO][main][PdsFoldersResolver] Consistent ID used for local
node is [9aae0abd-bec2-4d53-93c2-cdf34201d256] according to persistence data
storage folders
[10:10:59,277][INFO][main][CacheObjectBinaryProcessorImpl] Resolved
directory for serialized binary metadata:
/home/dsudev/ignite-master/work/binary_meta/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,655][INFO][main][FilePageStoreManager] Resolved page store work
directory:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead
log work directory:
/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead
log archive directory:
/media/linux_hdd/keymanagementdata/db/wal/archive/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:10:59,786][INFO][main][FileWriteAheadLogManager] Reached logical end of
the segment for file
/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal
[10:10:59,791][INFO][main][FileHandleManagerImpl] Initialized write-ahead
log manager [mode=LOG_ONLY]
[10:10:59,796][WARNING][main][GridCacheDatabaseSharedManager]
DataRegionConfiguration.maxWalArchiveSize instead
DataRegionConfiguration.walHistorySize would be used for removing old
archive wal files
[10:10:59,830][INFO][main][GridCacheDatabaseSharedManager] Configured data
regions initialized successfully [total=4]
[10:10:59,863][INFO][main][PartitionsEvictManager] Evict partition permits=2
[10:10:59,950][WARNING][main][IgniteH2Indexing] Serialization of Java
objects in H2 was enabled.
[10:11:00,422][INFO][main][ClientListenerProcessor] Client connector
processor has started on TCP port 10800
[10:11:00,530][INFO][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[10:11:00,592][WARNING][main][PlatformProcessorImpl] Binary configuration is
automatically initiated, note that binary name mapper is set to
BinaryBaseNameMapper [isSimpleName=true] and binary ID mapper is set to
BinaryBaseIdMapper [isLowerCase=true] (other nodes must have the same binary
name and ID mapper types).
[10:11:00,632][INFO][main][IgniteKernal] Non-loopback local IPs:
192.168.10.5, 192.168.100.10, 192.168.111.111, 192.168.12.112,
192.168.15.111, fe80:0:0:0:a00:27ff:fe72:e439%enp0s9
[10:11:00,632][INFO][main][IgniteKernal] Enabled local MACs: 08002706A1F2,
0800270A9679, 08002772E439, 080027BD35B7
[10:11:00,654][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:00,679][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,683][INFO][main][GridCacheDatabaseSharedManager] Checking memory
state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0],
lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:00,759][INFO][main][GridCacheDatabaseSharedManager] Applying lost
cache updates since last checkpoint record [lastMarked=FileWALPointer
[idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:00,767][INFO][main][GridCacheDatabaseSharedManager] Finished applying
WAL changes [updatesApplied=0, time=10 ms]
[10:11:00,767][INFO][main][GridCacheProcessor] Restoring partition state for
local groups.
[10:11:00,768][INFO][main][GridCacheProcessor] Finished restoring partition
state for local groups [groupsProcessed=0, partitionsProcessed=0, time=0ms]
[10:11:00,866][INFO][main][FilePageStoreManager] Cleanup cache stores
[total=1, left=0, cleanFiles=false]
[10:11:00,898][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,898][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,910][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
[10:11:00,929][INFO][main][GridCacheDatabaseSharedManager] Configured data
regions started successfully [total=4]
[10:11:00,930][INFO][main][GridCacheDatabaseSharedManager] Starting binary
memory restore for: [-2100569601, -510489548]
[10:11:01,167][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:01,169][INFO][main][GridCacheDatabaseSharedManager] Checking memory
state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0],
lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:01,182][INFO][main][FileWriteAheadLogManager] Resuming logging to WAL
segment
[file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal,
offset=0, ver=2]
[10:11:01,613][INFO][main][GridCacheProcessor] Started cache in recovery
mode [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc,
mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false]
[10:11:01,632][INFO][main][PageMemoryImpl] Started page memory
[memoryAllocated=400.0 MiB, pages=99259, tableSize=7.7 MiB,
checkpointBuffer=256.0 MiB]
[10:11:01,740][INFO][main][GridCacheProcessor] Started cache in recovery
mode [name=DSU_DataBase, id=-510489548, dataRegionName=default,
mode=PARTITIONED, atomicity=ATOMIC, backups=1, mvcc=false]
[10:11:01,753][INFO][main][GridCacheDatabaseSharedManager] Binary recovery
performed in 823 ms.
[10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint
status [startMarker=null, endMarker=null]
[10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Applying lost
cache updates since last checkpoint record [lastMarked=FileWALPointer
[idx=0, fileOff=0, len=0],
lastCheckpointId=00000000-0000-0000-0000-000000000000]
[10:11:01,888][INFO][main][GridCacheDatabaseSharedManager] Finished applying
WAL changes [updatesApplied=0, time=127 ms]
[10:11:01,888][INFO][main][GridCacheProcessor] Restoring partition state for
local groups.
[10:11:01,976][INFO][main][GridCacheProcessor] Finished restoring partition
state for local groups [groupsProcessed=2, partitionsProcessed=0, time=93ms]
[10:11:02,176][INFO][main][TcpDiscoverySpi] Connection check threshold is
calculated: 10000
[10:11:02,244][INFO][disco-notifier-worker-#38][MvccProcessorImpl] Assigned
mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=0], nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
ver=1599885662229, local=true, initialized=false]]
[10:11:02,320][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'null' to
'false'
[10:11:02,328][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustTimeout' was changed from 'null' to
'300000'
[10:11:02,604][INFO][main][IgniteKernal] Performance suggestions for grid
(fix if possible)
[10:11:02,605][INFO][main][IgniteKernal] To disable, set
-DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[10:11:02,605][INFO][main][IgniteKernal]   ^-- Decrease number of backups
(set 'backups' to 0)
[10:11:02,606][INFO][main][IgniteKernal]   ^-- Disable fully synchronous
writes (set 'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC)
[10:11:02,607][INFO][main][IgniteKernal]   ^-- Switch to the most recent 1.8
JVM version
[10:11:02,608][INFO][main][IgniteKernal]   ^-- Set max direct memory size if
getting 'OOME: Direct buffer memory' (add
'-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[10:11:02,608][INFO][main][IgniteKernal]   ^-- Disable processing of calls
to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
[10:11:02,610][INFO][main][IgniteKernal] Refer to this page for more
performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[10:11:02,610][INFO][main][IgniteKernal]
[10:11:02,611][INFO][main][IgniteKernal] To start Console Management &
Monitoring run ignitevisorcmd.{sh|bat}
[10:11:02,615][INFO][main][IgniteKernal] Data Regions Configured:
[10:11:02,616][INFO][main][IgniteKernal]   ^-- default [initSize=100.0 MiB,
maxSize=400.0 MiB, persistence=true, lazyMemoryAllocation=true]
[10:11:02,616][INFO][main][IgniteKernal]

>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.8.0#20200226-sha1:341b01dfd8abf2d9b01d468ad1bb26dfe84ac4f6
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 4.19.0-8-686 i386
>>> CPU(s): 1
>>> Heap: 0.48GB
>>> VM name: 1461@dsudev
>>> Local node [ID=0AAD560A-FAAD-4F86-B65C-A7F161BF2639, order=1,
>>> clientMode=false]
>>> Local node addresses: [<a href="http://192.168.15.111/0:0:0:0:0:0:0:1%lo" rel="noreferrer" target="_blank">192.168.15.111/0:0:0:0:0:0:0:1%lo,
>>> 192.168.111.111/127.0.0.1, 192.168.100.10/192.168.10.5,
>>> 192.168.10.5/192.168.100.10, 192.168.12.112/192.168.111.111,
>>> /192.168.12.112, /192.168.15.111]
>>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500

[10:11:02,623][INFO][main][IgniteKernal] >>> Ignite cluster is not active
(limited functionality available). Use control.(sh|bat) script or
IgniteCluster interface to activate.
[10:11:02,630][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1,
locNode=0aad560a, servers=1, clients=0, state=INACTIVE, CPUs=1,
offheap=0.39GB, heap=0.48GB]
[10:11:02,631][INFO][main][G] Node started : [stage="Configure system pool"
(158 ms),stage="Start managers" (1940 ms),stage="Configure binary metadata"
(324 ms),stage="Start processors" (1268 ms),stage="Init metastore" (241
ms),stage="Init and start regions" (63 ms),stage="Restore binary memory"
(824 ms),stage="Restore logical state" (222 ms),stage="Finish recovery" (183
ms),stage="Join topology" (81 ms),stage="Await transition" (17
ms),stage="Await exchange" (372 ms),stage="Total time" (5693 ms)]
[10:11:02,666][INFO][main][GridClusterStateProcessor] Sending activate
request with BaselineTopology BaselineTopology [id=0,
branchingHash=-173057973, branchingType='New BaselineTopology',
baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]]
[10:11:02,711][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Received activate request with BaselineTopology[id=0]
[10:11:02,713][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Started state transition: true
[10:11:02,724][INFO][exchange-worker-#40][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true,
evt=DISCOVERY_CUSTOM_EVT, evtNode=0aad560a-faad-4f86-b65c-a7f161bf2639,
customEvt=ChangeGlobalStateMessage
[id=86bbd908471-259db0fa-9d36-40c3-ad9f-b8359727dc28,
reqId=e5a2e4ff-c7aa-4924-bc83-7f86c3f16ea1,
initiatingNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, activate=true,
baselineTopology=BaselineTopology [id=0, branchingHash=-173057973,
branchingType='New BaselineTopology',
baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]],
forceChangeBaselineTopology=false, timestamp=1599885662666],
allowMerge=false, exchangeFreeSwitch=false]
[10:11:02,725][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Start activation process [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,726][INFO][exchange-worker-#40][FilePageStoreManager] Resolved
page store work directory:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256
[10:11:02,730][INFO][exchange-worker-#40][FileWriteAheadLogManager] Resuming
logging to WAL segment
[file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal,
offset=13069, ver=2]
[10:11:02,732][INFO][exchange-worker-#40][GridClusterStateProcessor] Writing
BaselineTopology[id=0]
[10:11:02,768][INFO][exchange-worker-#40][GridCacheDatabaseSharedManager]
Finish recovery performed in 42 ms.
[10:11:02,770][INFO][exchange-worker-#40][msg] Components activation
performed in 44 ms.
[10:11:02,840][INFO][exchange-worker-#40][GridCacheProcessor] Finished
recovery for cache [cache=DSU_DataBase, grp=DSU_DataBase,
startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,840][INFO][sys-#49][GridCacheProcessor] Finished recovery for
cache [cache=ignite-sys-cache, grp=ignite-sys-cache,
startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,886][INFO][exchange-worker-#40][GridAffinityAssignmentCache] Local
node affinity assignment distribution is not ideal [cache=DSU_DataBase,
expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00,
actualBackups=0, warningThreshold=50.00%]
[10:11:02,909][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Successfully activated caches [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639,
client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:02,922][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms,
futInfo=NA, mode=LOCAL]
[10:11:03,485][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:03,513][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=d6caae21-b222-4a61-9956-671e24030c13,
startPtr=FileWALPointer [idx=0, fileOff=142444, len=21425],
checkpointBeforeLockTime=87ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=56ms, checkpointLockHoldTime=80ms,
walCpRecordFsyncDuration=9ms, writeCheckpointEntryDuration=10ms,
splitAndSortCpPagesDuration=0ms,  pages=20, reason='node started']
[10:11:03,545][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager]
Checkpoint finished [cpId=d6caae21-b222-4a61-9956-671e24030c13, pages=20,
markPos=FileWALPointer [idx=0, fileOff=142444, len=21425],
walSegmentsCleared=0, walSegmentsCovered=[], markDuration=109ms,
pagesWrite=5ms, fsync=26ms, total=227ms]
[10:11:03,667][INFO][disco-notifier-worker-#38][GridClusterStateProcessor]
Received state change finish message: true
[10:11:03,669][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
err=null, rebalanced=true, wasRebalanced=false]
[10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy
initialization, cacheName=ignite-sys-cache,
localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639
[10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy
initialization, cacheName=DSU_DataBase,
localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639
[10:11:03,766][INFO][sys-#53][DistributedBaselineConfiguration] Baseline
auto-adjust is 'disabled' with timeout='300000' ms
[10:11:03,767][INFO][sys-#53][GridClusterStateProcessor] Successfully
performed final activation steps
[nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[10:11:03,848][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Completed partition exchange
[localNode=0aad560a-faad-4f86-b65c-a7f161bf2639,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
evtNode=TcpDiscoveryNode [id=0aad560a-faad-4f86-b65c-a7f161bf2639,
consistentId=9aae0abd-bec2-4d53-93c2-cdf34201d256, addrs=ArrayList
[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.10.5, 192.168.100.10,
192.168.111.111, 192.168.12.112, 192.168.15.111], sockAddrs=HashSet
[/192.168.111.111:47500, /192.168.15.111:47500, /192.168.12.112:47500,
/192.168.10.5:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
/192.168.100.10:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1599885662175, loc=true, ver=2.8.0#20200226-sha1:341b01df,
isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=1]]
[10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
stage="Waiting in exchange queue" (1 ms), stage="Exchange parameters
initialization" (2 ms), stage="Update caches registry" (57 ms), stage="Start
caches" (67 ms), stage="Affinity initialization on cache group start" (59
ms), stage="Determine exchange type" (9 ms), stage="Preloading notification"
(0 ms), stage="WAL history reservation" (1 ms), stage="Wait partitions
release" (5 ms), stage="Restore partition states" (22 ms), stage="After
states restored callback" (536 ms), stage="Waiting for all single messages"
(0 ms), stage="Affinity recalculation (crd)" (0 ms), stage="Collect update
counters and create affinity messages" (0 ms), stage="Validate partitions
states" (8 ms), stage="Assign partitions states" (54 ms), stage="Ideal
affinity diff calculation (enforced)" (41 ms), stage="Apply update counters"
(4 ms), stage="Full message preparing" (70 ms), stage="Full message sending"
(0 ms), stage="State finish message sending" (2 ms), stage="Exchange done"
(180 ms), stage="Total time" (1118 ms)]
[10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=1,
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1],
stage="Affinity initialization on cache group start [grp=DSU_DataBase]" (59
ms) (parent=Affinity initialization on cache group start), stage="Affinity
initialization on cache group start [grp=ignite-sys-cache]" (37 ms)
(parent=Affinity initialization on cache group start), stage="Affinity
centralized initialization (crd) [grp=ignite-sys-cache, crd=true]" (5 ms)
(parent=Determine exchange type), stage="Affinity centralized initialization
(crd) [grp=DSU_DataBase, crd=true]" (5 ms) (parent=Determine exchange type),
stage="Restore partition states [grp=DSU_DataBase]" (22 ms) (parent=Restore
partition states), stage="Restore partition states [grp=ignite-sys-cache]"
(14 ms) (parent=Restore partition states), stage="Affinity recalculation
(partitions availability) [grp=DSU_DataBase]" (40 ms) (parent=Ideal affinity
diff calculation (enforced)), stage="Affinity recalculation (partitions
availability) [grp=ignite-sys-cache]" (14 ms) (parent=Ideal affinity diff
calculation (enforced))]
[10:11:03,849][INFO][exchange-worker-#40][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true]
[10:11:03,918][INFO][exchange-worker-#40][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT,
node=0aad560a-faad-4f86-b65c-a7f161bf2639]
[10:11:05,808][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustTimeout' was changed from '300000' to
'5'
[10:11:05,817][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration]
Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'false' to
'true'
[10:12:02,608][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:01:00.002]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-95.83%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=32MB, free=93.48%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=8, qSize=0]
[10:12:12,305][INFO][ignite-update-notifier-timer][GridUpdateNotifier]
Update status is not available.
[10:13:02,616][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:02:00.020]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-97.78%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=35MB, free=92.91%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[10:14:02,624][INFO][grid-timeout-worker-#23][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=0aad560a, uptime=00:03:00.025]
    ^-- H/N/C [hosts=1, nodes=1, CPUs=1]
    ^-- CPU [cur=-100%, avg=-98.63%, GC=0%]
    ^-- PageMemory [pages=20]
    ^-- Heap [used=36MB, free=92.58%, comm=247MB]
    ^-- Off-heap [used=0MB, free=99.99%, comm=600MB]
    ^--   sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
    ^--   default region [used=0MB, free=100%, comm=400MB]
    ^--   metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB]
    ^--   TxLog region [used=0MB, free=100%, comm=100MB]
    ^-- Ignite persistence [used=0MB]
    ^--   sysMemPlc region [used=0MB]
    ^--   default region [used=0MB]
    ^--   metastoreMemPlc region [used=0MB]
    ^--   TxLog region [used=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=0, idle=6, qSize=0]
[10:14:03,465][SEVERE][db-checkpoint-thread-#54][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409],
cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]]
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931,
len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8,
type=START]
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3248)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:4220)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3738)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3623)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.NoSuchFileException:
/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/cp/1599885843404-78332869-fedc-4687-9202-180b8de1c2b8-START.bin.tmp
        at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at
java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201)
        at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253)
        at
java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:66)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:44)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3231)
        ... 5 more
[10:14:03,499][WARNING][db-checkpoint-thread-#54][CacheDiagnosticManager]
Page locks dump:

Thread=[name=db-checkpoint-thread-#54, id=84], state=RUNNABLE
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#54 time=(1599885843482,
2020-09-12 10:14:03.482)


Thread=[name=dms-writer-thread-#45, id=75], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#45 time=(1599885843482, 2020-09-12
10:14:03.482)


Thread=[name=exchange-worker-#40, id=69], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=exchange-worker-#40 time=(1599885843482, 2020-09-12
10:14:03.482)


Thread=[name=main, id=1], state=RUNNABLE
Locked pages = []
Locked pages log: name=main time=(1599885843482, 2020-09-12 10:14:03.482)



[10:14:03,499][SEVERE][db-checkpoint-thread-#54][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409],
cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]]




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/