[data-streamer-stripe-2-#35][] Critical system error detected.

classic Classic list List threaded Threaded
3 messages Options
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

[data-streamer-stripe-2-#35][] Critical system error detected.

Hello.

My ignite crash with following errors.

[16:39:31,163][SEVERE][data-streamer-stripe-2-#35][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize
partition file:
/XXXX/data/persistence/node00-51465194-9ac4-4c65-b156-9d61d3cd3299/cache-SQL_PUBLIC_TEMP_TABLE_94_1_5_20190507163914_94/part-439.bin]]
class
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to initialize partition file:
/XXXX/ignite/data/persistence/node00-51465194-9ac4-4c65-b156-9d61d3cd3299/cache-SQL_PUBLIC_TEMP_TABLE_94_1_5_20190507163914_94/part-439.bin
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:466)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:355)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:490)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:474)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:857)
        at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:698)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1657)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1494)
        at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2136)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:457)
        at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2357)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2584)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2041)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1870)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1690)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:317)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:504)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:464)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:266)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1175)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:633)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2517)
        at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2494)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1306)
        at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:835)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual.receive(DataStreamerCacheUpdaters.java:138)
        at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:157)
        at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6836)
        at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:984)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:137)
        at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:522)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:137)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException:
/data01/eippoc/ignite/data/persistence/node00-51465194-9ac4-4c65-b156-9d61d3cd3299/cache-SQL_PUBLIC_TEMP_TABLE_94_1_5_20190507163914_94/part-439.bin:
Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at
sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:196)
        at
java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:248)
        at
java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:301)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:74)
        at
org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:61)
        at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:438)
        ... 32 more


My ignite version is 2.7.I use 'SET STREAMING ON ORDERED ALLOW_OVERWRITE
ON;' to write data to ignite.

Here is the thread dump:






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yangjiajun yangjiajun
Reply | Threaded
Open this post in threaded view
|

Re: [data-streamer-stripe-2-#35][] Critical system error detected.

ezhuravlev ezhuravlev
Reply | Threaded
Open this post in threaded view
|

Re: [data-streamer-stripe-2-#35][] Critical system error detected.