Possible starvation in striped pool

classic Classic list List threaded Threaded
14 messages Options
Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Possible starvation in striped pool

We are working on integrating the JEE web application with Apache Ignite and caching around 900 millions records from Oracle database to off heap cache. The client was started from WebLogic using the servlet listener. When users are trying to access the application, the application queries their data from cache. We see the below issue when there are more than 2 users performing the similar operation on their own data. This was not the performance we expected from the documentation.

 

WARN [org.apache.ignite.internal.util.typedef.G] - >>> Possible starvation in striped pool.

    Thread name: sys-stripe-14-#15%AppCluster%

    Queue: []

    Deadlock: false

    Completed: 4

Thread [name="sys-stripe-14-#15% AppCluster%", id=107, state=WAITING, blockCnt=0, waitCnt=6]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5a7be62, ownerName=exchange-worker-#57% AppCluster%, ownerId=171]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:317)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)

        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)

        at java.lang.Thread.run(Thread.java:748)

 

 

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

Unfortunately it's hard to say without looking at full thread dumps from all nodes in cluster. Can you post them somewhere?

Regards,

--
Ilya Kasnacheev

2018-07-12 18:04 GMT+03:00 Gohil, Shailendrasinh (INTL) <[hidden email]>:

We are working on integrating the JEE web application with Apache Ignite and caching around 900 millions records from Oracle database to off heap cache. The client was started from WebLogic using the servlet listener. When users are trying to access the application, the application queries their data from cache. We see the below issue when there are more than 2 users performing the similar operation on their own data. This was not the performance we expected from the documentation.

 

WARN [org.apache.ignite.internal.util.typedef.G] - >>> Possible starvation in striped pool.

    Thread name: sys-stripe-14-#15%AppCluster%

    Queue: []

    Deadlock: false

    Completed: 4

Thread [name="sys-stripe-14-#15% AppCluster%", id=107, state=WAITING, blockCnt=0, waitCnt=6]

    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@5a7be62, ownerName=exchange-worker-#57% AppCluster%, ownerId=171]

        at sun.misc.Unsafe.park(Native Method)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:317)

        at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)

        at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)

        at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)

        at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)

        at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)

        at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)

        at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)

        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)

        at java.lang.Thread.run(Thread.java:748)

 

 


Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Thank you for your response. Please find attached thread dumps for client and
server nodes. ClientThreadDump.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ClientThreadDump.txt>  
ThreadDumpServer1.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer1.txt>  
ThreadDumpServer2.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer2.txt>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

I can see here that you are trying to destroy a cache:

        at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
        at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.close(GatewayProtectedCacheProxy.java:1514)
        at dao.impl.IgniteLookupServiceImpl.getProdCountForEst(IgniteLookupServiceImpl.java:2645)
        at service.impl.RegistrationServiceImpl.getProdCountForEst(RegistrationServiceImpl.java:2588)

Is it supposed to be destroyed in this method? I can see other threads accessing cache from the same Dao.

Regards,

--
Ilya Kasnacheev

2018-07-12 19:35 GMT+03:00 Shailendrasinh Gohil <[hidden email]>:
Thank you for your response. Please find attached thread dumps for client and
server nodes. ClientThreadDump.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ClientThreadDump.txt
ThreadDumpServer1.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer1.txt
ThreadDumpServer2.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ThreadDumpServer2.txt



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Thanks again for the response.

We have tried removing the close statements but the result was same. And
yes, other threads accessing cache from the same Dao.

We also tried both the atomicityMode to see if any improvement. We also have
write behind enabled for the large tables with frequent get and put
operations.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

Can you please provide the thread dump of problematic cluster after removal of close statements on caches?

Regards,

--
Ilya Kasnacheev

2018-07-16 17:21 GMT+03:00 Shailendrasinh Gohil <[hidden email]>:
Thanks again for the response.

We have tried removing the close statements but the result was same. And
yes, other threads accessing cache from the same Dao.

We also tried both the atomicityMode to see if any improvement. We also have
write behind enabled for the large tables with frequent get and put
operations.

Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Please find attached thread dump as requested.

ServerThreadDump0716.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/t1917/ServerThreadDump0716.txt>  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

I have noticed that you are using putAll in your code.

Apache Ignite is susceptible to deadlocks in the same fashion as regular
multi-threaded code: i.e., if you take multiple locks (as putAll does, on
partitions for its keys), you can get deadlock unless you maintain sequence
of locks, i.e., always lock B after A and not the other way around.

You can attain that by passing TreeMap's to putAll as opposed to HashMap's.

Can you try to pass TreeMap's to putAll, see if it fixed your issue?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
sam1708 sam1708
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello same issue occurred when trying to put object in cache using cache.put method.after changing put to putAsync issue was solved.

I have read about when you using putAll methode pass sorted collection to it so it avoid deadlock. So is it true?

On Tue, Jul 17, 2018, 8:22 PM ilya.kasnacheev <[hidden email]> wrote:
Hello!

I have noticed that you are using putAll in your code.

Apache Ignite is susceptible to deadlocks in the same fashion as regular
multi-threaded code: i.e., if you take multiple locks (as putAll does, on
partitions for its keys), you can get deadlock unless you maintain sequence
of locks, i.e., always lock B after A and not the other way around.

You can attain that by passing TreeMap's to putAll as opposed to HashMap's.

Can you try to pass TreeMap's to putAll, see if it fixed your issue?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

In reply to this post by ilya.kasnacheev
We are using the TreeMap for all the putAll operations. We also tried
streamer API to create the automatic batches. Still the issue is same.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

Can you please share the configuration of your Apache Ignite nodes, especially the cache store's of caches. I have just noticed that you're actually waiting on cache store lock.

Regards,

--
Ilya Kasnacheev

2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil <[hidden email]>:
We are using the TreeMap for all the putAll operations. We also tried
streamer API to create the automatic batches. Still the issue is same.

ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello again!

I have just noticed the following stack trace:
"flusher-0-#588%AppCluster%" #633 prio=5 os_prio=0 tid=0x00007f18d424f800 nid=0xe1bb runnable [0x00007f197c1cd000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at oracle.net.ns.Packet.receive(Packet.java:311)
	at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
	at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
	at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
	at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
	at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
	at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
	at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
	at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
	at oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:12029)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12140)
	- locked <0x00007f2aaa591778> (a oracle.jdbc.driver.T4CConnection)
	at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
	at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.executeBatch(CacheAbstractJdbcStore.java:1226)
	at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1111)
	at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:809)
	at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:725)
	at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
	at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1113)
	at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1011)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:748)
It looks like you're waiting for Oracle to complete batch execution, but it won't.

Regards,

--
Ilya Kasnacheev

2018-07-18 17:47 GMT+03:00 Ilya Kasnacheev <[hidden email]>:
Hello!

Can you please share the configuration of your Apache Ignite nodes, especially the cache store's of caches. I have just noticed that you're actually waiting on cache store lock.

Regards,

--
Ilya Kasnacheev

2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil <[hidden email]>:
We are using the TreeMap for all the putAll operations. We also tried
streamer API to create the automatic batches. Still the issue is same.


Shailendrasinh Gohil Shailendrasinh Gohil
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Here you go...

<bean class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="name" value="GeneratedProdIdCache" />
        <property name="cacheMode" value="PARTITIONED" />
        <property name="atomicityMode" value="TRANSACTIONAL" />
        <property name="partitionLossPolicy" value="READ_WRITE_SAFE"/>

        <property name="cacheStoreFactory">
                <bean
                        class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
                        <property name="dataSourceBean" value="appDataSource" />
                        <property name="dialect">
                                <bean class="org.apache.ignite.cache.store.jdbc.dialect.OracleDialect">
                                </bean>
                        </property>

                        <property name="types">
                                <list>
                                        <bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
                                                <property name="cacheName" value="GeneratedProdIdCache" />
                                                <property name="keyType" value="java.lang.String" />
                                                <property name="valueType"
                                                        value="ignite.model.GeneratedProdId" />
                                                <property name="databaseSchema" value="APPSCHEMA" />
                                                <property name="databaseTable" value="GENERATED_PROD_ID" />

                                                <property name="keyFields">
                                                        <list>
                                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodId" />
                                                                </bean>
                                                        </list>
                                                </property>

                                                <property name="valueFields">
                                                        <list>
                                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodId" />
                                                                </bean>
                                                               
                                                                 <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID_KEY" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodIdKey" />
                                                                </bean>        
                                                        </list>
                                                </property>
                                        </bean>
                                </list>
                        </property>
                </bean>
        </property>

        <property name="readThrough" value="true" />
        <property name="writeThrough" value="true" />
        <property name="writeBehindEnabled" value="true" />
        <property name="writeBehindFlushSize" value="2048" />
        <property name="writeBehindFlushFrequency" value="0" />
        <property name="writeBehindFlushThreadCount" value="1" />
        <property name="writeBehindBatchSize" value="512" />
       
        <property name="queryEntities">
                <list>
                        <bean class="org.apache.ignite.cache.QueryEntity">
                                <property name="keyType" value="java.lang.String" />
                                <property name="valueType"
                                        value="ignite.model.GeneratedProdId" />
                                <property name="tableName" value="GENERATED_PROD_ID" />
                                <property name="keyFieldName" value="prodId" />

                                <property name="keyFields">
                                        <list>
                                                <value>prodId</value>
                                        </list>
                                </property>

                                <property name="fields">
                                        <map>
                                                <entry key="prodId" value="java.lang.String" />
                                                <entry key="prodIdKey" value="java.lang.String" />
                                        </map>
                                </property>

                                <property name="aliases">
                                        <map>
                                                <entry key="prodId" value="PROD_ID" />
                                                <entry key="prodIdKey" value="PROD_ID_KEY" />
                                        </map>
                                </property>
                               
                                <property name="indexes">
                                        <list>
                                                <bean class="org.apache.ignite.cache.QueryIndex">
                                                        <property name="name" value="INDX_GNRTD_PROD_ID" />
                                                        <property name="indexType" value="FULLTEXT" />

                                                        <property name="fields">
                                                                <map>
                                                                        <entry key="prodId" value="true" />
                                                                </map>
                                                        </property>
                                                </bean>
                                        </list>
                                </property>
                        </bean>
                </list>
        </property>
</bean>



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Possible starvation in striped pool

Hello!

At this point I recommend debugging which statements are ran on Oracle and why they take long.

Also I have noticed: appDataSource - is it behind some kind of connection pool? I am afraid it is possible that this data source is single-threaded in the absense of connection pool, hence you might see contention.

Regards,

--
Ilya Kasnacheev

2018-07-18 20:48 GMT+03:00 Shailendrasinh Gohil <[hidden email]>:
Here you go...

<bean class="org.apache.ignite.configuration.CacheConfiguration">
        <property name="name" value="GeneratedProdIdCache" />
        <property name="cacheMode" value="PARTITIONED" />
        <property name="atomicityMode" value="TRANSACTIONAL" />
        <property name="partitionLossPolicy" value="READ_WRITE_SAFE"/>

        <property name="cacheStoreFactory">
                <bean
                        class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
                        <property name="dataSourceBean" value="appDataSource" />
                        <property name="dialect">
                                <bean class="org.apache.ignite.cache.store.jdbc.dialect.OracleDialect">
                                </bean>
                        </property>

                        <property name="types">
                                <list>
                                        <bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
                                                <property name="cacheName" value="GeneratedProdIdCache" />
                                                <property name="keyType" value="java.lang.String" />
                                                <property name="valueType"
                                                        value="ignite.model.GeneratedProdId" />
                                                <property name="databaseSchema" value="APPSCHEMA" />
                                                <property name="databaseTable" value="GENERATED_PROD_ID" />

                                                <property name="keyFields">
                                                        <list>
                                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodId" />
                                                                </bean>
                                                        </list>
                                                </property>

                                                <property name="valueFields">
                                                        <list>
                                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodId" />
                                                                </bean>

                                                                 <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                                        <constructor-arg>
                                                                                <util:constant static-field="java.sql.Types.VARCHAR" />
                                                                        </constructor-arg>
                                                                        <constructor-arg value="PROD_ID_KEY" />
                                                                        <constructor-arg value="java.lang.String" />
                                                                        <constructor-arg value="prodIdKey" />
                                                                </bean>         
                                                        </list>
                                                </property>
                                        </bean>
                                </list>
                        </property>
                </bean>
        </property>

        <property name="readThrough" value="true" />
        <property name="writeThrough" value="true" />
        <property name="writeBehindEnabled" value="true" />
        <property name="writeBehindFlushSize" value="2048" />
        <property name="writeBehindFlushFrequency" value="0" />
        <property name="writeBehindFlushThreadCount" value="1" />
        <property name="writeBehindBatchSize" value="512" />

        <property name="queryEntities">
                <list>
                        <bean class="org.apache.ignite.cache.QueryEntity">
                                <property name="keyType" value="java.lang.String" />
                                <property name="valueType"
                                        value="ignite.model.GeneratedProdId" />
                                <property name="tableName" value="GENERATED_PROD_ID" />
                                <property name="keyFieldName" value="prodId" />

                                <property name="keyFields">
                                        <list>
                                                <value>prodId</value>
                                        </list>
                                </property>

                                <property name="fields">
                                        <map>
                                                <entry key="prodId" value="java.lang.String" />
                                                <entry key="prodIdKey" value="java.lang.String" />
                                        </map>
                                </property>

                                <property name="aliases">
                                        <map>
                                                <entry key="prodId" value="PROD_ID" />
                                                <entry key="prodIdKey" value="PROD_ID_KEY" />
                                        </map>
                                </property>

                                <property name="indexes">
                                        <list>
                                                <bean class="org.apache.ignite.cache.QueryIndex">
                                                        <property name="name" value="INDX_GNRTD_PROD_ID" />
                                                        <property name="indexType" value="FULLTEXT" />

                                                        <property name="fields">
                                                                <map>
                                                                        <entry key="prodId" value="true" />
                                                                </map>
                                                        </property>
                                                </bean>
                                        </list>
                                </property>
                        </bean>
                </list>
        </property>
</bean>