Scheduling Cache Refresh using Ignite

classic Classic list List threaded Threaded
8 messages Options
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Scheduling Cache Refresh using Ignite

Hi

We are doing a  a POC on exploring the Ignite in memory capabilities and
building a rest api on
top of it using node express.
       
       
Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB
to Ignite Cache using Cache JDBC Pojo Store.
       
Can someone help me whether the following scenarios can be handled using
Ignite as i couldn't find this in the official documentation.
               
                1. If we want to add/drop/modify a  column to the cache, can we update the
bean file directly
                   when the node is running or do we need to stop the node and then again
restart.
                   It would be really helpful if you can  share sample code or
documentation link.
                   
                2. How to refresh the ignite cache automatically or schedule the cache
refresh.
                   It would be really helpful if you can  share sample code or
documentation link.

                3. Is incremental refresh allowed? It would be really helpful if you can
share sample code or
                   documentation link.
                   
               
                4. Is there any other way to load the caches fast other Cache JDBC POJO
Store.
                   It would be really helpful if you can  share sample code or
documentation link.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

Hi,

Please read my comments:

1)Ignite generally doesn't support changing of the cache configuration
without re-creation of the the cache. But for SQL caches that were
created via QueryEntity or CREATE TABLE you can add and remove the
columns using ALTER TABLE commands:

https://apacheignite-sql.readme.io/docs/alter-table
https://apacheignite.readme.io/docs/cache-queries#query-configuration-using-queryentity
https://apacheignite-sql.readme.io/docs/create-table
2)First of all, you can use the following options:

https://apacheignite.readme.io/docs/3rd-party-store#section-read-through-and-write-through

Read through can load the requested keys from DB
Write through will load all the updates to DB.

In case if you require some cache invalidation or refresh then you can
create some cron job for it.

3)I guess that loadCache is the only to do it. It will filter the values
that have already existed in the cache.

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#loadCache-org.apache.ignite.lang.IgniteBiPredicate-java.lang.Object...-

4)You can use a different subset of integrations that can do distributed
streaming to Ignite like Spark or Kafka:

https://apacheignite-mix.readme.io/docs/getting-started

BR,
Andrei
2/12/2020 9:11 PM, nithin91 пишет:

> Hi
>
> We are doing a  a POC on exploring the Ignite in memory capabilities and
> building a rest api on
> top of it using node express.
>
>
> Currently as a part of POC, installed Ignite in UNIX and trying to load the
> data from Oracle DB
> to Ignite Cache using Cache JDBC Pojo Store.
>
> Can someone help me whether the following scenarios can be handled using
> Ignite as i couldn't find this in the official documentation.
>
> 1. If we want to add/drop/modify a  column to the cache, can we update the
> bean file directly
>   when the node is running or do we need to stop the node and then again
> restart.
>   It would be really helpful if you can  share sample code or
> documentation link.
>
> 2. How to refresh the ignite cache automatically or schedule the cache
> refresh.
>   It would be really helpful if you can  share sample code or
> documentation link.
>
> 3. Is incremental refresh allowed? It would be really helpful if you can
> share sample code or
>   documentation link.
>
>
> 4. Is there any other way to load the caches fast other Cache JDBC POJO
> Store.
>   It would be really helpful if you can  share sample code or
> documentation link.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

Thanks aealexsandrov. This information is very useful.

Also i have one more query.

Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB to Ignite Cache using Cache JDBC Pojo Store.
       
        As a part of this process, bean file is custom configured  to start
ignite
node in unix. Attached the bean file.
        This bean file consists of both cache configuration details and
ignite
        configuration details.
       
        Once the node is running, we are trying to do the following
       
        1. Connecting to the ignite node running on unix through eclipse by
creating a replica of
the attached bean file from local system and adding an additional property
in Bean file with
Client Mode = true and
           then loading the cache that are defined in the bean file deployed
in
unix using the
           following method from local system using JAVA
       
                    ignite.cache("CacheName").loadCache(null);
                               
               * We are able to do this successfully.*
                               
        2.  Connecting to the ignite node running on unix by creating a
replica of
the attached bean file
                in local system and adding an additional property in Bean
file with Client
Mode = true
                and then trying to create a cache and configure the cache
and then finally
loading
                the cache using the attached JAVA Code.
               
               
               * When we are trying this approach, we are getting an error
like dynamic
cache change
                is not allowed.Not getting this error when Ignite server
node and client node  is running on local machine.Getting this error when
server node is running in unix and trying to connect to this node from local
system.*
               
                It would be really helpful if you can help me in resolving
this issue.
               
                If this not the right approach, then
                Configuring all the caches in the bean file is the only
available
option?If this is case,
                What should be the approach for  building some additional
caches in ignite
and load these Caches using Cache JDBC POJO Store when the node is running.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

This post was updated on .
Following is the format of bean file used

<?xml version="1.0" encoding="UTF-8"?>
        <beans xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:util="http://www.springframework.org/schema/util"
        xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">

       

        <bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource">
                <property name="URL" value="..." />
                <property name="user" value="..." />
                <property name="password" value="..." />
        </bean>
        <bean id="ignite.cfg"
                class="org.apache.ignite.configuration.IgniteConfiguration">
               
                <property name="peerClassLoadingEnabled" value="true" />
               
                <property name="discoverySpi">
                        <bean
                                class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                                <property name="ipFinder">
                                       
                                       
                                       
                                        <bean
                                                class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                                                <property name="addresses">
                                                        <list>
                                                               
                                                               
                                                                ....

                                                        </list>
                                                </property>
                                        </bean>
                                </property>
                        </bean>
                </property>
                <property name="cacheConfiguration">
                        <list>
                                <bean
                                        class="org.apache.ignite.configuration.CacheConfiguration">
                                        <property name="name" value="PieProductNAVCache" />
                                        <property name="cacheMode" value="PARTITIONED" />
                                        <property name="atomicityMode" value="ATOMIC" />
                                        <property name="readThrough" value="true" />
                                       

                                        <property name="cacheStoreFactory">
                                                <bean
                                                        class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
                                                        <property name="dataSourceBean" value="dataSource" />
                                                        <property name="dialect">
                                                                <bean
                                                                        class="org.apache.ignite.cache.store.jdbc.dialect.OracleDialect">
                                                                </bean>
                                                        </property>

                                                        <property name="types">
                                                                <list>
                                                                        <bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
                                                                                <property name="cacheName" value="PieProductNAVCache" />
                                                                                <property name="keyType"
                                                                                        value="ignite.example.IgniteUnixImplementation.PieProductNAVKey" />
                                                                                <property name="valueType"
                                                                                        value="ignite.example.IgniteUnixImplementation.PieProductNAV" />
                                                                                <property name="databaseSchema" value="..." />
                                                                                <property name="databaseTable"
                                                                                        value="..." />

                                                                                <property name="keyFields">
                                                                                        <list>
                                                                                                ....
                                                                                                ....
                                                                                               
                                                                                        </list>
                                                                                </property>
                                                                                <property name="valueFields">
                                                                                        <list>
                                                                                               
                                                                                                ....
                                                                                                ....

                                                                                        </list>
                                                                                </property>
                                                                        </bean>
                                                                </list>
                                                        </property>
                                                </bean>
                                        </property>

                                        <property name="queryEntities">
                                                <list>
                                                        <bean class="org.apache.ignite.cache.QueryEntity">
                                                                <property name="keyType"
                                                                        value="ignite.example.IgniteUnixImplementation.PieProductNAVKey" />
                                                                <property name="valueType"
                                                                        value="ignite.example.IgniteUnixImplementation.PieProductNAV" />

                                                                <property name="fields">
                                                                        <map>

                                                                                ...
                                                                                ...
                                                                               
                                                                        </map>
                                                                </property>

                                                                <property name="aliases">
                                                                        <map>
                                                                                ...
                                                                                ...
                                                                               
                                                                        </map>
                                                                </property>
                                                        </bean>
                                                </list>
                                        </property>
                                </bean>
                        </list>
                </property>
        </bean>
</beans>


--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

In reply to this post by nithin91
Following is the java code that loads the cache.

                package Load;

import java.sql.Types;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory;
import org.apache.ignite.cache.store.jdbc.JdbcType;
import org.apache.ignite.cache.store.jdbc.JdbcTypeField;
import org.apache.ignite.cache.store.jdbc.dialect.OracleDialect;
import org.apache.ignite.configuration.CacheConfiguration;
import ignite.example.IgniteUnixImplementation.OrderDetails;
import ignite.example.IgniteUnixImplementation.OrderKey;

public class OrdersLoad {

        private static final class CacheJdbcPojoStoreExampleFactory extends
CacheJdbcPojoStoreFactory<OrderKey, OrderDetails> {
        /**
         *
         */
        private static final long serialVersionUID = 1L;

                /** {@inheritDoc} */
        @Override public CacheJdbcPojoStore<OrderKey, OrderDetails> create()
{
           
        setDataSourceBean("dataSource");
            return super.create();
        }
    }


private static CacheConfiguration<OrderKey, OrderDetails>
cacheConfiguration() {
    CacheConfiguration<OrderKey, OrderDetails> cfg = new
CacheConfiguration<>("OrdersCache");
   
    CacheJdbcPojoStoreExampleFactory storefactory =new
CacheJdbcPojoStoreExampleFactory();
   
    storefactory.setDialect(new OracleDialect());
   
    storefactory.setDataSourceBean("dataSource");

    JdbcType jdbcType = new JdbcType();

    jdbcType.setCacheName("OrdersCache");
    jdbcType.setDatabaseSchema("PDS_CACHE");
    jdbcType.setDatabaseTable("ORDERS2");

    jdbcType.setKeyType("ignite.example.IgniteUnixImplementation.OrderKey");
    jdbcType.setKeyFields(new JdbcTypeField(Types.INTEGER, "ORDERID",
Long.class, "OrderID"),
    new JdbcTypeField(Types.INTEGER, "CITYID", Long.class, "CityID")
   
   
    );

   
jdbcType.setValueType("ignite.example.IgniteUnixImplementation.OrderDetails");
    jdbcType.setValueFields(
        new JdbcTypeField(Types.VARCHAR, "PRODUCTNAME", String.class,
"Productname"),
        new JdbcTypeField(Types.VARCHAR, "CUSTOMERNAME", String.class,
"CustomerName"),
        new JdbcTypeField(Types.VARCHAR, "STORENAME", String.class,
"StoreName")
    );

    storefactory.setTypes(jdbcType);

    cfg.setCacheStoreFactory(storefactory);
   
    cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);

    cfg.setReadThrough(true);
    cfg.setWriteThrough(true);
    cfg.setSqlSchema("PIE");

    return cfg;
}

public static void main(String[] args) throws Exception {
    try (Ignite ignite = Ignition.start("Ignite-Client.xml")) {
   
        System.out.println(">>> Loading cache OrderDetails");
       
        IgniteCache<OrderKey, OrderDetails> cache =
ignite.getOrCreateCache(cacheConfiguration());
       
        cache.clear();
       
        ignite.cache("OrdersCache").loadCache(null);
       
        System.out.println(">>> Loaded cache: OrdersCache
Size="+cache.size());
       
    }
}
}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

In reply to this post by nithin91
Hi,

Can you please attach the full logs with the mentioned exception? BTW I
don't see any attaches in the previous message (probably user list can't
do it).

BR,
Andrei

2/13/2020 3:44 PM, nithin91 пишет:
> Attached the bean file used
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
nithin91 nithin91
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

Hi

I am unable to attach any file as a result of which i pasted the code and
bean file in my previous messages.

Following is error i get.

Feb 13, 2020 11:34:40 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage

 

[sndNodeId=null, id=b9bb52d3071-613fd9b8-0c00-4dde-ba8f-8f5341734a3c,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Feb 13, 2020 11:34:47 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to reconnect to cluster (consider increasing 'networkTimeout'
configuration property) [networkTimeout=5000]
[11:34:52] Ignite node stopped OK [uptime=00:00:24.772]
Exception in thread "main" javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Failed to execute
dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3023)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2992)
        at Load.OrdersLoad.main(OrdersLoad.java:82)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Failed
to execute dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:952)
        at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:948)
        ... 4 more
Caused by: class
org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Failed
to execute dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:1180)
        at
org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3949)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:821)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
aealexsandrov aealexsandrov
Reply | Threaded
Open this post in threaded view
|

Re: Scheduling Cache Refresh using Ignite

Hi Nithin,

You face current message because your client lost the connection to the server. It tries to get the acknowledge message on some operation (I guess it should be some cache operation).

You can see that IgniteClientDisconnectedException was thrown. In this case, you can get the reconnect future and wait for the client reconnection:

https://apacheignite.readme.io/docs/clients-vs-servers#reconnecting-a-client

Please add try/catch blocks around your cache operation and add next logic:

    catch (IgniteClientDisconnectedException e) {
        e.reconnectFuture().get(); // Wait for reconnect.

        // Can proceed and use the same IgniteCompute instance.
    }

I can't say why your client was disconnected. Highly likely it's because of some network issues. You can try to take a look at server logs and find there NODE_LEFT or NODE_FAILED messages.

BR,
Andrei

2/14/2020 8:08 AM, nithin91 пишет:
Hi 

I am unable to attach any file as a result of which i pasted the code and
bean file in my previous messages.

Following is error i get.

Feb 13, 2020 11:34:40 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage

 

[sndNodeId=null, id=b9bb52d3071-613fd9b8-0c00-4dde-ba8f-8f5341734a3c,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
        at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Feb 13, 2020 11:34:47 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to reconnect to cluster (consider increasing 'networkTimeout'
configuration property) [networkTimeout=5000]
[11:34:52] Ignite node stopped OK [uptime=00:00:24.772]
Exception in thread "main" javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Failed to execute
dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3023)
        at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2992)
        at Load.OrdersLoad.main(OrdersLoad.java:82)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Failed
to execute dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:952)
        at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:948)
        ... 4 more
Caused by: class
org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Failed
to execute dynamic cache change request, client node disconnected.
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:1180)
        at
org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3949)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:821)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
        at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/