Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

classic Classic list List threaded Threaded
8 messages Options
bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Hi, we have used this query in ignite 2.3 for a while now. But we had some
data streamer exceptions that seemed to have been resolved on 2.4, so we
decided to upgrade. However, in 2.6 and also downgrading to 2.4 we have been
seeing this issue, that a SQL query does not return the data that is in
cache. when we go back to 2.3, it works as expected.

Here is the cache config for that cache.


        <bean id="tradeOrderCache"
class="org.apache.ignite.configuration.CacheConfiguration">
                 
                <property name="name" value="TradeOrder" />
                <property name="readThrough" value="True" />
                <property name="cacheMode" value="PARTITIONED" />
                <property name="atomicityMode" value="TRANSACTIONAL" />
                <property name="backups" value="1" />
                <property name="queryDetailMetricsSize" value="512" />
                <property name="cacheStoreFactory">
                        <bean class="javax.cache.configuration.FactoryBuilder$SingletonFactory">
                                <constructor-arg>
                                        <ref bean="tradeOrderCacheStore" />
                                </constructor-arg>
                        </bean>
                </property>
                <property name="writeSynchronizationMode" value="FULL_SYNC" />
                <property name="statisticsEnabled" value="true" />
                <property name="queryEntities">
                        <list>
                                <bean id="tradeOrderQueryEntity"
class="org.apache.ignite.cache.QueryEntity">
                                        <property name="keyType"
                                                value="com.tudor.datagridI.client.data.trading.OrderKey" />
                                        <property name="valueType"
value="com.tudor.datagridI.client.data.trading.TradeOrder" />

                                        <property name="fields">
                                                <map>
                                                        <entry key="traderId" value="java.lang.Integer" />
                                                        <entry key="orderId" value="java.lang.Integer" />
                                                        <entry key="insIid" value="java.lang.Integer" />
                                                        <entry key="settlement" value="java.util.Date" />
                                                        <entry key="clearAgent" value="java.lang.String" />
                                                        <entry key="strategy" value="java.lang.String" />
                                                        <entry key="pvId" value="java.lang.Integer" />
                                                        <entry key="pvDate" value="java.util.Date" />
                                                        <entry key="linkId" value="java.lang.Integer" />
                                                        <entry key="parentId" value="java.lang.Integer" />
                                                </map>
                                        </property>
                                        <property name="indexes">
                                                <list>
                                                        <bean class="org.apache.ignite.cache.QueryIndex">
                                                                <constructor-arg>
                                                                        <list>
                                                                                <value>traderId</value>
                                                                                <value>orderId</value>
                                                                        </list>
                                                                </constructor-arg>
                                                                <constructor-arg>
                                                                        <value>SORTED</value>
                                                                </constructor-arg>
                                                                <property name="name" value="tradeOrder_key_index" />
                                                        </bean>
                                                        <bean class="org.apache.ignite.cache.QueryIndex">
                                                                <constructor-arg>
                                                                        <list>
                                                                                <value>traderId</value>
                                                                                <value>insIid</value>
                                                                                <value>clearAgent</value>
                                                                                <value>strategy</value>
                                                                        </list>
                                                                </constructor-arg>
                                                                <constructor-arg>
                                                                        <value>SORTED</value>
                                                                </constructor-arg>
                                                                <property name="name" value="loadTradeOrders_index" />
                                                        </bean>
                                                          <bean class="org.apache.ignite.cache.QueryIndex">
                                                                <constructor-arg>
                                                                        <list>
                                                                                <value>parentId</value>
                                                                        </list>
                                                                </constructor-arg>
                                                                <constructor-arg>
                                                                        <value>SORTED</value>
                                                                </constructor-arg>
                                                                <property name="name" value="parentId_index" />
                                                        </bean>
                                                </list>
                                        </property>
                                </bean>
                        </list>
                </property>
        </bean>

and here is the query

        public List<TradeOrder> getTradeOrdersForPSGroup(Integer traderId, Short
psRuleId, Integer tid, String clearAgent, String strategy, Integer pvId,
Date settlementDate, Date psTime) {
                logger.info(String.format("Getting TradeOrders from the cache for
traderId: %s, tid: %s, clearAgent: %s, strategy: %s, pvId: %s, settlement:
%s, psTime: %s", traderId, tid, clearAgent, strategy, pvId, settlementDate,
psTime));
                List<TradeOrder> tradeOrders = new ArrayList<TradeOrder>();
                String sqlQuery = "select * from \"TradeOrder\".TradeOrder where traderId
= ? and insIid = ? and clearAgent = ? and strategy = ? and isnull(pvId,0) =
?";
                SqlQuery<OrderKey, TradeOrder> sql = new SqlQuery<OrderKey,
TradeOrder>(TradeOrder.class, sqlQuery).setArgs(traderId, tid, clearAgent,
strategy, pvId);

                if(settlementDate != null) {
                        sqlQuery = sqlQuery + " and isnull(pvDate, settlement) = ?";
                        sql = new SqlQuery<OrderKey, TradeOrder>(TradeOrder.class,
sqlQuery).setArgs(traderId, tid, clearAgent, strategy, pvId,
settlementDate);
                }
        sql.setTimeout(10, TimeUnit.SECONDS);
                SortedSet<OrderKey> keys = new TreeSet<OrderKey>();
                try (QueryCursor<Entry&lt;OrderKey, TradeOrder>> cursor =
tradeOrderCache.query(sql)) {
                        for (Entry<OrderKey, TradeOrder> e : cursor) {
                                boolean addThisTo = false;
                                TradeOrder to = e.getValue();
                                logger.info("Query read this tradeOrder: " + to.getOrderKey());
                                for(Trade t: to.getTrades()) {
                                        if(t.getPsTime().compareTo(psTime) >= 0) {
                                                addThisTo = true;
                                                break;
                                        }
                                }
                                if (addThisTo) {
                                        //exclude open orders
                                        if ("open".equals(to.getStatus().trim()))
                                                addThisTo = false;
                                        String repoFlag = to.getRepo();
                                        if (repoFlag == null)
                                                repoFlag = "N";
                                        repoFlag = repoFlag.trim();
                                        //open and term repos should also be excluded
                                        if (repoFlag.equals("T") || repoFlag.equals("O"))
                                                addThisTo = false;
                                        if (to.getPsRuleId() != psRuleId)
                                                addThisTo = false;
                                }
                                if(addThisTo) {
                                        logger.info("Adding tradeOrder to psGroup returnList:" +
to.getOrderKey());
                                        keys.add(e.getKey());
                                        tradeOrders.add(to);
                                }
                        }
                }

                /*
                 * getting all keys to acquire locks on them in a transaction,
                 * if we do not do this, then another transaction can update these objects
leading cache in an inconsistent state
                 */
                tradeOrderCache.getAll(keys);
                return tradeOrders;
        }

we do a getAll(keys) because we are running this in a transaction and want
to acquire locks. however, I have tried this outside of a transaction as
well. The data is never returned by the SQL.

Do you have any ideas what could be causing this?

Let us know if you need anything else. The code not throw exceptions.

Thanks,
Binti







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
akurbanov akurbanov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Hi,

Do you use persistence? If so, please check and set up  Baseline topology
<https://apacheignite.readme.io/docs/baseline-topology>   for your cluster
which describes a set of server nodes that are used to store data. The
reason for this behaviour may be a simple fact that some of your nodes are
out of baseline topology.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
bintisepaha bintisepaha
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Thanks for getting back, but we do not use Ignite's native persistence.
Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Hi,

There were a lot of changes in the product since 2.3 which may affect it. Most important change was baseline topology, as already mentioned.
I am aware of a case when incorrect result might be returned [1], which is already fixed in master. Not sure if this is the same issue, but you may try to build Ignite from recent master and check if the problem is still there.

Is it possible to create isolated reproducer for this issue?


On Wed, Aug 15, 2018 at 11:34 PM bintisepaha <[hidden email]> wrote:
Thanks for getting back, but we do not use Ignite's native persistence.
Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

I also want to point out that Ignite has nightly builds, so you can try them instead of doing your own build as well.


D.

On Thu, Aug 16, 2018 at 1:38 AM, Vladimir Ozerov <[hidden email]> wrote:
Hi,

There were a lot of changes in the product since 2.3 which may affect it. Most important change was baseline topology, as already mentioned.
I am aware of a case when incorrect result might be returned [1], which is already fixed in master. Not sure if this is the same issue, but you may try to build Ignite from recent master and check if the problem is still there.

Is it possible to create isolated reproducer for this issue?


On Wed, Aug 15, 2018 at 11:34 PM bintisepaha <[hidden email]> wrote:
Thanks for getting back, but we do not use Ignite's native persistence.
Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Vladimir Ozerov Vladimir Ozerov
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Dmitriy,

I wanted to suggest this, but looks like we do not have direct links for regular distribution. User can download it from TeamCity, but username/password is needed for this. May be I missing something.

On Thu, Aug 16, 2018 at 11:44 AM Dmitriy Setrakyan <[hidden email]> wrote:
I also want to point out that Ignite has nightly builds, so you can try them instead of doing your own build as well.


D.

On Thu, Aug 16, 2018 at 1:38 AM, Vladimir Ozerov <[hidden email]> wrote:
Hi,

There were a lot of changes in the product since 2.3 which may affect it. Most important change was baseline topology, as already mentioned.
I am aware of a case when incorrect result might be returned [1], which is already fixed in master. Not sure if this is the same issue, but you may try to build Ignite from recent master and check if the problem is still there.

Is it possible to create isolated reproducer for this issue?


On Wed, Aug 15, 2018 at 11:34 PM bintisepaha <[hidden email]> wrote:
Thanks for getting back, but we do not use Ignite's native persistence.
Anything else changed from 2.3 to 2.4 to cause this around SQL Queries?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

yfernando yfernando
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

Thanks Vlad / Dmitry, I was able to download the nightly build from teamcity
as a guest.

However I'm now getting the following error when trying to add data to the
cache. Any ideas?

class org.apache.ignite.IgniteException: Failed to get affinity mapping from
node: TcpDiscoveryNode [id=eeef1e32-db85-4425-8718-29884d9af6d0,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.32.50.59, 127.0.0.1],
sockAddrs=HashSet [/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
appl183-stc.global.tudor.com/10.32.50.59:47500], discPort=47500, order=3,
intOrder=3, lastExchangeTime=1534414697561, loc=false,
ver=2.7.0#19691231-sha1:00000000, isClient=false]
        at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:990)
        at
org.apache.ignite.internal.IgniteComputeImpl.affinityRunAsync0(IgniteComputeImpl.java:186)
        at
org.apache.ignite.internal.IgniteComputeImpl.affinityRun(IgniteComputeImpl.java:145)
        at
com.tudor.datagridI.client.TradeOrderStoreHelper.processOrderHolders(TradeOrderStoreHelper.java:30)
        at
com.tudor.datagridI.TradingDataAccessImpl.saveOrders(TradingDataAccessImpl.java:294)
        at
orderserver.client.GridClient.updatePreparedOrderHoldersInGrid(GridClient.java:180)
        at
orderserver.client.GridClient.updateOrderHoldersInGrid(GridClient.java:135)
        at orderserver.OrderFactory.saveOrders(OrderFactory.java:6705)
        at orderserver.OrderFactory.saveOrders(OrderFactory.java:6620)
        at
com.tudor.test.performancetest.PerformanceTester$CurrencyBooker.run(PerformanceTester.java:286)
        at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to get
affinity mapping from node: TcpDiscoveryNode
[id=eeef1e32-db85-4425-8718-29884d9af6d0, addrs=ArrayList
[0:0:0:0:0:0:0:1%lo, 10.32.50.59, 127.0.0.1], sockAddrs=HashSet
[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500,
appl183-stc.global.tudor.com/10.32.50.59:47500], discPort=47500, order=3,
intOrder=3, lastExchangeTime=1534414697561, loc=false,
ver=2.7.0#19691231-sha1:00000000, isClient=false]
        at
org.apache.ignite.internal.processors.affinity.GridAffinityProcessor.affinityCache(GridAffinityProcessor.java:515)
        at
org.apache.ignite.internal.processors.affinity.GridAffinityProcessor.affinityKey(GridAffinityProcessor.java:342)
        at
org.apache.ignite.internal.IgniteComputeImpl.affinityRunAsync0(IgniteComputeImpl.java:176)
        ... 12 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
unmarshal object with optimized marshaller
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10025)
        at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:830)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1077)
        at
org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1312)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
        ... 3 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
unmarshal object with optimized marshaller
        at
org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1765)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10019)
        ... 10 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
deserialize object with given class loader:
[clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, err=Failed to deserialize
object [typeName=org.apache.ignite.internal.util.lang.GridTuple3]]
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:237)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
        at
org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1762)
        ... 16 more
Caused by: java.io.IOException: Failed to deserialize object
[typeName=org.apache.ignite.internal.util.lang.GridTuple3]
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:228)
        ... 18 more
Caused by: java.io.IOException: Failed to deserialize object
[typeName=org.apache.ignite.internal.processors.affinity.GridAffinityAssignment]
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.internal.util.lang.GridTuple3.readExternal(GridTuple3.java:197)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
        ... 21 more
Caused by: java.io.IOException: Failed to deserialize field
[name=assignment]
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:526)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:611)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
        ... 27 more
Caused by: java.io.IOException: Failed to deserialize object
[typeName=org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode]
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readArrayList(OptimizedObjectInputStream.java:643)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:301)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:519)
        ... 30 more
Caused by: java.io.IOException: Unexpected error occurred during
unmarshalling of an instance of the class:
org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot. Check that
all nodes are running the same version of Ignite and that all nodes have
GridOptimizedMarshaller configured with identical optimized classes lists,
if any (see setClassNames and setClassNamesPath methods). If your serialized
classes implement java.io.Externalizable interface, verify that
serialization logic is correct.
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:364)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416)
        at
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:616)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917)
        at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
        ... 41 more



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
yfernando yfernando
Reply | Threaded
Open this post in threaded view
|

Re: Ignite SQL Queries not getting all data back in ignite 2.4 and 2.6

The above error appears to be an issue in
org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot in a
mismatch between the readExternal() and writeExternal() methods.

I've made a change locally and it fixed the error and also the testing so
far seems to show that the queries return the correct results (so it's
likely that the cause may have been
https://issues.apache.org/jira/browse/IGNITE-8900)

If you would like me to contribute the fix back to the ignite codebase
please let me know and i'll be happy to do it



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/