Issue with adding nested index dynamically

classic Classic list List threaded Threaded
47 messages Options
123
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Issue with adding nested index dynamically

We have to add indexes on cache dynamically on java pojo with nested objects.
In the below example we do not have @QuerySqlField. In this case if I try to
add index on "username" dynamically using "CREATE INDEX" it worked. But if I
want to add index on "Address.zipcode" - this is not working as we are NOT
creating aliases for these fields which are getting added dynamically. I see
there is a bug in this implementation. I fixed the bug (please see below)
and rebuilt the jar in my local and it worked fine. Proposing this bug to
this open source community so that it can be fixed in ignite and we can
start using the jar from central repo. We have just started using ignite and
planning to take this to production. So it would be helpful if we can
implement this bug fix.

User{
String userName;
Address address;

}

Address{
String streetName;
String zipcode;
}

Bug Fix: QueryUtils.class
------------------------------
Method name :
------------------
QueryBinaryProperty buildBinaryProperty(GridKernalContext ctx, String
pathStr,
        Class<?> resType, Map<String, String> aliases, @Nullable Boolean
isKeyField, boolean notNull, Object dlftVal,
        int precision, int scale)

--------

Change from :
-----------------
String alias = (String)aliases.get(fullName.toString());

to :
-----
String alias = aliases.get(fullName.toString()) == null ?
fullName.toString() : aliases.get(fullName.toString());

With this way we will always make sure we have default alias (if not
provided), otherwise aliases on dynamic query fields and dynamic indexes are
not working properly





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hi, thanks for starting this thread!

SQL folks, could you please confirm that's neither known nor expected limitation? If that's a real issue then we will just need to go through a standard contribution process.

-
Denis


On Thu, Oct 3, 2019 at 12:19 PM Hemambara <[hidden email]> wrote:
We have to add indexes on cache dynamically on java pojo with nested objects.
In the below example we do not have @QuerySqlField. In this case if I try to
add index on "username" dynamically using "CREATE INDEX" it worked. But if I
want to add index on "Address.zipcode" - this is not working as we are NOT
creating aliases for these fields which are getting added dynamically. I see
there is a bug in this implementation. I fixed the bug (please see below)
and rebuilt the jar in my local and it worked fine. Proposing this bug to
this open source community so that it can be fixed in ignite and we can
start using the jar from central repo. We have just started using ignite and
planning to take this to production. So it would be helpful if we can
implement this bug fix.

User{
String userName;
Address address;

}

Address{
String streetName;
String zipcode;
}

Bug Fix: QueryUtils.class
------------------------------
Method name :
------------------
QueryBinaryProperty buildBinaryProperty(GridKernalContext ctx, String
pathStr,
        Class<?> resType, Map<String, String> aliases, @Nullable Boolean
isKeyField, boolean notNull, Object dlftVal,
        int precision, int scale)

--------

Change from :
-----------------
String alias = (String)aliases.get(fullName.toString());

to :
-----
String alias = aliases.get(fullName.toString()) == null ?
fullName.toString() : aliases.get(fullName.toString());

With this way we will always make sure we have default alias (if not
provided), otherwise aliases on dynamic query fields and dynamic indexes are
not working properly





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

In reply to this post by Hemambara
Hello!

Please find/create a ticket about this issue and attach your patch to this ticket (in plain text or pull request form).

Then it can be tested and merged.

Regards,
--
Ilya Kasnacheev


чт, 3 окт. 2019 г. в 22:19, Hemambara <[hidden email]>:
We have to add indexes on cache dynamically on java pojo with nested objects.
In the below example we do not have @QuerySqlField. In this case if I try to
add index on "username" dynamically using "CREATE INDEX" it worked. But if I
want to add index on "Address.zipcode" - this is not working as we are NOT
creating aliases for these fields which are getting added dynamically. I see
there is a bug in this implementation. I fixed the bug (please see below)
and rebuilt the jar in my local and it worked fine. Proposing this bug to
this open source community so that it can be fixed in ignite and we can
start using the jar from central repo. We have just started using ignite and
planning to take this to production. So it would be helpful if we can
implement this bug fix.

User{
String userName;
Address address;

}

Address{
String streetName;
String zipcode;
}

Bug Fix: QueryUtils.class
------------------------------
Method name :
------------------
QueryBinaryProperty buildBinaryProperty(GridKernalContext ctx, String
pathStr,
        Class<?> resType, Map<String, String> aliases, @Nullable Boolean
isKeyField, boolean notNull, Object dlftVal,
        int precision, int scale)

--------

Change from :
-----------------
String alias = (String)aliases.get(fullName.toString());

to :
-----
String alias = aliases.get(fullName.toString()) == null ?
fullName.toString() : aliases.get(fullName.toString());

With this way we will always make sure we have default alias (if not
provided), otherwise aliases on dynamic query fields and dynamic indexes are
not working properly





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

https://issues.apache.org/jira/browse/IGNITE-12261

I have created JIRA. Please let me know how I can assign to myself. My
username on jira board is "kotari"



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hello!

I have added you to Contributor role. Now you should be able to assign issue to yourself.


Regards,
--
Ilya Kasnacheev


пт, 4 окт. 2019 г. в 20:49, Hemambara <[hidden email]>:
https://issues.apache.org/jira/browse/IGNITE-12261

I have created JIRA. Please let me know how I can assign to myself. My
username on jira board is "kotari"



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

I have made changes and created pull request. Can you please check and let me
know if there are any issues with the commit. Otherwise can you please let
me know when it can it be available in maven repo so that we can start
pulling it in our organization

https://github.com/apache/ignite/pull/6947




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

https://github.com/apache/ignite/pull/6949

Created pull request with suggestion. Can you please check and close



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hello!

Can you add test case to this change?

Otherwise, can you explain when it is useful? Can't zipcode just be annotated with @QuerySqlField in your use case?

Regards,
--
Ilya Kasnacheev


вт, 8 окт. 2019 г. в 13:57, Hemambara <[hidden email]>:
https://github.com/apache/ignite/pull/6949

Created pull request with suggestion. Can you please check and close



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Sure, will add test case.

My domain model is existing domain model and I cannot change or add
@QuerySqlField to my domain model. I am trying to achieve dynamic index
creation on any domain model object (pojo) irrespective @QuerySqlField's



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Added test case and created pull request. Please check both the code changes
under pull request #6949 and test case pull request #6955 and let me know
for any suggestions. Thank you.

IGNITE-12261 - Issue with adding nested index dynamically - Adding tests
#6955



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hello!

Frankly speaking, I don't think this is a scenario that we want to support.

If you are interested, you may like to start a discussion in dev@ list, but: working with nested objects is not a priority, and implicit configuration of a corner case is discouraged.

In this case we recommend you to have custom serialization of some sort (via BinaryTypeConfiguration for example).

Regards,
--
Ilya Kasnacheev


ср, 9 окт. 2019 г. в 06:03, Hemambara <[hidden email]>:
Added test case and created pull request. Please check both the code changes
under pull request #6949 and test case pull request #6955 and let me know
for any suggestions. Thank you.

IGNITE-12261 - Issue with adding nested index dynamically - Adding tests
#6955



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

I have tested with adding QuerySqlFields as well, it is enabling to add
dynamic field and indexes on nested objects but if I restart the client
again it is not able to join the grid because it is not able to find this
column. But with the fix and maintaining proper aliases we are able to add
fields and indexes dynamically on direct/ nested attribute. This
functionality will be out of the box with this fix. Please let me know if
you still see any issue.

Ignite does provide dynamic addition of columns and indexes but it is just
that there is a bug on adding indexes dynamically on QuerySqlFields nested
objects. With this small fix it will run with out any issues.

SEVERE: Failed to reinitialize local partitions (rebalancing will be
stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=3, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7af3d442-946f-4572-9fb7-08971b8d82cc, addrs=[0:0:0:0:0:0:0:1,
114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0,
DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=3,
intOrder=0, lastExchangeTime=1570647484111, loc=true,
ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=3,
nodeId8=7af3d442, msg=null, type=NODE_JOINED, tstamp=1570647484265],
nodeId=7af3d442, evt=NODE_JOINED]
class org.apache.ignite.IgniteCheckedException: Field not found:
Users.userName
        at
org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124)
        at
org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612)
        at
org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

In reply to this post by ilya.kasnacheev
ignite_bug_share.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2625/ignite_bug_share.zip>  

I have tested by adding QuerySqlFields as well, with or with out it, it is
working when I add it, but if I restart the client it is not able to
identify nested columns as its alias is null or not equal to whole path.
Please see below exception. If we can make alias with whole nested object
path we will be able to add indexes and fields dynamically with out any
issues. Ignite, out of the box provides dynamic field and index alterations.
But it has a bug when we add nested field / index dynamically. This small
fix will fix this bug.

PFA for the code.


SEVERE: Failed to reinitialize local partitions (rebalancing will be
stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=5, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=a5379438-868b-43e4-b402-02edf2913795, addrs=[0:0:0:0:0:0:0:1,
114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0,
DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=5,
intOrder=0, lastExchangeTime=1570648315703, loc=true,
ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=5,
nodeId8=a5379438, msg=null, type=NODE_JOINED, tstamp=1570648318843],
nodeId=a5379438, evt=NODE_JOINED]
class org.apache.ignite.IgniteCheckedException: Field not found:
Users.userName
        at
org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124)
        at
org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688)
        at
org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612)
        at
org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172)
        at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927)
        at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
        at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
        at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

In reply to this post by ilya.kasnacheev
ignite_bug_share.zip I have tested by adding QuerySqlFields as well, with or with out it, it is working when I add it, but if I restart the client it is not able to identify nested columns as its alias is null or not equal to whole path. Please see below exception. If we can make alias with whole nested object path we will be able to add indexes and fields dynamically with out any issues. Ignite, out of the box provides dynamic field and index alterations. But it has a bug when we add nested field / index dynamically. This small fix will fix this bug. PFA for the code. SEVERE: Failed to reinitialize local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=a5379438-868b-43e4-b402-02edf2913795, addrs=[0:0:0:0:0:0:0:1, 114.18.146.93, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:0, /127.0.0.1:0, DTC4C4A8C85A622.ent.wfb.bank.corp/114.18.146.93:0], discPort=0, order=5, intOrder=0, lastExchangeTime=1570648315703, loc=true, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], topVer=5, nodeId8=a5379438, msg=null, type=NODE_JOINED, tstamp=1570648318843], nodeId=a5379438, evt=NODE_JOINED] class org.apache.ignite.IgniteCheckedException: Field not found: Users.userName at org.apache.ignite.internal.processors.query.QueryIndexDescriptorImpl.addField(QueryIndexDescriptorImpl.java:124) at org.apache.ignite.internal.processors.query.QueryUtils.createIndexDescriptor(QueryUtils.java:735) at org.apache.ignite.internal.processors.query.QueryUtils.processIndex(QueryUtils.java:752) at org.apache.ignite.internal.processors.query.QueryUtils.processIndexes(QueryUtils.java:688) at org.apache.ignite.internal.processors.query.QueryUtils.processBinaryMeta(QueryUtils.java:612) at org.apache.ignite.internal.processors.query.QueryUtils.typeForQueryEntity(QueryUtils.java:480) at org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:706) at org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1337) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:2172) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:2030) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:927) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:769) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748)

Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

My apologies for multiple replies. Please consider latest reply



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hello!

Currently we do not support altering tables which are created with indexedTypes/queryEntities.

There are plans to implement it, but your fix, while solving your particular problem, isn't fit in this scheme IMHO.

Regards,
--
Ilya Kasnacheev


ср, 9 окт. 2019 г. в 22:45, Hemambara <[hidden email]>:
My apologies for multiple replies. Please consider latest reply



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Sorry for the push. This is a major blocker for us. We have coherence clients
where they can add indexes dynamically and we want to move them to ignite in
next 3 months. Coherence has a way to add indexes dynamically and ignite
also does supporting it. This issue exists event with QuerySqlField. If this
fix cannot be merged, I am not sure how to proceed further. Do you see any
issue putting this fix ? or it is just that we are not ready yet ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Ivan,

As a member of the Ignite SQL group, could you please check this thread and help Hemambara with a proper fix?


-
Denis


On Thu, Oct 10, 2019 at 11:50 AM Hemambara <[hidden email]> wrote:
Sorry for the push. This is a major blocker for us. We have coherence clients
where they can add indexes dynamically and we want to move them to ignite in
next 3 months. Coherence has a way to add indexes dynamically and ignite
also does supporting it. This issue exists event with QuerySqlField. If this
fix cannot be merged, I am not sure how to proceed further. Do you see any
issue putting this fix ? or it is just that we are not ready yet ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Hello Ivan, to be more specific, if I want add field

"Users.userName" - current ignite version 2.7.6 is setting the field name as
"Users.userName" and default alias name as "userName" with which makes it
non-queryable, non -indexable and issues during restart due to mismatch in
configuration . But with this fix for fieldname "Users.userName" it will set
alias as "Users.userName" and the same can be used to add index and
everywhere and there wont be any issues



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hemambara Hemambara
Reply | Threaded
Open this post in threaded view
|

Re: Issue with adding nested index dynamically

Can you please provide any update in this...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
123