Performance of join SQL queries

classic Classic list List threaded Threaded
32 messages Options
12
alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Performance of join SQL queries

We are doing performance test that include two Ignite caches (one with 1000000 records and second with 3000000 records). The tested SQL query has join of these two caches and can include several criteria for different fields from both caches. Part of queries with one or two criteria works pretty fast but when we have three or more criteria for both caches the execution time too much high.

Can you help us understand how execution plan of query looks like and why performance goes so bad.  
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

alex.glau wrote
Can you help us understand how execution plan of query looks like and why performance goes so bad.
Is there any chance you can provide your SQL query here?
alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

I prepared test with smaller number of data:
I have two Ignite caches: "ShipmentCache" including 200000 records and "ShipmentSegmentCache" including 600500 records. These two caches (tables) are related by "shipmentId" field.
We run this test on single Ignite node. The first query (see countJoinQuery() method) that has to criteria takes ~900msec but second one (see countJoinQuery_2()) that has one additional criteria took ~2900000msec !!!. So we want to understand execution plan. Is it possible to request execution plan as it supported by H2 database. Please see relevant code below.

//  Set Cache for shipments
CacheConfiguration<?, ?> shipmentCacheConfig = new CacheConfiguration<>();
shipmentCacheConfig.setName(SHIPMENT_CACHE);
shipmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentCacheConfig.setIndexedTypes(UUID.class, CachedShipment.class);
//  Set Cache for shipment segments
CacheConfiguration<?, ?> shipmentSegmentCacheConfig = new CacheConfiguration<>();
shipmentSegmentCacheConfig.setName(SHIPMENT_SEGMENT_CACHE);
shipmentSegmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentSegmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentSegmentCacheConfig.setIndexedTypes(AffinityKey.class, CachedShipmentSegment.class);

// Ignite configuration
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setCacheConfiguration(shipmentCacheConfig, shipmentSegmentCacheConfig);

Ignite ignite = Ignition.start(igniteConfig);

private void countJoinQuery() throws Exception {
        IgniteCache<AffinityKey<UUID>, CachedShipment> shipmentSegmentCache =   ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE + "\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "segment.startSiteName = ? and shipment.status = ?";
        QueryCursor<List<?>> results = shipmentSegmentCache.query(new SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        System.out.println("\nSuccessful get count(*) : " + results.getAll().get(0).get(0));
}

private void countJoinQuery_2() throws Exception {
        IgniteCache<AffinityKey<UUID>, CachedShipment> shipmentSegmentCache = ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE + "\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "shipment.status = ? and segment.startSiteName = ? and shipment.carrierType = ?";

        // Execute queries for find employees for different organizations.
        QueryCursor<List<?>> results = shipmentSegmentCache.query(new SqlFieldsQuery(joinSql).setArgs(ShipmentStatus.CLOSED, "Beta", CarrierType.AVIA));
        System.out.println("\nSuccessful get count(*): " + results.getAll().get(0).get(0));
}

public class CachedShipment implements Externalizable {
        @QuerySqlField(index = true)
        private UUID id;
        @QuerySqlField(index = true)
        private String name;
        @QuerySqlField
        private String description;
        @QuerySqlField(index=true)
        private String startSiteName;
        @QuerySqlField(index=true)
        private Date startDate;
        @QuerySqlField(index = true)
        private ShipmentStatus status;
        @QuerySqlField(index = true)
        private String destOrgName;
        @QuerySqlField(index = true)
        private CarrierType carrierType;
       
// constructor & getter/setter...
}

public class CachedShipmentSegment implements Externalizable {

        private UUID id;
        @QuerySqlField(index = true)
        private UUID shipmentId;
        @QuerySqlField(index = true)
        private String startSiteName;
        @QuerySqlField(index = true)
        private String endSiteName;

    public AffinityKey<UUID> key() {
        if (key == null)
            key = new AffinityKey<>(id, shipmentId);
        return key;
    }

// constructor & getter/setter...
}



Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Hi!

Ignite does support "EXPLAIN ..." syntax in SQL queries and reading execution plans is a main way to analyze query performance in Ignite. Note that plan cursor will contain multiple rows: the first one will contain query for reducing node, others are for map nodes.

I think in your case SQL engine chooses wrong index in a second query.
The problem on Ignite side is that we don't collect selectivity for each index (all the single field indexes look equal to query optimizer).
The problem on your side is that you've created many single field indexes instead of creating a smaller number of group indexes (see @QuerySqlField.Group).

For example for both queries it will be enough to have one group index on CachedShipment (status, carrierType) instead of having two separate indexes on (status) and (carrierType). Or as I see you already have startSiteName in CachedShipment, if it has the same meaning as in Segment, then you should add it to this group index as well like (startSiteName, status, carrierType) and use CachedShipment.startSiteName instead of Segment.startSiteName in these queries.

Note that if you have an index on (startSiteName, status, carrierType) query engine will be able to use it if query will contain conditions like:
1. startSiteName = ?
2. startSiteName = ? and status = ?
3. startSiteName = ? and status = ? and carrierType = ?
4. startSiteName = ? and carrierType = ? /** this one is Ok as well, but for the index it looks no better than the first one. **/

If the query contains condition like `status = ? and carrierType = ?` but does not contain condition `startSiteName = ?` you need to have a separate index, so choose order of fields in index wisely.

Sergi


2015-08-09 14:13 GMT+03:00 alex.glau <[hidden email]>:
I prepared test with smaller number of data:
I have two Ignite caches: "ShipmentCache" including 200000 records and
"ShipmentSegmentCache" including 600500 records. These two caches (tables)
are related by "shipmentId" field.
We run this test on single Ignite node. The first query (see
countJoinQuery() method) that has to criteria takes ~900msec but second one
(see countJoinQuery_2()) that has one additional criteria took ~2900000msec
!!!. So we want to understand execution plan. Is it possible to request
execution plan as it supported by H2 database. Please see relevant code
below.

//  Set Cache for shipments
CacheConfiguration<?, ?> shipmentCacheConfig = new CacheConfiguration<>();
shipmentCacheConfig.setName(SHIPMENT_CACHE);
shipmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentCacheConfig.setIndexedTypes(UUID.class, CachedShipment.class);
//  Set Cache for shipment segments
CacheConfiguration<?, ?> shipmentSegmentCacheConfig = new
CacheConfiguration<>();
shipmentSegmentCacheConfig.setName(SHIPMENT_SEGMENT_CACHE);
shipmentSegmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentSegmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentSegmentCacheConfig.setIndexedTypes(AffinityKey.class,
CachedShipmentSegment.class);

// Ignite configuration
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setCacheConfiguration(shipmentCacheConfig,
shipmentSegmentCacheConfig);

Ignite ignite = Ignition.start(igniteConfig);

private void countJoinQuery() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "segment.startSiteName = ? and shipment.status = ?";
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        System.out.println("\nSuccessful get count(*) : " +
results.getAll().get(0).get(0));
}

private void countJoinQuery_2() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "shipment.status = ? and segment.startSiteName = ? and
shipment.carrierType = ?";

        // Execute queries for find employees for different organizations.
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs(ShipmentStatus.CLOSED, "Beta",
CarrierType.AVIA));
        System.out.println("\nSuccessful get count(*): " +
results.getAll().get(0).get(0));
}

public class CachedShipment implements Externalizable {
        @QuerySqlField(index = true)
        private UUID id;
        @QuerySqlField(index = true)
        private String name;
        @QuerySqlField
        private String description;
        @QuerySqlField(index=true)
        private String startSiteName;
        @QuerySqlField(index=true)
        private Date startDate;
        @QuerySqlField(index = true)
        private ShipmentStatus status;
        @QuerySqlField(index = true)
        private String destOrgName;
        @QuerySqlField(index = true)
        private CarrierType carrierType;

//      constructor & getter/setter...
}

public class CachedShipmentSegment implements Externalizable {

        private UUID id;
        @QuerySqlField(index = true)
        private UUID shipmentId;
        @QuerySqlField(index = true)
        private String startSiteName;
        @QuerySqlField(index = true)
        private String endSiteName;

    public AffinityKey<UUID> key() {
        if (key == null)
            key = new AffinityKey<>(id, shipmentId);
        return key;
    }

//      constructor & getter/setter...
}







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p870.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Alexey Kuznetsov Alexey Kuznetsov
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Sergi, I think your answer (with appropriate changes) should be added to "Performance and Usability Considerations" in documentation https://apacheignite.readme.io/docs/cache-queries
Users should know about EXPLAIN and group indexes from documentation.

On Sun, Aug 9, 2015 at 11:08 PM, Sergi Vladykin <[hidden email]> wrote:
Hi!

Ignite does support "EXPLAIN ..." syntax in SQL queries and reading execution plans is a main way to analyze query performance in Ignite. Note that plan cursor will contain multiple rows: the first one will contain query for reducing node, others are for map nodes.

I think in your case SQL engine chooses wrong index in a second query.
The problem on Ignite side is that we don't collect selectivity for each index (all the single field indexes look equal to query optimizer).
The problem on your side is that you've created many single field indexes instead of creating a smaller number of group indexes (see @QuerySqlField.Group).

For example for both queries it will be enough to have one group index on CachedShipment (status, carrierType) instead of having two separate indexes on (status) and (carrierType). Or as I see you already have startSiteName in CachedShipment, if it has the same meaning as in Segment, then you should add it to this group index as well like (startSiteName, status, carrierType) and use CachedShipment.startSiteName instead of Segment.startSiteName in these queries.

Note that if you have an index on (startSiteName, status, carrierType) query engine will be able to use it if query will contain conditions like:
1. startSiteName = ?
2. startSiteName = ? and status = ?
3. startSiteName = ? and status = ? and carrierType = ?
4. startSiteName = ? and carrierType = ? /** this one is Ok as well, but for the index it looks no better than the first one. **/

If the query contains condition like `status = ? and carrierType = ?` but does not contain condition `startSiteName = ?` you need to have a separate index, so choose order of fields in index wisely.

Sergi


2015-08-09 14:13 GMT+03:00 alex.glau <[hidden email]>:
I prepared test with smaller number of data:
I have two Ignite caches: "ShipmentCache" including 200000 records and
"ShipmentSegmentCache" including 600500 records. These two caches (tables)
are related by "shipmentId" field.
We run this test on single Ignite node. The first query (see
countJoinQuery() method) that has to criteria takes ~900msec but second one
(see countJoinQuery_2()) that has one additional criteria took ~2900000msec
!!!. So we want to understand execution plan. Is it possible to request
execution plan as it supported by H2 database. Please see relevant code
below.

//  Set Cache for shipments
CacheConfiguration<?, ?> shipmentCacheConfig = new CacheConfiguration<>();
shipmentCacheConfig.setName(SHIPMENT_CACHE);
shipmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentCacheConfig.setIndexedTypes(UUID.class, CachedShipment.class);
//  Set Cache for shipment segments
CacheConfiguration<?, ?> shipmentSegmentCacheConfig = new
CacheConfiguration<>();
shipmentSegmentCacheConfig.setName(SHIPMENT_SEGMENT_CACHE);
shipmentSegmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentSegmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentSegmentCacheConfig.setIndexedTypes(AffinityKey.class,
CachedShipmentSegment.class);

// Ignite configuration
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setCacheConfiguration(shipmentCacheConfig,
shipmentSegmentCacheConfig);

Ignite ignite = Ignition.start(igniteConfig);

private void countJoinQuery() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "segment.startSiteName = ? and shipment.status = ?";
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        System.out.println("\nSuccessful get count(*) : " +
results.getAll().get(0).get(0));
}

private void countJoinQuery_2() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "shipment.status = ? and segment.startSiteName = ? and
shipment.carrierType = ?";

        // Execute queries for find employees for different organizations.
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs(ShipmentStatus.CLOSED, "Beta",
CarrierType.AVIA));
        System.out.println("\nSuccessful get count(*): " +
results.getAll().get(0).get(0));
}

public class CachedShipment implements Externalizable {
        @QuerySqlField(index = true)
        private UUID id;
        @QuerySqlField(index = true)
        private String name;
        @QuerySqlField
        private String description;
        @QuerySqlField(index=true)
        private String startSiteName;
        @QuerySqlField(index=true)
        private Date startDate;
        @QuerySqlField(index = true)
        private ShipmentStatus status;
        @QuerySqlField(index = true)
        private String destOrgName;
        @QuerySqlField(index = true)
        private CarrierType carrierType;

//      constructor & getter/setter...
}

public class CachedShipmentSegment implements Externalizable {

        private UUID id;
        @QuerySqlField(index = true)
        private UUID shipmentId;
        @QuerySqlField(index = true)
        private String startSiteName;
        @QuerySqlField(index = true)
        private String endSiteName;

    public AffinityKey<UUID> key() {
        if (key == null)
            key = new AffinityKey<>(id, shipmentId);
        return key;
    }

//      constructor & getter/setter...
}







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p870.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.




--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com
alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

In reply to this post by Sergi Vladykin
Hi Sergi,
Thanks for clear explanation.

1) Can you either post syntax of EXPLAIN query or give me a reference to appropriate documentation
2) If we have index on field representing Date (or Float or Integer), does it help in queries with criteria like date > AnyFixDate (or number > AnyFixNumber)?
3) If answer on the previous question is Yes and we prepare compound index that should include Date (or Float or Integer) field, have this field be located in the last position inside the indexing group? If so what about the case where we have more than one such field?

Regards,
Alex.
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Alex,

1) Just prepend your query with EXPLAIN and run it as usual.
For example: EXPLAIN SELECT count(*) FROM t WHERE x = ?

2) Yes, indexes are sorted. It means that they will be used for range conditions.
Basically condition `x = 10` is equivalent to `x between 10 and 10`.

3) It is not a requirement that a field participating in condition like `>` must be
in the end of indexing group, but this way query may work faster, because
previous conditions are bounded on both sides and the last one will be only
half-bounded. If you will have a half-bounded field in the middle of
indexing group then you'll make all the fields going after in indexing group
half-bounded as well even if conditions on them are fully bounded.
So definitely it is preferable.

On the other hand if first fields in the indexing group already have a good selectivity,
then you will not win much by putting this half-bounded field in the end of indexing group,
you can put it it in the middle as well, or even not index it at all.
Just try and look what works for your data and your queries better.

Sergi


2015-08-10 9:12 GMT+03:00 alex.glau <[hidden email]>:
Hi Sergi,
Thanks for clear explanation.

1) Can you either post syntax of EXPLAIN query or give me a reference to
appropriate documentation
2) If we have index on field representing Date (or Float or Integer), does
it help in queries with criteria like date > AnyFixDate (or number >
AnyFixNumber)?
3) If answer on the previous question is Yes and we prepare compound index
that should include Date (or Float or Integer) field, have this field be
located in the last position inside the indexing group? If so what about the
case where we have more than one such field?

Regards,
Alex.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p878.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

In reply to this post by Alexey Kuznetsov
Alexey,

Sure, we should document this better.

Sergi

2015-08-09 19:22 GMT+03:00 Alexey Kuznetsov <[hidden email]>:
Sergi, I think your answer (with appropriate changes) should be added to "Performance and Usability Considerations" in documentation https://apacheignite.readme.io/docs/cache-queries
Users should know about EXPLAIN and group indexes from documentation.

On Sun, Aug 9, 2015 at 11:08 PM, Sergi Vladykin <[hidden email]> wrote:
Hi!

Ignite does support "EXPLAIN ..." syntax in SQL queries and reading execution plans is a main way to analyze query performance in Ignite. Note that plan cursor will contain multiple rows: the first one will contain query for reducing node, others are for map nodes.

I think in your case SQL engine chooses wrong index in a second query.
The problem on Ignite side is that we don't collect selectivity for each index (all the single field indexes look equal to query optimizer).
The problem on your side is that you've created many single field indexes instead of creating a smaller number of group indexes (see @QuerySqlField.Group).

For example for both queries it will be enough to have one group index on CachedShipment (status, carrierType) instead of having two separate indexes on (status) and (carrierType). Or as I see you already have startSiteName in CachedShipment, if it has the same meaning as in Segment, then you should add it to this group index as well like (startSiteName, status, carrierType) and use CachedShipment.startSiteName instead of Segment.startSiteName in these queries.

Note that if you have an index on (startSiteName, status, carrierType) query engine will be able to use it if query will contain conditions like:
1. startSiteName = ?
2. startSiteName = ? and status = ?
3. startSiteName = ? and status = ? and carrierType = ?
4. startSiteName = ? and carrierType = ? /** this one is Ok as well, but for the index it looks no better than the first one. **/

If the query contains condition like `status = ? and carrierType = ?` but does not contain condition `startSiteName = ?` you need to have a separate index, so choose order of fields in index wisely.

Sergi


2015-08-09 14:13 GMT+03:00 alex.glau <[hidden email]>:
I prepared test with smaller number of data:
I have two Ignite caches: "ShipmentCache" including 200000 records and
"ShipmentSegmentCache" including 600500 records. These two caches (tables)
are related by "shipmentId" field.
We run this test on single Ignite node. The first query (see
countJoinQuery() method) that has to criteria takes ~900msec but second one
(see countJoinQuery_2()) that has one additional criteria took ~2900000msec
!!!. So we want to understand execution plan. Is it possible to request
execution plan as it supported by H2 database. Please see relevant code
below.

//  Set Cache for shipments
CacheConfiguration<?, ?> shipmentCacheConfig = new CacheConfiguration<>();
shipmentCacheConfig.setName(SHIPMENT_CACHE);
shipmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentCacheConfig.setIndexedTypes(UUID.class, CachedShipment.class);
//  Set Cache for shipment segments
CacheConfiguration<?, ?> shipmentSegmentCacheConfig = new
CacheConfiguration<>();
shipmentSegmentCacheConfig.setName(SHIPMENT_SEGMENT_CACHE);
shipmentSegmentCacheConfig.setCacheMode(CacheMode.PARTITIONED);
shipmentSegmentCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
shipmentSegmentCacheConfig.setIndexedTypes(AffinityKey.class,
CachedShipmentSegment.class);

// Ignite configuration
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setCacheConfiguration(shipmentCacheConfig,
shipmentSegmentCacheConfig);

Ignite ignite = Ignition.start(igniteConfig);

private void countJoinQuery() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "segment.startSiteName = ? and shipment.status = ?";
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        System.out.println("\nSuccessful get count(*) : " +
results.getAll().get(0).get(0));
}

private void countJoinQuery_2() throws Exception {
        IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
        String joinSql =
                "select count(*) " +
                "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                "where segment.shipmentId = shipment.id and " +
                "shipment.status = ? and segment.startSiteName = ? and
shipment.carrierType = ?";

        // Execute queries for find employees for different organizations.
        QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs(ShipmentStatus.CLOSED, "Beta",
CarrierType.AVIA));
        System.out.println("\nSuccessful get count(*): " +
results.getAll().get(0).get(0));
}

public class CachedShipment implements Externalizable {
        @QuerySqlField(index = true)
        private UUID id;
        @QuerySqlField(index = true)
        private String name;
        @QuerySqlField
        private String description;
        @QuerySqlField(index=true)
        private String startSiteName;
        @QuerySqlField(index=true)
        private Date startDate;
        @QuerySqlField(index = true)
        private ShipmentStatus status;
        @QuerySqlField(index = true)
        private String destOrgName;
        @QuerySqlField(index = true)
        private CarrierType carrierType;

//      constructor & getter/setter...
}

public class CachedShipmentSegment implements Externalizable {

        private UUID id;
        @QuerySqlField(index = true)
        private UUID shipmentId;
        @QuerySqlField(index = true)
        private String startSiteName;
        @QuerySqlField(index = true)
        private String endSiteName;

    public AffinityKey<UUID> key() {
        if (key == null)
            key = new AffinityKey<>(id, shipmentId);
        return key;
    }

//      constructor & getter/setter...
}







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p870.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.




--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

In reply to this post by Sergi Vladykin
Sergi,

I have already guessed (correctly, as I see from your reply) what is the syntax of EXPLAIN and try it (see my example below). Unfortunately, I got the following exception:

javax.cache.CacheException: java.lang.ClassCastException: org.h2.command.dml.Explain cannot be cast to org.h2.command.dml.Select
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:457)
        at com.bt9.x2s.tasks.ShipmentSearchTest.joinFieldQuery_Explain(ShipmentSearchTest.java:381)
        at com.bt9.x2s.tasks.ShipmentSearchTest.test(ShipmentSearchTest.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:142)
        at mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:71)
        at mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:40)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
        at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.ClassCastException: org.h2.command.dml.Explain cannot be cast to org.h2.command.dml.Select
        at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parse(GridSqlQueryParser.java:182)
        at org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:67)
        at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:789)
        at org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:515)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:448)
        ... 31 more

Example Code:

        private void joinFieldQuery_Explain() {
                IgniteCache<AffinityKey<UUID>, CachedShipment> shipmentSegmentCache = ignite.cache(SHIPMENT_SEGMENT_CACHE);
                String joinSql =
                        "explain select shipment.name, shipment.status, segment.startSiteName " +
                        "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE + "\".CachedShipment as shipment " +
                        "where segment.shipmentId = shipment.id and " +
                        "segment.startSiteName = ? and shipment.status = ?";

                // Execute queries for find employees for different organizations.
                QueryCursor<List<?>> results = shipmentSegmentCache.query(new SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        }
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Alex,

I believe you are using outdated version of Ignite. Could you try with the latest 1.3.0?

Sergi

2015-08-10 11:21 GMT+03:00 alex.glau <[hidden email]>:
Sergi,

I have already guessed (correctly, as I see from your reply) what is the
syntax of EXPLAIN and try it (see my example below). Unfortunately, I got
the following exception:

javax.cache.CacheException: java.lang.ClassCastException:
org.h2.command.dml.Explain cannot be cast to org.h2.command.dml.Select
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:457)
        at
com.bt9.x2s.tasks.ShipmentSearchTest.joinFieldQuery_Explain(ShipmentSearchTest.java:381)
        at com.bt9.x2s.tasks.ShipmentSearchTest.test(ShipmentSearchTest.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:142)
        at
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:71)
        at
mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:40)
        at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
        at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
        at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
        at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
        at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
        at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.ClassCastException: org.h2.command.dml.Explain cannot
be cast to org.h2.command.dml.Select
        at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQueryParser.parse(GridSqlQueryParser.java:182)
        at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:67)
        at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:789)
        at
org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:515)
        at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:448)
        ... 31 more

Example Code:

        private void joinFieldQuery_Explain() {
                IgniteCache<AffinityKey&lt;UUID>, CachedShipment> shipmentSegmentCache =
ignite.cache(SHIPMENT_SEGMENT_CACHE);
                String joinSql =
                        "explain select shipment.name, shipment.status, segment.startSiteName " +
                        "from CachedShipmentSegment as segment, \"" + SHIPMENT_CACHE +
"\".CachedShipment as shipment " +
                        "where segment.shipmentId = shipment.id and " +
                        "segment.startSiteName = ? and shipment.status = ?";

                // Execute queries for find employees for different organizations.
                QueryCursor<List&lt;?>> results = shipmentSegmentCache.query(new
SqlFieldsQuery(joinSql).setArgs("Beta", ShipmentStatus.CLOSED));
        }



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p882.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Sergi, you are right: with new version of Ignite it waked good.
I am afraid to be too bothering, but I need a little bit more information about reducing node and map node in order to understand two statements obtained from EXPLAIN query.
Please point me to relevant documentation.

Thank you. I'm really appreciate you quick and useful help.
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Alex,

This stuff is not quite yet documented, (I'm going to fix this soon) so I'll answer here.

Basically Ignite SQL works the following way:

It receives SQL query parses it into AST, then splits this query in two parts:
map and reduce. Map query runs on each participating cache node, reduce
query aggregates results received from all nodes running map query.
In reduce query you will see function __Z0() which is basically
a function that returns results of map queries from all the nodes.

Currently you will see only two statements in explain, but this is going to change soon
and Ignite will be able to cut the original query into more parts. Respectively
in reduce query you will see multiple artificial tables __T0,__T1 ... __TN
depending on query structure.

Sergi

P.S. Your questions are quite relevant, I think I will be writing docs looking at this thread,
so feel free to ask more, I will be happy to help!



2015-08-10 12:09 GMT+03:00 alex.glau <[hidden email]>:
Sergi, you are right: with new version of Ignite it waked good.
I am afraid to be too bothering, but I need a little bit more information
about reducing node and map node in order to understand two statements
obtained from EXPLAIN query.
Please point me to relevant documentation.

Thank you. I'm really appreciate you quick and useful help.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p884.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Hi Sergi,

I am trying to test grouping index for SQL queries.
My query is pretty simple (without join), it has two conditions where two fields participated in these conditions are grouped in one grouping index. For this case I got the following execution plan where __SCAN_ (I suppose) means scanning through the table and not see any relation to the grouping index. Execution time of this case was 510 ms.

[SELECT
    ID AS __C0,
    NAME AS __C1
FROM "ShipmentCache".CACHEDSHIPMENT
    /* "ShipmentCache".CACHEDSHIPMENT.__SCAN_ */
WHERE (STATUS = ?1)
    AND (STARTDATE > ?2)]

After I changes the grouping index to two individual indexes I got the following plan and execution time was 283 ms:

[SELECT
    ID AS __C0,
    NAME AS __C1
FROM "ShipmentCache".CACHEDSHIPMENT
    /* "ShipmentCache"."status_idx": STATUS = ?1 */
WHERE (STATUS = ?1)
    AND (STARTDATE > ?2)]

What is wrong in my understanding?
Please see below the used code:

IgniteCache<UUID, CachedShipment> shipmentCache = ignite.cache(SHIPMENT_CACHE);
String queryStmt =
        "select id, name " +
        "from CachedShipment " +
        "where status = ? and startDate > ?";

Date startDate = DateTime.parse("2015-07-01").toDate();

// Execution plan
String explainStmt = "explain analyze " + queryStmt;
QueryCursor<List<?>> planResults = shipmentCache.query(new SqlFieldsQuery(explainStmt).setArgs(ShipmentStatus.CLOSED, startDate));
System.out.println("Execution Plan:");
for (List<?> entry : planResults) {
        System.out.println("\n" + entry);
}

// Retrieve data
QueryCursor<List<?>> results = shipmentCache.query(new SqlFieldsQuery(queryStmt).setArgs(ShipmentStatus.CLOSED, startDate));

Fields Definition

public class CachedShipment implements Externalizable {
        @QuerySqlField(index = true)
        private UUID id;
        @QuerySqlField(index = true)
        private String name;
        @QuerySqlField
        private String description;
        @QuerySqlField(index=true)
        private String startSiteName;

// @QuerySqlField(index=true)
    @QuerySqlField.Group(name = "idx2", order = 1)
        private Date startDate;
       
// @QuerySqlField(index=true)
    @QuerySqlField.Group(name = "idx2", order = 0)
        private ShipmentStatus status;

        @QuerySqlField(index = true)
        private String destOrgName;
        @QuerySqlField(index = true)
        private CarrierType carrierType;





alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

In reply to this post by Sergi Vladykin
Small correction.
In the previous test I forgot to put annotation @QueryGroupIndex.List.
So now the fields definition looks like followed but execution plan still is with execution time 382 ms (vs. 283 ms in case of individual indexes)

[SELECT
    ID AS __C0,
    NAME AS __C1
FROM "ShipmentCache".CACHEDSHIPMENT
    /* "ShipmentCache".CACHEDSHIPMENT.__SCAN_ */
WHERE (STATUS = ?1)
    AND (STARTDATE > ?2)]

Fields definition

@QueryGroupIndex.List(
        @QueryGroupIndex(name="idx2")
)
public class CachedShipment implements Externalizable {

        @QuerySqlField(index = true)
        private UUID id;
       
        @QuerySqlField(index = true)
        private String name;
       
        @QuerySqlField
        private String description;
       
        @QuerySqlField(index=true)
        private String startSiteName;
       
        @QuerySqlField
    @QuerySqlField.Group(name = "idx2", order = 1)
        private Date startDate;
       
        @QuerySqlField
    @QuerySqlField.Group(name = "idx2", order = 0)
        private ShipmentStatus status;
       
        @QuerySqlField(index = true)
        private String destOrgName;
       
        @QuerySqlField(index = true)
        private CarrierType carrierType;

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

In reply to this post by Sergi Vladykin
Hi Sergi, how are you.

Did you see my two previous questions related to execution plan?
Should I wait with patience?

Regards,
Alex.
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Alex,

Sorry for the late reply, I'm not always online and have some other tasks than looking at my mailbox:)

Your setup looks correct to me. BTW I think @QueryGroupIndex annotation must be optional here.

I suggest to do 2 things to identify the problem:
1. Set an env variable IGNITE_H2_DEBUG_CONSOLE to true and start locally a single node. H2 console will open in your browser and you will be able to see your tables and indexes at SQL layer. Check that your index "idx2" has been created and it contatins correct fields with correct order.
2. If the index is ok, then you can run the same query with different conditions like `STATUS = ? AND STARTDATE = ?` or `STATUS = ?` and see if the index will be used in the query plan.

Sergi





2015-08-12 13:53 GMT+03:00 alex.glau <[hidden email]>:
Hi Sergi, how are you.

Did you see my two previous questions related to execution plan?
Should I wait with patience?

Regards,
Alex.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p922.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Hi Sergi,

This H2 Console is good thing.
Now I see that problem is that grouping index includes only the build-in primary key (_KEY) but not my fields. I have twice verified my code vs. your GroupIndexExample.
What may be the reason that created index is wrong?

Regards,
Alex.
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Alex,

Where did you see that GroupIndexExample?
Actually it should work like this:

@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp1", order = 2)})

Agree that it is a usability hell, we should allow using @QuerySqlField.Group outside of @QuerySqlField or at least throw exception with informative message. Created issue https://issues.apache.org/jira/browse/IGNITE-1247

Initially it was done this way to allow multiple annotations of type @Group on an element.
Java 8 already supports multiple annotations of the same type, probably we should gradually deprecate and drop current approach.

Sergi

2015-08-13 11:02 GMT+03:00 alex.glau <[hidden email]>:
Hi Sergi,

This H2 Console is good thing.
Now I see that problem is that grouping index includes only the build-in
primary key (_KEY) but not my fields. I have twice verified my code vs. your
GroupIndexExample.
What may be the reason that created index is wrong?

Regards,
Alex.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p944.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

alex.glau alex.glau
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

Hi Sergi,

1) Group index example is in:
https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/query/GroupIndexExample.java

2) Can you add the syntax of group index to documentation?

3) Is ordering of group index started from 0 or from 1?

4) If some field has individual index and in addition is participated in several group indexes, the set of annotations should look like:

@QuerySqlField(index=true)
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp1", order = 1)})
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp2", order = 2)})

or

@QuerySqlField(index=true)
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp1", order = 1), @QuerySqlField.Group(name = "grp2", order = 2)})

which variant is right?

Thanks,
Alex.
Sergi Vladykin Sergi Vladykin
Reply | Threaded
Open this post in threaded view
|

Re: Performance of join SQL queries

1) Ok, sorry for that, will be fixed soon.
https://issues.apache.org/jira/browse/IGNITE-1249

2) No problem, will do.

3) You can start with -100500 and finish with Integer.MAX_VALUE as well, these numbers only needed for fields sorting inside of a group, they don't have any other meaning.

4)  It must be this way @QuerySqlField(index=true, orderedGroups = {@QuerySqlField.Group(name = "grp1", order =
1), @QuerySqlField.Group(name = "grp2", order = 2)})

I think you are using Java 8, currently Ignite is targeted to Java 7 where it is impossible to use the same annotation twice for the same field.

Sergi


2015-08-14 11:26 GMT+03:00 alex.glau <[hidden email]>:
Hi Sergi,

1) Group index example is in:
https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/query/GroupIndexExample.java

2) Can you add the syntax of group index to documentation?

3) Is ordering of group index started from 0 or from 1?

4) If some field has individual index and in addition is participated in
several group indexes, the set of annotations should look like:

@QuerySqlField(index=true)
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp1", order =
1)})
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp2", order =
2)})

or

@QuerySqlField(index=true)
@QuerySqlField(orderedGroups = {@QuerySqlField.Group(name = "grp1", order =
1), @QuerySqlField.Group(name = "grp2", order = 2)})

which variant is right?

Thanks,
Alex.




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-join-SQL-queries-tp845p977.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

12