Stupide question about Queue ?

classic Classic list List threaded Threaded
12 messages Options
yann.blazart@externe.bnpparibas.com yann.blazart@externe.bnpparibas.com
Reply | Threaded
Open this post in threaded view
|

Stupide question about Queue ?

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Denis Magda Denis Magda
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis

On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.


vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Agree with Denis. Sounds like this can be easily achieved by enabling swap layer for queue cache. So we just need to put this on configuration.

-Val

On Thu, Jan 14, 2016 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis


On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.



ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

What are the possibilities for swap spaces? Can we persist the cache/queue content to file system based on the eviction policy?

Thanks,
Murthy.

On Thu, Jan 14, 2016 at 5:23 PM, Valentin Kulichenko <[hidden email]> wrote:
Agree with Denis. Sounds like this can be easily achieved by enabling swap layer for queue cache. So we just need to put this on configuration.

-Val

On Thu, Jan 14, 2016 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis


On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.




vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Created ticket for queue swapping feature: https://issues.apache.org/jira/browse/IGNITE-2389

Feel free to provide feedback.

-Val

On Thu, Jan 14, 2016 at 3:24 PM, Murthy Kakarlamudi <[hidden email]> wrote:
What are the possibilities for swap spaces? Can we persist the cache/queue content to file system based on the eviction policy?

Thanks,
Murthy.

On Thu, Jan 14, 2016 at 5:23 PM, Valentin Kulichenko <[hidden email]> wrote:
Agree with Denis. Sounds like this can be easily achieved by enabling swap layer for queue cache. So we just need to put this on configuration.

-Val

On Thu, Jan 14, 2016 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis


On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.





vkulichenko vkulichenko
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Hi Murthy,

Swap space allows to write evicted data to disc to avoid out of memory. This is not persistence, because this data will be lost if the node leaves topology. For caches persistence can be implemented with the help of CacheStore interface [1]. For queues and sets it's not possible now, I will start a separate thread on dev@ about this.


-Val

On Thu, Jan 14, 2016 at 11:16 PM, Valentin Kulichenko <[hidden email]> wrote:
Created ticket for queue swapping feature: https://issues.apache.org/jira/browse/IGNITE-2389

Feel free to provide feedback.

-Val

On Thu, Jan 14, 2016 at 3:24 PM, Murthy Kakarlamudi <[hidden email]> wrote:
What are the possibilities for swap spaces? Can we persist the cache/queue content to file system based on the eviction policy?

Thanks,
Murthy.

On Thu, Jan 14, 2016 at 5:23 PM, Valentin Kulichenko <[hidden email]> wrote:
Agree with Denis. Sounds like this can be easily achieved by enabling swap layer for queue cache. So we just need to put this on configuration.

-Val

On Thu, Jan 14, 2016 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis


On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.






ksatya ksatya
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Thanks for the explanation.

Murthy.

On Fri, Jan 15, 2016 at 2:27 AM, Valentin Kulichenko <[hidden email]> wrote:
Hi Murthy,

Swap space allows to write evicted data to disc to avoid out of memory. This is not persistence, because this data will be lost if the node leaves topology. For caches persistence can be implemented with the help of CacheStore interface [1]. For queues and sets it's not possible now, I will start a separate thread on dev@ about this.


-Val

On Thu, Jan 14, 2016 at 11:16 PM, Valentin Kulichenko <[hidden email]> wrote:
Created ticket for queue swapping feature: https://issues.apache.org/jira/browse/IGNITE-2389

Feel free to provide feedback.

-Val

On Thu, Jan 14, 2016 at 3:24 PM, Murthy Kakarlamudi <[hidden email]> wrote:
What are the possibilities for swap spaces? Can we persist the cache/queue content to file system based on the eviction policy?

Thanks,
Murthy.

On Thu, Jan 14, 2016 at 5:23 PM, Valentin Kulichenko <[hidden email]> wrote:
Agree with Denis. Sounds like this can be easily achieved by enabling swap layer for queue cache. So we just need to put this on configuration.

-Val

On Thu, Jan 14, 2016 at 5:44 AM, Denis Magda <[hidden email]> wrote:
Hi,

This seems to be an interesting functionality. As I understand you're referring to [1].

Personally I think that it makes sense to support the same in Ignite.
However I would just extend CollectionConfiguration adding ability to set an eviction policy and enable swap on demand. When eviction policy faces with a limit entries will be evicted to swap space.
Don't think that we need to support other kind of storages here.

Igniters, do you have other thoughts on this?

[1] http://docs.hazelcast.org/docs/3.5/manual/html/queue-persistence.html

--
Denis


On 1/14/2016 1:41 PM, Yann BLAZART wrote:

Hello,

 

In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception.

 

But I don’t see this possibility in Ignite docs ?

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.







dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

In reply to this post by yann.blazart@externe.bnpparibas.com
Yann BLAZART wrote
In hazelcast we can persist Queues in Database, this  help me for example when consumer are down to no have OOM exception. But I don't see this possibility in Ignite docs ?
Hazelcast queues are not distributed. They can only be stored on one Hazelcast node and, therefore, can only utilize memory of that node only.

In Ignite, queues can be colocated or distributed. In distributed mode, queue elements will be spread across all the participating nodes, hence, running out of memory is much less likely. This also means that you can always add more nodes to increase the total memory available to your queue.

D.
Stolidedog Stolidedog
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

So if I understand correctly you cannot store queue info in a DB currently with Ignite V1.5?

We are a software vendor and want to build services that communicate between each other.  The thing to note, is this is on-prem software and we don't have anyway to know ahead of time how much nodes we can deploy.  Probably in most cases we have two - four machines, so spinning up more nodes adding more memory isn't "usually" a luxury afforded to us.  It is simply out of our control.  

The ability to off load storage in a distributed cluster is what our customers are requiring more and more (but not all of them).  Also we cannot loose any messages we distribute work on and it is very likely the whole cluster can go down either on purpose or not.  Backing up data in a storage engine (probably for us a RDBMS or NoSQL) is something we just need to have for our situation.

Hazelcast allows this, but I've been experimenting with Hz but seems like transacted queues have been never really tested and mostly broken and I don't get a good feeling about how fast they turn things around when issues are reported.
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

At this point queues cannot be persisted yet. The ability to store any data structure or any cache to disk is currently being designed. I believe that it will become part of Ignite release by the end of March.

As far as transaction logic, I would like other community members to chime in. How difficult will it be to make queues and all other data structures transactional? 

D.



On Wed, Feb 17, 2016 at 10:37 AM, Stolidedog <[hidden email]> wrote:
So if I understand correctly you cannot store queue info in a DB currently
with Ignite V1.5?

We are a software vendor and want to build services that communicate between
each other.  The thing to note, is this is on-prem software and we don't
have anyway to know ahead of time how much nodes we can deploy.  Probably in
most cases we have two - four machines, so spinning up more nodes adding
more memory isn't "usually" a luxury afforded to us.  It is simply out of
our control.

The ability to off load storage in a distributed cluster is what our
customers are requiring more and more (but not all of them).  Also we cannot
loose any messages we distribute work on and it is very likely the whole
cluster can go down either on purpose or not.  Backing up data in a storage
engine (probably for us a RDBMS or NoSQL) is something we just need to have
for our situation.

Hazelcast allows this, but I've been experimenting with Hz but seems like
transacted queues have been never really tested and mostly broken and I
don't get a good feeling about how fast they turn things around when issues
are reported.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Stupide-question-about-Queue-tp2557p3058.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

krishd krishd
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

Have IgniteQueues been made persistent in subsequent releases of ignite ?

If not, there are there any plans for the same in forthcoming releases ?

thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Stupide question about Queue ?

This post was updated on .
Yes, this this functionality was presented in 2.1. Just turn on persistence
by calling
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setPersistentStoreConfiguration(new PersistentStoreConfiguration());

In 2.3 API has changed. Now it is required to enable persistence in a
default data region to make queues persistent:
DataStorageConfiguration dsCfg = new
DataStorageConfiguration().setDefaultDataRegionConfiguration(new
DataRegionConfiguration().setPersistenceEnabled(true));
cfg.setDataStorageConfiguration(dsCfg);



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/