RE: Support for hdfs caching

classic Classic list List threaded Threaded
4 messages Options
Munoz, Robert Munoz, Robert
Reply | Threaded
Open this post in threaded view
|

RE: Support for hdfs caching

Hello,

 

With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?

 

 

Thanks,

 

Robert Munoz

 

This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction.  Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.

The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser.  To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO.  To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer.  PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments.  In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Support for hdfs caching

Robert,

Could you please describe your use case? There several options that might suit your needs.

Denis

On Thursday, October 24, 2019, Munoz, Robert <[hidden email]> wrote:

Hello,

 

With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?

 

 

Thanks,

 

Robert Munoz

 

This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction.  Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.

The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser.  To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO.  To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer.  PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments.  In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.



--
-
Denis

Munoz, Robert Munoz, Robert
Reply | Threaded
Open this post in threaded view
|

RE: Support for hdfs caching

Hi Denis,

 

Thank you for responding. The use case is to deploy ignite cluster on our Hadoop cluster nodes and make use of persistence to HDFS since the data disks have already been allocated for HDFS. I am new to ignite so am unfamiliar with its file/folder structure but the goal would be to use ignite as in memory cache for sql tables and to persist the data to HDFS as write through / read through . This should not impact the current HDFS configuration as we have users/applications using HDFS already.  The implementation should be standard sql and not hbase get/put operations. Let me know if this helps.

 

 

 

Thanks,

 

Robert Munoz

949-220-5169

https://pimco.webex.com/meet/robert.munoz

 

 

 

From: Denis Magda <[hidden email]>
Sent: Thursday, October 24, 2019 10:08 PM
To: [hidden email]
Subject: Re: Support for hdfs caching

 

Robert,

 

Could you please describe your use case? There several options that might suit your needs.

 

Denis

On Thursday, October 24, 2019, Munoz, Robert <[hidden email]> wrote:

Hello,

 

With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?

 

 

Thanks,

 

Robert Munoz

 

This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction.  Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.

The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser.  To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO.  To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer.  PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments.  In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.



--

-

Denis

 

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Support for hdfs caching

Hi Robert,

The architecture [1] suggested here is an ideal one. You deploy Ignite separately as an in-memory store and use one of the replication techniques to update the HDFS cluster:
* Hadoop Connector form [1].
* Implement a custom version of a CacheStore [2] interface that will push all the changes from Ignite to Hadoop.

On Fri, Oct 25, 2019 at 8:21 AM Munoz, Robert <[hidden email]> wrote:

Hi Denis,

 

Thank you for responding. The use case is to deploy ignite cluster on our Hadoop cluster nodes and make use of persistence to HDFS since the data disks have already been allocated for HDFS. I am new to ignite so am unfamiliar with its file/folder structure but the goal would be to use ignite as in memory cache for sql tables and to persist the data to HDFS as write through / read through . This should not impact the current HDFS configuration as we have users/applications using HDFS already.  The implementation should be standard sql and not hbase get/put operations. Let me know if this helps.

 

 

 

Thanks,

 

Robert Munoz

949-220-5169

https://pimco.webex.com/meet/robert.munoz

 

 

 

From: Denis Magda <[hidden email]>
Sent: Thursday, October 24, 2019 10:08 PM
To: [hidden email]
Subject: Re: Support for hdfs caching

 

Robert,

 

Could you please describe your use case? There several options that might suit your needs.

 

Denis

On Thursday, October 24, 2019, Munoz, Robert <[hidden email]> wrote:

Hello,

 

With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?

 

 

Thanks,

 

Robert Munoz

 

This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction.  Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.

The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser.  To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO.  To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer.  PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments.  In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.



--

-

Denis