NameNode sync using DUAL_ASYNC mode on IGFS

classic Classic list List threaded Threaded
4 messages Options
Masayuki Takahashi Masayuki Takahashi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

NameNode sync using DUAL_ASYNC mode on IGFS

Hi,

I am trying to use IGFS on HDFS.

If I set DUAL_ASYNC to IGFS mode and execute 'hdfs dfs -put ...', when
does the file info write to HDFS NameNode?

Also, if I set PRIMARY to IGFS mode and put a new file, does the file
info write to HDFS NameNode?

thanks.
--
Masayuki Takahashi
dsetrakyan dsetrakyan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NameNode sync using DUAL_ASYNC mode on IGFS

Hi Massayuki,

Ignite itself does not have a concept of a NameNode. It goes directly to the cluster node responsible for storing the data based on a key's hashcode.

The only time when a NameNode would come into a play, is when Hadoop HDFS is configured as an underlying write-through file system. Basically it means that every time a data is changed in Ignite, it will also be changed in the underlying HDFS, either synchronously or asynchronously, based on the IGFS configuration. In this case, HDFS would contact the NameNode whenever the data is written into it, based on the its own native protocol, which has nothing to do with Ignite itself.

Let me know if you have more questions.

D.

On Sun, Apr 2, 2017 at 5:47 AM, Masayuki Takahashi <[hidden email]> wrote:
Hi,

I am trying to use IGFS on HDFS.

If I set DUAL_ASYNC to IGFS mode and execute 'hdfs dfs -put ...', when
does the file info write to HDFS NameNode?

Also, if I set PRIMARY to IGFS mode and put a new file, does the file
info write to HDFS NameNode?

thanks.
--
Masayuki Takahashi

Masayuki Takahashi Masayuki Takahashi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NameNode sync using DUAL_ASYNC mode on IGFS

Hi Dmitriy,

Thanks for explaining. I understand.
In situations using Name node, it's better to use DUAL_SYNC mode.

May I ask one more?

> Basically it means that every time a data is changed in Ignite, it will also be changed in the underlying HDFS, either synchronously or asynchronously, based on the IGFS configuration.

When using PRIMARY mode, I think that name node is not referenced.
What kind of use case is assumed for PRIMARY mode?

thanks.

2017-04-04 15:11 GMT+09:00 Dmitriy Setrakyan <[hidden email]>:

> Hi Massayuki,
>
> Ignite itself does not have a concept of a NameNode. It goes directly to the
> cluster node responsible for storing the data based on a key's hashcode.
>
> The only time when a NameNode would come into a play, is when Hadoop HDFS is
> configured as an underlying write-through file system. Basically it means
> that every time a data is changed in Ignite, it will also be changed in the
> underlying HDFS, either synchronously or asynchronously, based on the IGFS
> configuration. In this case, HDFS would contact the NameNode whenever the
> data is written into it, based on the its own native protocol, which has
> nothing to do with Ignite itself.
>
> Let me know if you have more questions.
>
> D.
>
> On Sun, Apr 2, 2017 at 5:47 AM, Masayuki Takahashi <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> I am trying to use IGFS on HDFS.
>>
>> If I set DUAL_ASYNC to IGFS mode and execute 'hdfs dfs -put ...', when
>> does the file info write to HDFS NameNode?
>>
>> Also, if I set PRIMARY to IGFS mode and put a new file, does the file
>> info write to HDFS NameNode?
>>
>> thanks.
>> --
>> Masayuki Takahashi
>
>



--
高橋 真之
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NameNode sync using DUAL_ASYNC mode on IGFS

Hi,

You may use it in cases when you use temporary files that should be removed after node stop, or any other data that don't require persistence.

-Dmitry.
Loading...