command line for IGFS

classic Classic list List threaded Threaded
3 messages Options
Ty Ty
Reply | Threaded
Open this post in threaded view
|

command line for IGFS

This post has NOT been accepted by the mailing list yet.
I may be trying to use Ignite the wrong way here so please correct me if I'm on a tangent to the intended use cases.

I want to store some files in a parallel file system and then operate on them in parallel.  I know I can do this with Hadoop but the system setup is a pain and Ignite seems a lot easier to get up and running on a few hosts.

The way it seems I can do this is to create a cluster that provides IGFS.  Then create another cluster that provides compute and data grid.  The computer and data grid cluster would access the IGFS cluster to read the input data files, do the computation and use the data grid to hold results; optionally pushing results back into files in the IGFS cluster.

To start my journey I have an Ignite node up and running using the example-igfs.xml config.  Unfortunately I can't seem to work out how to get my local files into IGFS from the command line.  In Hadoop I'd just use the hadoop -fs copyFromLocal command.  Is there a command line tool for IGFS operations?

I've had a quick look at ignitevisorcmd.sh but it doesn't seem to have any of this functionality.  I also had a look over the REST API and it doesn't seem to have these functions.  The examples on the web pages show that this can be done in code but I was hoping for a command line to make it a little easier to get started.

If a command line doesn't exist I can make a start on something if others think it would be a useful contribution.

Thanks,
Ty
alexey.goncharuk alexey.goncharuk
Reply | Threaded
Open this post in threaded view
|

Re: command line for IGFS

There is no a built-in command-line utility for IGFS, but you can definitely use the Hadoop command-line utility for these purposes. You do not need to start up and configure the full Hadoop cluster, just have a Hadoop installation available and configure it to work with Ignite as described here https://apacheignite.readme.io/v1.3/docs/file-system in "Configure Hadoop" section. After this you will be able to access IGFS with standard Hadoop commands.

And of course, if you think that a standalone command-line utility is a something that is worth having in Ignite, you're welcome to contribute :) Personally I think that would be great!
Ty Ty
Reply | Threaded
Open this post in threaded view
|

Re: command line for IGFS

This post has NOT been accepted by the mailing list yet.
thanks.  I'll put something basic together as a starting point and see what the community thinks of it.