Local secondary file system - problem to list files

classic Classic list List threaded Threaded
7 messages Options
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Local secondary file system - problem to list files

This post was updated on .
asked by alexandre verri

I am trying to write a class to implement IgfsSecondaryFileSystem in order to use the local file system as a secondary file system. Each Ignite path is mapped to a path in the local file system, example: if the Ignite path is '/myDir/test.txt', the file is created in '/home/user/myDir/test.txt'.

I have created the following test using two nodes of Ignite:

1) Create a file in the local filesystem on the first node (ex: /home/user/myDir/test.txt). 2) Start a second node in other host, and call the listFiles(new IgfsPath("/myDir")) in the IgniteFileSystem instance.

The file 'test.txt' is not returned by the method listFiles in the second node.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss

pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

commented by alexandre verri

Obs: I have forgotten to mention: the '/home/user/myDir/test.txt' file is NOT present in the local file system on the second node.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss

pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

This post was updated on .
commented by alexandre verri

The code:

public static void main(String[] args) throws IOException {
    // New configuration.
    final IgniteConfiguration cfg = new IgniteConfiguration();

    final String filesystemName = "myFs";
    final String metaCacheName = "myMetaCache";
    final String dataCacheName = "myDataCache";

    // Data cache configuration.
    final CacheConfiguration dataCacheCfg = new CacheConfiguration();
    dataCacheCfg.setName(dataCacheName);
    dataCacheCfg.setOffHeapMaxMemory(500 * 1024 * 1024);
    dataCacheCfg.setCacheMode(CacheMode.PARTITIONED);
    dataCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
    dataCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
    dataCacheCfg.setAffinityMapper(new IgfsGroupDataBlocksKeyMapper(512));

    // Meta cache configuration.
    final CacheConfiguration metaCacheCfg = new CacheConfiguration();
    metaCacheCfg.setName(metaCacheName);
    metaCacheCfg.setCacheMode(CacheMode.REPLICATED);
    metaCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
    metaCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);

    // Filesystem configuration.
    final FileSystemConfiguration fsConfig = new FileSystemConfiguration();
    fsConfig.setSecondaryFileSystem(new DefaultLocalSecondaryFilesystem(Paths.get("/home/averri/ignite")));
    fsConfig.setDefaultMode(IgfsMode.DUAL_SYNC);
    fsConfig.setName(filesystemName);
    fsConfig.setMetaCacheName(metaCacheName);
    fsConfig.setDataCacheName(dataCacheName);
    fsConfig.setIpcEndpointEnabled(false);

    cfg.setFileSystemConfiguration(fsConfig);
    cfg.setCacheConfiguration(dataCacheCfg, metaCacheCfg);

    final Ignite ignite = Ignition.start(cfg);
    // Get the filesystem.
    final IgniteFileSystem fs = ignite.fileSystem(filesystemName);

    // List the contents of the directory.
    fs.listFiles(new IgfsPath("/myDir")).forEach(f -> System.out.println("### File: " + f.toString()));
}


The DefaultLocalSecondaryFilesystem class:

public class DefaultLocalSecondaryFilesystem implements IgfsSecondaryFileSystem {
        // The root of the secondary filesystem.
        private final Path root;

        public DefaultLocalSecondaryFilesystem(Path root) throws IOException {
            if (root == null) {
                throw new IllegalArgumentException("The root of DefaultLocalSecondaryFilesystem cannot be null.");
            }
            try {
                Files.createDirectory(root);
            } catch (FileAlreadyExistsException e) {
                // Ignore.
            }
            this.root = root;
        }

        @Override
        public boolean exists(IgfsPath igPath) {
            return Files.exists(resolve(igPath));
        }

        @Override
        public IgfsFile update(IgfsPath path, Map<String, String> props) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public void rename(IgfsPath src, IgfsPath dest) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public boolean delete(IgfsPath path, boolean recursive) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public void mkdirs(IgfsPath path) throws IgniteException {
            call(() -> Files.createDirectories(resolve(path)));
        }

        @Override
        public void mkdirs(IgfsPath path, Map<String, String> props) throws IgniteException {
            // TODO: to consider the props.
            mkdirs(path);
        }

        @Override
        public Collection<IgfsPath> listPaths(IgfsPath path) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public Collection<IgfsFile> listFiles(IgfsPath path) throws IgniteException {
            return call(() -> Files.list(resolve(path)).map(this::info).collect(Collectors.toList()));
        }

        @Override
        public IgfsSecondaryFileSystemPositionedReadable open(IgfsPath path, int bufSize) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public OutputStream create(IgfsPath path, boolean overwrite) throws IgniteException {
            return call(() -> Files.newOutputStream(resolve(path),
                    overwrite ? StandardOpenOption.WRITE : StandardOpenOption.CREATE));
        }

        @Override
        public OutputStream create(IgfsPath path, int bufSize, boolean overwrite, int replication, long blockSize, Map<String, String> props) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public OutputStream append(IgfsPath path, int bufSize, boolean create, Map<String, String> props) throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public IgfsFile info(IgfsPath igPath) throws IgniteException {
            return call(() -> {
                final Path fsPath = resolve(igPath);
                if (Files.exists(fsPath)) {
                    IgfsFileInfo info = new IgfsFileInfo(Files.isDirectory(fsPath), new HashMap<>());
                    // TODO: How to calculate the last parameter?
                    return new IgfsFileImpl(igPath, info, 0);
                }
                return null;
            });
        }

        @Override
        public long usedSpaceSize() throws IgniteException {
            throw new UnsupportedOperationException("Not implemented yet.");
        }

        @Override
        public Map<String, String> properties() {
            // TODO: Implement.
            return new ConcurrentHashMap<>();
        }

        private Path resolve(IgfsPath igPath) {
            return root.resolve(igPath.toString().replaceFirst("[/\\\\]", ""));
        }

        private IgfsFile info(Path fsPath) {
            final int length = root.toString().length();
            return info(new IgfsPath(fsPath.toString().substring(length).replaceAll("\\\\", "/")));
        }

        private static <T> T call(Callable<T> callable) throws IgniteException {
            if (callable != null) {
                try {
                    return callable.call();
                } catch (Exception e) {
                    throw new IgniteException(e);
                }
            }
            return null;
        }
}  


In this code example, the file '/home/averri/ignite/myDir/test.txt' exists only in the first node. If we call the method listFiles in the second node, there will be no files.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

commented by alexey goncharuk

Local file system will not work as a secondary file system because secondary file system must be distributed, i.e. each node must see the same file system state (the same way HDFS works). You can point your implementation to a network-mapped folder, like SMB or NFS share, then it should work.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

commented by alexandre verri

Alexey, thank you very much for the information. I will test the DefaultLocalSecondaryFilesystem using the NFS. I will share the code when finished.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

commented by dmitriy setrakyan

Hi Alexandre, do you have an update on this? Should we mark it as "solved"?

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss
pgarg pgarg
Reply | Threaded
Open this post in threaded view
|

Re: Local secondary file system - problem to list files

commented by alexandre verri

Hi Dmitriy, you may mark it as solved. I did not finish the code yet due to lack of time. I am resigning from the current company, and moving to another country.

-----
This post is migrated from now discontinued Apache Ignite forum at
http://apacheignite.readme.io/v1.0/discuss