Too many file descriptors on ignite node

classic Classic list List threaded Threaded
4 messages Options
ihalilaltun ihalilaltun
Reply | Threaded
Open this post in threaded view
|

Too many file descriptors on ignite node

Hi Igniters it is me again :)

We are having a wierd behivor on some of cluster nodes. Cluster uses native
persistance with MMAP disabled. Some clusters have too many wal files even
if they are already deleted, but for some reason they are stll persisted on
the disk. I do not have any logs on cluster or related machines, but I have
ss's and file descriptors list;
here is the ss from related node;
Screen_Shot_2019-09-23_at_14.png
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/Screen_Shot_2019-09-23_at_14.png>  
here is the file descriptor list;
open_files.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/open_files.zip>  

I am not sure if this is related to
https://issues.apache.org/jira/browse/IGNITE-12127, i hope it is, since we
are planning to upgrade all clusters before end of this week. If this is not
related to IGNITE-12127, then any comments how this is possible.

cheers



-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Too many file descriptors on ignite node

Hi,

This sounds more like a need for configuration changes. You might need to adjust a number of WAL segments and tweak WAL archive related parameters. Please check this discussion below

and WAL parameters of DataStorageConfiguration class.

-
Denis


On Mon, Sep 23, 2019 at 4:29 AM ihalilaltun <[hidden email]> wrote:
Hi Igniters it is me again :)

We are having a wierd behivor on some of cluster nodes. Cluster uses native
persistance with MMAP disabled. Some clusters have too many wal files even
if they are already deleted, but for some reason they are stll persisted on
the disk. I do not have any logs on cluster or related machines, but I have
ss's and file descriptors list;
here is the ss from related node;
Screen_Shot_2019-09-23_at_14.png
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/Screen_Shot_2019-09-23_at_14.png
here is the file descriptor list;
open_files.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/open_files.zip

I am not sure if this is related to
https://issues.apache.org/jira/browse/IGNITE-12127, i hope it is, since we
are planning to upgrade all clusters before end of this week. If this is not
related to IGNITE-12127, then any comments how this is possible.

cheers



-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
ihalilaltun ihalilaltun
Reply | Threaded
Open this post in threaded view
|

Re: Too many file descriptors on ignite node

Hi Denis,

Our problem is not related to configuration parameters. We already limited
archive size to 8, but some nodes do not release file from filesystem. When
we look at the archive directory we only see 8 files, but when we look at
the file descriptors on the server, we get thousands of wal files that is
not released from filesystem. The list i shared is from filesystem. I think
there is a big issue here.



-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Too many file descriptors on ignite node

Ibrahim,

I've checked with our community members (copied) and it doesn't look like a known issue. Sergey, what details do we need to reproduce the behavior?

-
Denis


On Mon, Sep 30, 2019 at 4:24 AM ihalilaltun <[hidden email]> wrote:
Hi Denis,

Our problem is not related to configuration parameters. We already limited
archive size to 8, but some nodes do not release file from filesystem. When
we look at the archive directory we only see 8 files, but when we look at
the file descriptors on the server, we get thousands of wal files that is
not released from filesystem. The list i shared is from filesystem. I think
there is a big issue here.



-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/