• 10 dec 2017: forum version update. In case of issues use this topic.
  • 30 nov 2017: pilight moved servers. In case of issues use this topic.
Hello There, Guest! Login Register


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Too many open files
#1
This morning I couldn't open the gui anymore. The pilight process was running. When I looked into pilight.log, it appeared to be almost completely filled with  messages telling that there were too many open files.

Just a small part of pilight.log:
Code:
[Jun 23 08:11:26:964950] pilight-daemon: WARNING: (webserver) could not read /usr/local/lib/pilight/webgui//index.html
[Jun 23 08:11:27:682709] pilight-daemon: ERROR: open: Too many open files
[Jun 23 08:11:27:682835] pilight-daemon: WARNING: (webserver) could not read /usr/local/lib/pilight/webgui//index.html
[Jun 23 08:11:27:686036] pilight-daemon: ERROR: open: Too many open files
[Jun 23 08:11:27:686160] pilight-daemon: WARNING: (webserver) could not read /usr/local/lib/pilight/webgui//index.html
[Jun 23 08:11:35:171393] pilight-daemon: ERROR: open: Too many open files
[Jun 23 08:11:35:171515] pilight-daemon: WARNING: (webserver) could not read /usr/local/lib/pilight/webgui//index.html
[ Jun 23 08:11:45:96771] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97025] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97182] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97331] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97481] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97628] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97775] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:97920] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98068] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98220] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98367] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98513] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98659] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98804] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:98949] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99099] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99246] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99393] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99539] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99683] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99829] pilight-daemon: NOTICE: accept: Too many open files
[ Jun 23 08:11:45:99978] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100124] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100318] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100485] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100636] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100782] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:100927] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101073] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101220] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101366] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101512] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101659] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101803] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:101948] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:102095] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:102239] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:102383] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 08:11:45:102527] pilight-daemon: NOTICE: accept: Too many open files
[Jun 23 0

After restarting pilight, the gui was working again, but, using lsof, I found that pilight has a large number of files opened and most of them several times.

This seems to indicate that these files are being opened multiple times, without closing them in between. Maybe this is by design, but I assume it is a bug.

The lsof output for files opened by the pilight process  is so huge that I cannot add it in a code section, so I attached a zip file.


Attached Files
.zip   lsof.zip (Size: 8.97 KB / Downloads: 2)
 
Reply
#2
Yes, indeed, i'm probably missing a file close somewhere, but were? Are you using additional features besides the webserver that reads from files?
 
Reply
#3
Yes, I am using my write action, which is opening a file for write or append, but I am always closing it right after the write. But I will double check.
 
Reply
#4
Can you run a test instance without your write function to make sure it's not my bug.
 
Reply
#5
Hopefully I found something that can help finding the bug.

I ran pilight with my config, step by step removing the contents of the rules, gui- and device sections. Finally I only had my settings in the config, but I still saw all of those open files. If I ran pilight with a different config the number of open files was different too.

Looking closer at the lsof output you can see that in the PID column (apart from the first time the files were opened by pilight) there are two PIDs shown. Forgive me if my interpretation is wrong, but I assume that this means that pilight subprocesses (26 of them) have opened those files.  However, at the time lsof is run,  those subprocesses don't exist anymore.

Comparing the configs used for testing, I noticed that the one with less open files has less entries in the "settings" section. A striking thing is that the number of times that all pilight files have been opened is more ore less equal to the number of entries in the settings section of the config.  It can be just a coincidence, but the correlation is so strong that I hope this is helpfull.
 
Reply
#6
The number of open files doesn't say much about the files not being closed. You're the first one reporting it, but also one of the few having own additions. So i really want to make sure it's me who made the mistake.
 
Reply
#7
I don't understand your response Unsure

I was not talking about the number of open files, but about the number of times (27 times in my case) the same files have been opened and have not been closed.

I tested with all my own modules removed and a config only containing entries in the "settings" section. The behaviour is 100% reproducible with this setup.

The reason why nobody reported this before, could be that pilight seems to work normally as long as the number of open files doesn't reach the limit.
Because I have several modules of my own (16 in all), pilight has to load more files for me than for "standard users". If those additional files are being opened 27 times and not closed, I get an exta of 432 open files, making the chance of hitting the limit simply bigger.
 
Reply
#8
Can you post that minimal config that i can use, and describe the lsof approach that you used to validate this?
 
Reply
#9
You can simply use the default config provided with pilight.

Then run lsof (no paramaters) and look at the lines starting with "pilight-d". The second column is for the PID of the process that has opened the file. You can see that the files are being opened over and over again.

First there is a block with only  the PID of the daemon. Next come 12 blocks with 2 PIDS, the first PID again is the one of of the daemon. You can see the second PID number going up for each following block. I assume those are the PIDs of subprocesses created by the daemon, but those processes actially don't exist anymore.

First block:

Code:
pilight-d   745                 root  cwd       DIR      179,2     4096       1318 /home/pi
pilight-d   745                 root  rtd       DIR      179,2     4096          2 /
pilight-d   745                 root  txt       REG      179,2   120416      37066 /usr/local/sbin/pilight-daemon
pilight-d   745                 root  mem       REG      179,2   132928       1910 /lib/arm-linux-gnueabihf/liblzma.so.5.2.2
pilight-d   745                 root  mem       REG      179,2   116372       1899 /lib/arm-linux-gnueabihf/libgcc_s.so.1
pilight-d   745                 root  mem       REG      179,2  1234700       1884 /lib/arm-linux-gnueabihf/libc-2.24.so
pilight-d   745                 root  mem       REG      179,2    47808      37654 /usr/lib/arm-linux-gnueabihf/libunwind.so.8.0.1
pilight-d   745                 root  mem       REG      179,2   227512      20998 /usr/lib/arm-linux-gnueabihf/libpcap.so.1.8.1
pilight-d   745                 root  mem       REG      179,2   373676      40612 /usr/lib/arm-linux-gnueabihf/libluajit-5.1.so.2.0.4
pilight-d   745                 root  mem       REG      179,2    66956      19426 /usr/lib/arm-linux-gnueabihf/libmbedx509.so.2.4.2
pilight-d   745                 root  mem       REG      179,2   284704      19410 /usr/lib/arm-linux-gnueabihf/libmbedcrypto.so.2.4.2
pilight-d   745                 root  mem       REG      179,2   148896      19433 /usr/lib/arm-linux-gnueabihf/libmbedtls.so.2.4.2
pilight-d   745                 root  mem       REG      179,2   141496      19325 /usr/lib/libwiringx.so
pilight-d   745                 root  mem       REG      179,2    26632       1949 /lib/arm-linux-gnueabihf/librt-2.24.so
pilight-d   745                 root  mem       REG      179,2   127300       1945 /lib/arm-linux-gnueabihf/libpthread-2.24.so
pilight-d   745                 root  mem       REG      179,2   452152       1911 /lib/arm-linux-gnueabihf/libm-2.24.so
pilight-d   745                 root  mem       REG      179,2     9800       1893 /lib/arm-linux-gnueabihf/libdl-2.24.so
pilight-d   745                 root  mem       REG      179,2 10716592      13519 /usr/local/lib/libpilight.so.8.1.0
pilight-d   745                 root  mem       REG      179,2    18920      10467 /usr/lib/arm-linux-gnueabihf/libarmmem.so
pilight-d   745                 root  mem       REG      179,2   138576       1837 /lib/arm-linux-gnueabihf/ld-2.24.so
pilight-d   745                 root    0u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745                 root    1u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745                 root    2u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745                 root    3r     FIFO       0,10      0t0    6165026 pipe
pilight-d   745                 root    4w     FIFO       0,10      0t0    6165026 pipe
pilight-d   745                 root    5u  a_inode       0,11        0       3906 [eventpoll]
pilight-d   745                 root    6r     FIFO       0,10      0t0    6165027 pipe
pilight-d   745                 root    7w     FIFO       0,10      0t0    6165027 pipe
pilight-d   745                 root    8u  a_inode       0,11        0       3906 [eventfd]
pilight-d   745                 root    9u     IPv4    6165041      0t0        TCP *:35439 (LISTEN)
pilight-d   745                 root   10u     IPv4    6165042      0t0        TCP localhost:46932->localhost:35439 (ESTABLISHED)
pilight-d   745                 root   11u     IPv4    6165043      0t0        UDP *:1900
pilight-d   745                 root   12u     IPv4    6165044      0t0        TCP *:rfe (LISTEN)
pilight-d   745                 root   13u     IPv4    6165045      0t0        TCP *:5001 (LISTEN)
pilight-d   745                 root   14u     IPv4    6165046      0t0        TCP localhost:35439->localhost:46932 (ESTABLISHED)
pilight-d   745                 root   15u     IPv4    6165052      0t0        TCP 192.168.1.14:54870->192.168.1.14:35439 (ESTABLISHED)
pilight-d   745                 root   16u     IPv4    6165053      0t0        TCP 192.168.1.14:35439->192.168.1.14:54870 (ESTABLISHED)

Second block:

Code:
pilight-d   745 746             root  cwd       DIR      179,2     4096       1318 /home/pi
pilight-d   745 746             root  rtd       DIR      179,2     4096          2 /
pilight-d   745 746             root  txt       REG      179,2   120416      37066 /usr/local/sbin/pilight-daemon
pilight-d   745 746             root  mem       REG      179,2   132928       1910 /lib/arm-linux-gnueabihf/liblzma.so.5.2.2
pilight-d   745 746             root  mem       REG      179,2   116372       1899 /lib/arm-linux-gnueabihf/libgcc_s.so.1
pilight-d   745 746             root  mem       REG      179,2  1234700       1884 /lib/arm-linux-gnueabihf/libc-2.24.so
pilight-d   745 746             root  mem       REG      179,2    47808      37654 /usr/lib/arm-linux-gnueabihf/libunwind.so.8.0.1
pilight-d   745 746             root  mem       REG      179,2   227512      20998 /usr/lib/arm-linux-gnueabihf/libpcap.so.1.8.1
pilight-d   745 746             root  mem       REG      179,2   373676      40612 /usr/lib/arm-linux-gnueabihf/libluajit-5.1.so.2.0.4
pilight-d   745 746             root  mem       REG      179,2    66956      19426 /usr/lib/arm-linux-gnueabihf/libmbedx509.so.2.4.2
pilight-d   745 746             root  mem       REG      179,2   284704      19410 /usr/lib/arm-linux-gnueabihf/libmbedcrypto.so.2.4.2
pilight-d   745 746             root  mem       REG      179,2   148896      19433 /usr/lib/arm-linux-gnueabihf/libmbedtls.so.2.4.2
pilight-d   745 746             root  mem       REG      179,2   141496      19325 /usr/lib/libwiringx.so
pilight-d   745 746             root  mem       REG      179,2    26632       1949 /lib/arm-linux-gnueabihf/librt-2.24.so
pilight-d   745 746             root  mem       REG      179,2   127300       1945 /lib/arm-linux-gnueabihf/libpthread-2.24.so
pilight-d   745 746             root  mem       REG      179,2   452152       1911 /lib/arm-linux-gnueabihf/libm-2.24.so
pilight-d   745 746             root  mem       REG      179,2     9800       1893 /lib/arm-linux-gnueabihf/libdl-2.24.so
pilight-d   745 746             root  mem       REG      179,2 10716592      13519 /usr/local/lib/libpilight.so.8.1.0
pilight-d   745 746             root  mem       REG      179,2    18920      10467 /usr/lib/arm-linux-gnueabihf/libarmmem.so
pilight-d   745 746             root  mem       REG      179,2   138576       1837 /lib/arm-linux-gnueabihf/ld-2.24.so
pilight-d   745 746             root    0u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 746             root    1u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 746             root    2u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 746             root    3r     FIFO       0,10      0t0    6165026 pipe
pilight-d   745 746             root    4w     FIFO       0,10      0t0    6165026 pipe
pilight-d   745 746             root    5u  a_inode       0,11        0       3906 [eventpoll]
pilight-d   745 746             root    6r     FIFO       0,10      0t0    6165027 pipe
pilight-d   745 746             root    7w     FIFO       0,10      0t0    6165027 pipe
pilight-d   745 746             root    8u  a_inode       0,11        0       3906 [eventfd]
pilight-d   745 746             root    9u     IPv4    6165041      0t0        TCP *:35439 (LISTEN)
pilight-d   745 746             root   10u     IPv4    6165042      0t0        TCP localhost:46932->localhost:35439 (ESTABLISHED)
pilight-d   745 746             root   11u     IPv4    6165043      0t0        UDP *:1900
pilight-d   745 746             root   12u     IPv4    6165044      0t0        TCP *:rfe (LISTEN)
pilight-d   745 746             root   13u     IPv4    6165045      0t0        TCP *:5001 (LISTEN)
pilight-d   745 746             root   14u     IPv4    6165046      0t0        TCP localhost:35439->localhost:46932 (ESTABLISHED)
pilight-d   745 746             root   15u     IPv4    6165052      0t0        TCP 192.168.1.14:54870->192.168.1.14:35439 (ESTABLISHED)
pilight-d   745 746             root   16u     IPv4    6165053      0t0        TCP 192.168.1.14:35439->192.168.1.14:54870 (ESTABLISHED)

Third block:

Code:
pilight-d   745 747             root  cwd       DIR      179,2     4096       1318 /home/pi
pilight-d   745 747             root  rtd       DIR      179,2     4096          2 /
pilight-d   745 747             root  txt       REG      179,2   120416      37066 /usr/local/sbin/pilight-daemon
pilight-d   745 747             root  mem       REG      179,2   132928       1910 /lib/arm-linux-gnueabihf/liblzma.so.5.2.2
pilight-d   745 747             root  mem       REG      179,2   116372       1899 /lib/arm-linux-gnueabihf/libgcc_s.so.1
pilight-d   745 747             root  mem       REG      179,2  1234700       1884 /lib/arm-linux-gnueabihf/libc-2.24.so
pilight-d   745 747             root  mem       REG      179,2    47808      37654 /usr/lib/arm-linux-gnueabihf/libunwind.so.8.0.1
pilight-d   745 747             root  mem       REG      179,2   227512      20998 /usr/lib/arm-linux-gnueabihf/libpcap.so.1.8.1
pilight-d   745 747             root  mem       REG      179,2   373676      40612 /usr/lib/arm-linux-gnueabihf/libluajit-5.1.so.2.0.4
pilight-d   745 747             root  mem       REG      179,2    66956      19426 /usr/lib/arm-linux-gnueabihf/libmbedx509.so.2.4.2
pilight-d   745 747             root  mem       REG      179,2   284704      19410 /usr/lib/arm-linux-gnueabihf/libmbedcrypto.so.2.4.2
pilight-d   745 747             root  mem       REG      179,2   148896      19433 /usr/lib/arm-linux-gnueabihf/libmbedtls.so.2.4.2
pilight-d   745 747             root  mem       REG      179,2   141496      19325 /usr/lib/libwiringx.so
pilight-d   745 747             root  mem       REG      179,2    26632       1949 /lib/arm-linux-gnueabihf/librt-2.24.so
pilight-d   745 747             root  mem       REG      179,2   127300       1945 /lib/arm-linux-gnueabihf/libpthread-2.24.so
pilight-d   745 747             root  mem       REG      179,2   452152       1911 /lib/arm-linux-gnueabihf/libm-2.24.so
pilight-d   745 747             root  mem       REG      179,2     9800       1893 /lib/arm-linux-gnueabihf/libdl-2.24.so
pilight-d   745 747             root  mem       REG      179,2 10716592      13519 /usr/local/lib/libpilight.so.8.1.0
pilight-d   745 747             root  mem       REG      179,2    18920      10467 /usr/lib/arm-linux-gnueabihf/libarmmem.so
pilight-d   745 747             root  mem       REG      179,2   138576       1837 /lib/arm-linux-gnueabihf/ld-2.24.so
pilight-d   745 747             root    0u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 747             root    1u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 747             root    2u      CHR      136,0      0t0          3 /dev/pts/0
pilight-d   745 747             root    3r     FIFO       0,10      0t0    6165026 pipe
pilight-d   745 747             root    4w     FIFO       0,10      0t0    6165026 pipe
pilight-d   745 747             root    5u  a_inode       0,11        0       3906 [eventpoll]
pilight-d   745 747             root    6r     FIFO       0,10      0t0    6165027 pipe
pilight-d   745 747             root    7w     FIFO       0,10      0t0    6165027 pipe
pilight-d   745 747             root    8u  a_inode       0,11        0       3906 [eventfd]
pilight-d   745 747             root    9u     IPv4    6165041      0t0        TCP *:35439 (LISTEN)
pilight-d   745 747             root   10u     IPv4    6165042      0t0        TCP localhost:46932->localhost:35439 (ESTABLISHED)
pilight-d   745 747             root   11u     IPv4    6165043      0t0        UDP *:1900
pilight-d   745 747             root   12u     IPv4    6165044      0t0        TCP *:rfe (LISTEN)
pilight-d   745 747             root   13u     IPv4    6165045      0t0        TCP *:5001 (LISTEN)
pilight-d   745 747             root   14u     IPv4    6165046      0t0        TCP localhost:35439->localhost:46932 (ESTABLISHED)
pilight-d   745 747             root   15u     IPv4    6165052      0t0        TCP 192.168.1.14:54870->192.168.1.14:35439 (ESTABLISHED)
pilight-d   745 747             root   16u     IPv4    6165053      0t0        TCP 192.168.1.14:35439->192.168.1.14:54870 (ESTABLISHED)

And so on ….
 
Reply
#10
There was indeed a bug in the webserver.

The processes you see are completely normal. Those are the different threads pilight starts. The rewrite effort is working towards bringing those down to maximum four.
 
Reply
  


Possibly Related Threads...
Thread Author Replies Views Last Post
  Too many open files 0 55 Less than 1 minute ago
Last Post:
  Too many open files 0 114 Less than 1 minute ago
Last Post:

Forum Jump:


Browsing: 1 Guest(s)