UbuntuInotifyTuningDemo20191025 < CF

CF Web>Linux>UbuntuInCS>UbuntuHowTo>UbuntuKernelTuning>UbuntuInotifyTuning>UbuntuInotifyTuningDemo20191025 (2019-10-28, AdrianPepper) (raw view)
EditAttach
<!--
   * Set MyBasePageName = UbuntuInotifyTuningDemo20191025
   * Set MySize = %URLPARAM{"MySize" default="14"}%
   * Set MyPlus = 1
-->
<!--
%MyBasePageName%
-->
---+ Ubuntu Inotify Tuning Demo 20191025

<div style="line-height:120%;font-size:%MySize%px">
<PRE>
<font color="black">
<i>Or how something seemingly perplexing can devolve into something trivial</i> 

<i>I almost cancelled this as being too trivial.  However, in part of it
we can see how some of the LXC creators seemed to be similarly confused
about some of the details here.</i>
</font>
</PRE>
&nbsp;

<a href="?MySize=%MySize%#00intro">Start</a>

<PRE>
































</PRE>
%TOC%

<PRE>


























</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="00intro">  </a> [[%MyBasePageName%?MySize=14#00intro][-0-]] [[%MyBasePageName%?MySize=16#00intro][-1-]] [[%MyBasePageName%?MySize=18#00intro][-2-]] [[%MyBasePageName%?MySize=20#00intro][-3-]] -->
<a name="00intro">  </a> <a href="?MySize=14#00intro">-0-</a> <a href="?MySize=16#00intro">-1-</a> <a href="?MySize=18#00intro">-2-</a> <a href="?MySize=20#00intro">-3-</a>
</div>
<PRE>
</PRE>
---+ Ubuntu Inotify Tuning Demo 20191025

<PRE>
<font color="black">
<i>Or how something seemingly perplexing can devolve into something trivial</i> 

<i>I almost cancelled this as being too trivial.  However, in part of it
we can see how some of the LXC creators seemed to be similarly confused
about some of the details here.</i>
</font>
</PRE>
&nbsp;

<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173054.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173054.png" alt="Screenshot_20191022-173054.png"  height="40" /></a> &nbsp;
<PRE>
</PRE>
---++ 00-intro


<PRE>
   Linux systems have /proc/sys

   Ubuntu systems have /etc/sysctl.conf
   -rw-r--r-- 1 root root 3751 Sep 27 13:07 /etc/sysctl.conf

   Entries like the following:
   kernel.pty.max=32768

   Cause at boottime...

   cscf-adm@xsbook7:~% grep '^' /proc/sys/kernel/pty/max
   32768
   cscf-adm@xsbook7:~% 

   A.B.C.D => /proc/sys/A/B/C/D

   Lots of parameters affect obsure details of performance (limits).
</PRE>



<!-- [[#10lxcinotify][#10-lxc+inotify]] [[#95further][#95-further]] -->
<a href="?MySize=%MySize%#10lxcinotify">#10-lxc+inotify</a> <a href="?MySize=%MySize%#95further">#95-further</a>


<PRE>












<div style="line-height:120%;font-size:smaller">
./qd-stop-demos


Two other windows, both
root@xsbook7:/home/cscf-adm/demo-20191025
</div>
</PRE>



<PRE>




























</PRE>

<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="10lxcinotify">  </a> [[%MyBasePageName%-0#10lxcinotify][-0-]] [[%MyBasePageName%#10lxcinotify][-1-]] [[%MyBasePageName%-2#10lxcinotify][-2-]] [[%MyBasePageName%-3#10lxcinotify][-3-]] -->
<a name="10lxcinotify">  </a> <a href="?MySize=14#10lxcinotify">-0-</a> <a href="?MySize=16#10lxcinotify">-1-</a> <a href="?MySize=18#10lxcinotify">-2-</a> <a href="?MySize=20#10lxcinotify">-3-</a>
</div>
<PRE>

</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173054.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173054.png" alt="Screenshot_20191022-173054.png"  height="40" /></a> &nbsp;
---++ 10-lxc+inotify

<PRE>
    "lxc" is a containerization suite that CSCF uses.
          <a href="https://linuxcontainers.org/">https://linuxcontainers.org/</a>
       - create "fake machines", e.g. even on your own workstation/laptop

    I found I could not simultaneously run more than about 7 usefully running
       lxc containers on my 16G workstation.
    <a href="https://github.com/lxc/lxd/blob/master/doc/production-setup.md">https://github.com/lxc/lxd/blob/master/doc/production-setup.md</a>
       (seemingly part of source documentation for lxd)
                (lxd is would-be successor to lxc)
       points to "ls -ld /proc/sys/*/inotify/*"
 -rw-r--r-- 1 root root 0 Oct 17 12:02 /proc/sys/fs/inotify/max_queued_events
 -rw-r--r-- 1 root root 0 Oct 17 12:02 /proc/sys/fs/inotify/max_user_instances
 -rw-r--r-- 1 root root 0 Oct 17 12:02 /proc/sys/fs/inotify/max_user_watches

    (However, in details the information there seems to be wrong).

    My laptop (12G may be relevant) now uses something like...
       cscf-adm@xsbook7:~% grep '^' /proc/sys/*/inotify/*
       /proc/sys/fs/inotify/max_queued_events:262144
       /proc/sys/fs/inotify/max_user_instances:131072
       /proc/sys/fs/inotify/max_user_watches:196608
       cscf-adm@xsbook7:~% 
</PRE>

<!-- [[#20demo01][#20-demo01]] [[#00intro][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#20demo01">#20-demo01</a> <a href="?MySize=%MySize%#00intro">&lt;prev&gt;</a>


<PRE>










grep '^' /proc/sys/*/inotify/*
</PRE>


<PRE>





























</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="20demo01">  </a> [[%MyBasePageName%-0#20demo01][-0-]] [[%MyBasePageName%#20demo01][-1-]] [[%MyBasePageName%-2#20demo01][-2-]] [[%MyBasePageName%-3#20demo01][-3-]] -->
<a name="20demo01">  </a> <a href="?MySize=14#20demo01">-0-</a> <a href="?MySize=16#20demo01">-1-</a> <a href="?MySize=18#20demo01">-2-</a> <a href="?MySize=20#20demo01">-3-</a>
</div>
<PRE>


</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173156.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173156.png" alt="Screenshot_20191022-173156.png"  height="40" /></a> &nbsp;
---++ 20-demo01

<PRE>
     To demonstrate the problem containers run into, I use the following
     go back to the default state...

     root@xsbook7:~# cat /home/cscf-adm/demo-20191025/make-default-inotify
     #!/bin/bash
     
     echo 16384  >  /proc/sys/fs/inotify/max_queued_events
     echo 128  >  /proc/sys/fs/inotify/max_user_instances
     echo 8192  >  /proc/sys/fs/inotify/max_user_watches
     
     root@xsbook7:~# 
</PRE>

<!-- [[#30demo05startlxc][#30-demo05-startlxc]] [[#10lxcinotify][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#30demo05startlxc">#30-demo05-startlxc</a> <a href="?MySize=%MySize%#10lxcinotify">&lt;prev&gt;</a>

<PRE>











<div style="line-height:120%;font-size:smaller">
./make-default-inotify

./show-inotify
</div>
</PRE>


<PRE>



























</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="30demo05startlxc">  </a> [[%MyBasePageName%-0#30demo05startlxc][-0-]] [[%MyBasePageName%#30demo05startlxc][-1-]] [[%MyBasePageName%-2#30demo05startlxc][-2-]] [[%MyBasePageName%-3#30demo05startlxc][-3-]] -->
<a name="30demo05startlxc">  </a> <a href="?MySize=14#30demo05startlxc">-0-</a> <a href="?MySize=16#30demo05startlxc">-1-</a> <a href="?MySize=18#30demo05startlxc">-2-</a> <a href="?MySize=20#30demo05startlxc">-3-</a>
</div>
<PRE>

</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173407.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173407.png" alt="Screenshot_20191022-173407.png"  height="40" /></a> &nbsp;
---++ 30-demo05-startlxc

<PRE>

Now, after setting the parameters to default, I use the following to start
all my 41 trivial demo containers...

      root@xsbook7:~# cat /home/cscf-adm/demo-20191025/start-all-demos
      #!/bin/bash

      # Takes 30 seconds to run, maybe...

      # must be superuser
      CONTAINERS=`lxc-ls -f | grep '^u....tunedemo' | awk '{print $1}' | grep -v '00$' `
      for c in $CONTAINERS ; do
	      echo $c
	      lxc-start -n $c
      done

      root@xsbook7:~# 
</PRE>

<!-- [[#40demo10showlxc][#40-demo10-showlxc]] [[#20demo01][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#40demo10showlxc">#40-demo10-showlxc</a> <a href="?MySize=%MySize%#20demo01">&lt;prev&gt;</a>

<PRE>












<div style="line-height:120%;font-size:smaller">
./start-all-demos
./show-all-demos
./show-all-demos
./show-all-demos

./qd-show-hung-demos
</div>
</PRE>





<PRE>

































</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="40demo10showlxc">  </a> [[%MyBasePageName%-0#40demo10showlxc][-0-]] [[%MyBasePageName%#40demo10showlxc][-1-]] [[%MyBasePageName%-2#40demo10showlxc][-2-]] [[%MyBasePageName%-3#40demo10showlxc][-3-]] -->
<a name="40demo10showlxc">  </a> <a href="?MySize=14#40demo10showlxc">-0-</a> <a href="?MySize=16#40demo10showlxc">-1-</a> <a href="?MySize=18#40demo10showlxc">-2-</a> <a href="?MySize=20#40demo10showlxc">-3-</a>
</div>
<PRE>


</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173501.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173501.png" alt="Screenshot_20191022-173501.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173541.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173541.png" alt="Screenshot_20191022-173541.png"  height="40" /></a> &nbsp;
---++ 40-demo10-showlxc

<PRE>
    Very sad, we wait and wait, but a lot of containers fail to get
    an IP address.  We detect that with the following command...


    root@xsbook7:~# cat /home/cscf-adm/demo-20191025/qd-show-hung-demos
    #!/bin/bash
    
    lxc-ls -f | grep RUNNING | grep -v ' 10[.]'

    root@xsbook7:~# 
</PRE>

<!-- [[#45demo12restart][#45-demo12-restart]] [[#30demo05startlxc][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#45demo12restart">#45-demo12-restart</a> <a href="?MySize=%MySize%#30demo05startlxc">&lt;prev&gt;</a>

<PRE>








<div style="line-height:120%;font-size:smaller">
./show-all-demos
./qd-show-hung-demos

./qd-show-hung-demos

./qd-show-hung-demos
</div>
</PRE>




<PRE>





















</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="45demo12restart">  </a> [[%MyBasePageName%-0#45demo12restart][-0-]] [[%MyBasePageName%#45demo12restart][-1-]] [[%MyBasePageName%-2#45demo12restart][-2-]] [[%MyBasePageName%-3#45demo12restart][-3-]] -->
<a name="45demo12restart">  </a> <a href="?MySize=14#45demo12restart">-0-</a> <a href="?MySize=16#45demo12restart">-1-</a> <a href="?MySize=18#45demo12restart">-2-</a> <a href="?MySize=20#45demo12restart">-3-</a>
</div>
<PRE>


</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173742.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173742.png" alt="Screenshot_20191022-173742.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173845.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173845.png" alt="Screenshot_20191022-173845.png"  height="40" /></a> &nbsp;
---++ 45-demo12-restart

<PRE>
     Let's try restarting all the hung containers--but it won't totally help.
     (But it did in my case let me demonstrate a red herring).
     Use the following script...


     root@xsbook7:~# cat ./qd-restart-demos
     #!/bin/bash

     STALLED=`lxc-ls -f | grep RUNNING | grep -v ' 10[.]' | awk '{print $1}' `
     for c in $STALLED; do
	     echo $c
	     lxc-stop --timeout 2 -n $c
	     lxc-start -n $c
     done

     root@xsbook7:~# 
</PRE>

<!-- [[#47demo14restart][#47-demo14-restart]] [[#40demo10showlxc][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#47demo14restart">#47-demo14-restart</a> <a href="?MySize=%MySize%#40demo10showlxc">&lt;prev&gt;</a>

<PRE>





















</PRE>


<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="47demo14restart">  </a> [[%MyBasePageName%-0#47demo14restart][-0-]] [[%MyBasePageName%#47demo14restart][-1-]] [[%MyBasePageName%-2#47demo14restart][-2-]] [[%MyBasePageName%-3#47demo14restart][-3-]] -->
<a name="47demo14restart">  </a> <a href="?MySize=14#47demo14restart">-0-</a> <a href="?MySize=16#47demo14restart">-1-</a> <a href="?MySize=18#47demo14restart">-2-</a> <a href="?MySize=20#47demo14restart">-3-</a>
</div>
<PRE>


</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173742.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173742.png" alt="Screenshot_20191022-173742.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173845.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173845.png" alt="Screenshot_20191022-173845.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173945.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-173945.png" alt="Screenshot_20191022-173945.png"  height="40" /></a> &nbsp;
---++ 47-demo14-restart

<PRE>
./qd-restart-demos
./show-all-demos
./show-all-demos
./qd-show-hung-demos
./qd-show-hung-demos | wc
</PRE>
&nbsp;
<PRE>
     Oh well.  One or two containers might advance.

     If we are lucky I can show you the "too many open files" diagnostic
     from "tail".  But that doesn't seem to happen in a live demonstration!
     (Red herring: "open files" is referring to inotify attempts)


     You can set /proc/sys/fs/file-max as high as you want and it won't help.

     cscf-adm@xsbook7:~$ grep -H '^' /proc/sys/fs/file-max
     /proc/sys/fs/file-max:2317350
     cscf-adm@xsbook7:~$ 

</PRE>

<!-- [[#50demo15fixit][#50-demo15-fixit]] [[#45demo12restart][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#50demo15fixit">#50-demo15-fixit</a> <a href="?MySize=%MySize%#45demo12restart">&lt;prev&gt;</a>

<PRE>











tail -f /var/log/syslog
</PRE>




<PRE>

















</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="50demo15fixit">  </a> [[%MyBasePageName%-0#50demo15fixit][-0-]] [[%MyBasePageName%#50demo15fixit][-1-]] [[%MyBasePageName%-2#50demo15fixit][-2-]] [[%MyBasePageName%-3#50demo15fixit][-3-]] -->
<a name="50demo15fixit">  </a> <a href="?MySize=14#50demo15fixit">-0-</a> <a href="?MySize=16#50demo15fixit">-1-</a> <a href="?MySize=18#50demo15fixit">-2-</a> <a href="?MySize=20#50demo15fixit">-3-</a>
</div>
<PRE>

</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174107.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174107.png" alt="Screenshot_20191022-174107.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174203.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174203.png" alt="Screenshot_20191022-174203.png"  height="40" /></a> &nbsp;
---++ 50-demo15-fixit

<PRE>
     So, having found the hints at
        <a href="https://github.com/lxc/lxd/blob/master/doc/production-setup.md">https://github.com/lxc/lxd/blob/master/doc/production-setup.md</a>
     I use a conservative version of them...
     (Actually while creating this demo I determined that
      max_user_instances is the only crucially important one).

     root@xsbook7:~# cat /home/cscf-adm/demo-20191025/make-good-inotify
     #!/bin/bash
     #https://github.com/lxc/lxd/blob/master/doc/production-setup.md
     # says use 1048576 = (1024*1024) for all three.
     
     # That seems sloppy.
     
     # In tests, 320 for max_user_instances seemed minimally adequate.
     # 262144 for all seemed (more than) adequate.
     # max_queued_events > max_user_watches > max_user_instances
     #echo 262144 > /proc/sys/fs/inotify/max_queued_events
     echo  360  > /proc/sys/fs/inotify/max_user_instances
     #echo 196608 > /proc/sys/fs/inotify/max_user_watches

     root@xsbook7:~# 
</PRE>
&nbsp;

<PRE>
The above action will not immediately fix the problem.
That is, the containers will not spontaneously unlock.
</PRE>

<!-- [[#60demo20restart][#60-demo20-restart]] [[#47demo14restart][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#60demo20restart">#60-demo20-restart</a> <a href="?MySize=%MySize%#47demo14restart">&lt;prev&gt;</a>

<PRE>




<div style="line-height:120%;font-size:smaller">
./make-good-inotify
./show-inotify

./show-all-demos

./qd-show-hung-demos
./qd-show-hung-demos | wc
</div>
</PRE>




<PRE>




























</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="60demo20restart">  </a> [[%MyBasePageName%-0#60demo20restart][-0-]]  [[%MyBasePageName%#60demo20restart][-1-]] [[%MyBasePageName%-2#60demo20restart][-2-]] [[%MyBasePageName%-3#60demo20restart][-3-]] -->
<a name="60demo20restart">  </a> <a href="?MySize=14#60demo20restart">-0-</a>  <a href="?MySize=16#60demo20restart">-1-</a> <a href="?MySize=18#60demo20restart">-2-</a> <a href="?MySize=20#60demo20restart">-3-</a>
</div>
<PRE>

</PRE>
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174349.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174349.png" alt="Screenshot_20191022-174349.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174440.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174440.png" alt="Screenshot_20191022-174440.png"  height="40" /></a> &nbsp;
<a href="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174529.png"><img src="%PUBURLPATH%/%WEB%/UbuntuInotifyTuningDemo20191025/Screenshot_20191022-174529.png" alt="Screenshot_20191022-174529.png"  height="40" /></a> &nbsp;
---++ 60-demo20-restart

<PRE>
     So restart all hung containers.
     root@xsbook7:~# cat /home/cscf-adm/demo-20191025/qd-restart-demos
     #!/bin/bash
     
     STALLED=`lxc-ls -f | grep RUNNING | grep -v ' 10[.]' | awk '{print $1}' `
     for c in $STALLED; do
	     echo $c
	     lxc-stop --timeout 2 -n $c
	     lxc-start -n $c
     done

     root@xsbook7:~# 


./qd-restart-demos
./show-all-demos
./qd-show-hung-demos
./qd-show-hung-demos | wc

       Hurray!
</PRE>


<!-- [[#90questions][#90-questions]] [[#50demo15fixit][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#90questions">#90-questions</a> <a href="?MySize=%MySize%#50demo15fixit">&lt;prev&gt;</a>



<PRE>



























</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="90questions">  </a> [[%MyBasePageName%-0#90questions][-0-]] [[%MyBasePageName%#90questions][-1-]] [[%MyBasePageName%-2#90questions][-2-]] [[%MyBasePageName%-3#90questions][-3-]] -->
<a name="90questions">  </a> <a href="?MySize=14#90questions">-0-</a> <a href="?MySize=16#90questions">-1-</a> <a href="?MySize=18#90questions">-2-</a> <a href="?MySize=20#90questions">-3-</a>
</div>
<PRE>


</PRE>
---++ 90-questions

<PRE>
     Take-aways...
     "lxc" <a href="https://linuxcontainers.org/">https://linuxcontainers.org/</a> is a powerful 
          near-virtualization method CSCF uses

     sysctl command and /etc/sysctl.conf are interfaces to the
          more hacky manipulation of /proc/sys (for kernel tuning)

     Some such values can impact heavy container use, and need to be changed.

     man 5 proc ; man 7 inotify
     (Actually require "manpages" package; sometimes not on containers).

     "Too many open files" red herring.


     This is your computer.                           <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/indifferent.gif">
     This is your computer on lxc.                    <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/smile.gif">
     This is your computer on lxc with some tuning.   <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/smile.gif">  <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/smile.gif">  <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/smile.gif">  <img height="%MySize%" src="%PUBURLPATH%/TWiki/SmiliesPlugin/smile.gif">


     Any questions?
</PRE>


<!-- [[#95further][#95-further]] [[#60demo20restart][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#95further">#95-further</a> <a href="?MySize=%MySize%#60demo20restart">&lt;prev&gt;</a>
<PRE>


















<div style="line-height:120%;font-size:smaller">
./qd-stop-demos
./make-new-inotify
</div>
</PRE>

<PRE>


















</PRE>
<PRE>


</PRE>
<div style="line-height:120%;font-size:smaller">
<!-- <a name="95further">  </a> [[%MyBasePageName%-0#95further][-0-]] [[%MyBasePageName%#95further][-1-]] [[%MyBasePageName%-2#95further][-2-]] [[%MyBasePageName%-3#95further][-3-]] -->
<a name="95further">  </a> <a href="?MySize=14#95further">-0-</a> <a href="?MySize=16#95further">-1-</a> <a href="?MySize=18#95further">-2-</a> <a href="?MySize=20#95further">-3-</a>
</div>
<PRE>


</PRE>
---++ 95-further

<PRE>
     /etc/sysctl.d/[0-9][0-9]-*

     Basic detailed problem in this small case is root is constrained
     as much as any other user for inotify instances.

     The <a href="http://man7.org/linux/man-pages/man8/sysctl.8.html">sysctl(8)</a> command is the correct interface to use for the
     tuning.   But direct manipulation of /proc is easier and more fun.
     In actual practice you use /etc/sysctl.conf etc.

     Recently we observed that ubuntu1804-200 had...
     ubuntu1804-200% grep -C3 inotify /etc/sysctl.d/*
     /etc/sysctl.d/10-lxd-inotify.conf:# Increase the user inotify instance limit to allow for about
     /etc/sysctl.d/10-lxd-inotify.conf-# 100 containers to run before the limit is hit again
     /etc/sysctl.d/10-lxd-inotify.conf:fs.inotify.max_user_instances = 1024
     ubuntu1804-200% 
     ubuntu1804-200% dpkg-query -S /etc/sysctl.d/10-lxd-inotify.conf
     lxd: /etc/sysctl.d/10-lxd-inotify.conf
     ubuntu1804-200% 

     So they backed off from their own suggestion of (1024*1024) for all three.

     Other things I have tweaked...
     /proc/sys/kernel/pty/max
     /proc/sys/fs/file-max
     (both probably red herrings)

     /proc/sys/kernel/keys/maxkeys            2000 from 200
     /proc/sys/net/core/netdev_max_backlog    25000 from 1000
     
     First is theoretical in my case.
     Second may be relevant if it applies to lxcbr0 local networking.

</PRE>


<PRE>


















</PRE>
<!-- <a href="#00intro">Start</a> [[#90questions][&lt;prev&gt;]] -->
<a href="?MySize=%MySize%#00intro">Start</a> <a href="?MySize=%MySize%#90questions">&lt;prev&gt;</a>
<PRE>


















</PRE>
</div>

-- Main.AdrianPepper - 2019-10-25
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	Screenshot_20191022-132438.png	r1	manage	225.3 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173054.png	r1	manage	211.1 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173156.png	r1	manage	213.7 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173407.png	r1	manage	129.2 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173501.png	r1	manage	204.0 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173541.png	r1	manage	200.1 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173742.png	r1	manage	188.8 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173845.png	r1	manage	198.6 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-173945.png	r1	manage	205.1 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-174107.png	r1	manage	197.8 K	2019-10-23 - 17:40	AdrianPepper
png	Screenshot_20191022-174203.png	r1	manage	189.3 K	2019-10-23 - 17:43	AdrianPepper
png	Screenshot_20191022-174349.png	r1	manage	188.8 K	2019-10-23 - 17:43	AdrianPepper
png	Screenshot_20191022-174440.png	r1	manage	190.6 K	2019-10-23 - 17:43	AdrianPepper
png	Screenshot_20191022-174529.png	r1	manage	213.5 K	2019-10-23 - 17:43	AdrianPepper
Topic revision: r6 - 2019-10-28 - AdrianPepper
Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.
Other Webs
My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images
Edit