This is a text-only version of the following page on https://raymii.org: --- Title : Limit specific process memory on desktop linux with cgroups and earlyoom Author : Remy van Elst Date : 13-02-2021 URL : https://raymii.org/s/articles/Limit_specific_process_memory_on_desktop_linux_with_cgroups.html Format : Markdown/HTML --- On my laptop I recently had trouble with out of memory issues when running `clion`, `firefox`, `thunderbird`, `teams` and a virtualbox VM. To combat this, I've setup cgroups to limit how much RAM specific applications can use and configured `earlyoom`, a very nifty tool that checks available memory and kills the process with the highest `oom_score` if available memory falls below 5%. Otherwise, my laptop would first grind to a halt (even without swap) and only after half an hour of seemingly being stuck would the OOM killer kick in. With `earlyoom` this hanging behavior is gone, although sometimes applications get killed when I don't expect it. I've given firefox, thunderbird and teams a cgroup with memory limit and clion and virtualbox use their own configuration to limit their RAM usage.This post details how to setup `cgroups` to limit memory of specific processes including automatically placing process inside a cgroup. ![teams requirements][3] I'm using Microsoft Teams in this example, that abomination of a chrome-browser / glorified IRC client has hardware requirements stating at least 4 GB of RAM, and that is way too much for what it's worth. Even my java-based CLion IDE doesn't use as much memory as Microsoft Teams. I've now given it 2 GB RAM max and it works just fine. 1.5 GB RAM also works in my experience. I think Teams has a memory leak because it starts with around 600 MB in use, but after running for a few hours, it is up and over 1 GB, eventually being killed by the OOM-killer in the cgroup. After restarting it's back around 600 MB again.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

This post is tested on Ubuntu 20.04. As far as documentation tells me, it should work on Debian 10 and Ubuntu 18.04 as well, but I've not tested that. Documentation on cgroups is spread out and not coherent. When looking up documentation, make sure you know which cgroup version you use and for what cgroup version the guide is written. I try to avoid systemd-specific cgroup configuration (`slices`) since this post should also be applicable for non-systemd users. Therefore you'll see me poking around in `/sys/fs/cgroup/` instead of using systemd tools. I've ended up with the following RAM limits: - Teams: 2 GB - Firefox: 2 GB - Thunderbird: 1 GB - CLion: 4 GB - Virtualbox VM: 2 GB About 11 GB reserved, leaving a bit for all other applications such as the desktop. This limited configuration runs for a few days and I haven't had the out of memory issues. Using `munin` and the [multips][5] plugin I can see that the processes stay inside their given limits. Here is the munin graph that shows the specific processes I monitor: ![multips][6] `java` is CLion, and since the machine is not on all the time, there are gaps in the graph, but it is more than enough to give a general overview of usage. The default memory usage graph is fun to look at as well: ![ram munin][7] ### Earlyoom, a more desktop friendly oom-killer Starting off with [earlyoom][1], as of Debian 10 and Ubuntu 18.04, available in the repositories, install it using the package manager: apt install earlyoom You don't have to do anything more, when your memory usage drops below 10% it will start killing processes, by default the one with the highest `oom_score`. In my case that was often `teams`, but sometimes it was firefox. I've changed the settings to only kick in when memory usage drops below 5% and have added a few processes which I'd rather not have killed. `kwin` is the KDE window manager, when that was killed my window borders were gone. Fixed by a `kwin --replace`, but annoying. `VirtualboxVM` killing could give disk corruption inside the VM, which is something I'd also rather avoid. vim /etc/default/earlyoom Contents: # Print every 60 seconds, act if free memory comes below 5% and avoid killing KDE and virtualbox EARLYOOM_ARGS="-r 60 -m 5 --avoid '(^|/)(kwin_x11|kwin|ssh|VirtualBoxVM)$'" ### Limiting memory per process with cgroups cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. It's the magic behind linux containers (lxc/docker) and I'm using it to make sure specific processes and their children cannot allocate more than a given amount of RAM. On Ubuntu you must install the following package: apt install cgroup-tools Create a cgroup, I named mine `cgTeams`: sudo cgcreate -t remy:remy -a remy:remy -g memory:/cgTeams Replace `remy:remy` by your username and group. Set the maximum amount of RAM for the newly created cgroup. The calculation to bytes isn't required on modern linux, you could just enter `2048m` but on older Debian systems you do need to specify the exact bytes: echo $(( 2048 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.limit_in_bytes #2 GB RAM If you have swap enabled, you can set a limit on that as well: echo $(( 2049 * 1024 * 1024 )) | sudo tee /sys/fs/cgroup/memory/cgTeams/memory.memsw.limit_in_bytes #2GB swap, only works if you have swap Launch Teams in the freshly created cgroup: cgexec -g memory:cgTeams teams In the next few sections I'll discuss automatic creation of cgroups at boot and automatic placement of processes inside cgroups. If all you wanted was to give a specific process a RAM limit, not making it persistent, this is all there is to it. ### What happens when a process tries to allocate more RAM than it is allowed? On modern systems, the OOM killer will kill the cgroup-ed process. Quoting the [kernel documentation][2] on this one: > When a cgroup goes over its limit, we first try to reclaim memory from the cgroup so as to make space for the new pages that the cgroup has touched. If the reclaim is unsuccessful, an OOM routine is invoked to select and kill the bulkiest task in the cgroup. I first tried to give Teams one gigabyte of RAM, which wasn't enough. Teams showed the splash screen and failed to start, `dmesg -T` showed me that it was killed right away inside the cgroup: [Thu Feb 11 12:46:42 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgTeams,task_memcg=/cgTeams,task=teams,pid=22920,uid=1000 [Thu Feb 11 12:46:42 2021] Memory cgroup out of memory: Killed process 22920 (teams) total-vm:2513452kB, anon-rss:322756kB, file-rss:57980kB, shmem-rss:0kB, UID:1000 pgtables:3444kB oom_score_adj:300 [Thu Feb 11 12:46:42 2021] oom_reaper: reaped process 22920 (teams), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Firefox behaves fun as well with a RAM limit. Large webpages get killed and a funny message is shown: ![firefox tab kill][4] `dmesg -T` shows that it is not the `firefox` process, but a child named `Web Content`: [Fri Feb 12 12:49:50 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/cgFirefox,task_memcg=/cgFirefox,task=Web Content,pid=98779,uid=1000 [Fri Feb 12 12:49:50 2021] Memory cgroup out of memory: Killed process 98779 (Web Content) total-vm:4292056kB, anon-rss:1657332kB, file-rss:108680kB, shmem-rss:95364kB, UID:1000 pgtables:8884kB oom_score_adj:0 [Fri Feb 12 12:49:50 2021] oom_reaper: reaped process 98779 (Web Content), now anon-rss:0kB, file-rss:0kB, shmem-rss:94228kB Oh well, at least my machine doesn't crash when I open a large merge request on gitlab with hundreds of changes. ### Activate cgroups at boot with cgconfigparser When this is all working you can make it permanent by placing this configuration in the file `/etc/cgconfig.conf`: group cgTeams { perm { admin { uid = remy; } task { uid = remy; } } memory { memory.limit_in_bytes = 1585446912; } } Test the file for syntax errors: sudo cgconfigparser -l /etc/cgconfig.conf The command should not give any output. For reference, I have three groups configured: group cgTeams { perm { admin { uid = remy; } task { uid = remy; } } memory { memory.limit_in_bytes = 2048m; } } group cgFirefox { perm { admin { uid = remy; } task { uid = remy; } } memory { memory.limit_in_bytes = 2048m; } } group cgThunderbird { perm { admin { uid = remy; } task { uid = remy; } } memory { memory.limit_in_bytes = 2048m; } } #### systemd service for cgconfigparser On Ubuntu 20.04 there is no `systemd` service to start `cgconfigparser` at boot. Here is a relatively simple service file I use to start `cgconfigparser`: vim /lib/systemd/system/cgconfigparser.service Contents: [Unit] Description=cgroup config parser After=network.target [Service] User=root Group=root ExecStart=/usr/sbin/cgconfigparser -l /etc/cgconfig.conf Type=oneshot [Install] WantedBy=multi-user.target Enable and start the service: systemctl enable cgconfigparser systemctl start cgconfigparser On boot the cgroups will then be created via this service. Easy to adapt to other init systems like `openrc`. You can continue on with the next section to automatically place specific processes into specific cgroups via `cgrulesengine` but, for a simpler solution, you can edit your desktop launcher to run the command prefixed with `cgexec`. Or a simple cronjob every minute that runs `cgclassify -g memory:cgTeams $(pidof teams)` to put every running `teams` process in that cgroup. Before I had set up `cgrulesengined` I used three cronjobs every minute running `cgclassify`. ### Automatically put processes into a specific cgroup with cgrulesengined This part was a bit vague online, but there exists a daemon that automatically puts processes inside cgroups based on a few rules. The startup scripts in most versions of Debian and Ubuntu are either broken or missing, but the daemon seems to work. The [manpage][8] had more information about the syntax, I'll limit the example to our usecase. Edit the following file: vim /etc/cgrules.conf Contents: remy:teams memory cgTeams/ remy:firefox memory cgFirefox/ remy:thunderbird memory cgThunderbird/ Replace `remy` by your username, the processes (`teams`) and the cgroups (`cgTeams`) by your own. Save and check the file with: /usr/sbin/cgrulesengd -vvv No output means no errors. You can omit `:processname` to limit everything by a user, `memory` can be replaced by other cgroup categories like `cpu`, but in our situation that is not applicable. I want to limit the memory of a few specific applications, not CPU cores or other resources. #### systemd service for cgrulesengined Just as with cfconfigparser, there is no default service on Ubuntu, but adding one is just as simple as the previous one. Copy one of the configuration files: cp /usr/share/doc/cgroup-tools/examples/cgred.conf /etc/cgred.conf Add a systemd unit file: vim /lib/systemd/system/cgrulesgend.service Contents: [Unit] Description=cgroup rules generator After=network.target cgconfigparser.service [Service] User=root Group=root Type=forking EnvironmentFile=-/etc/cgred.conf ExecStart=/usr/sbin/cgrulesengd Restart=on-failure [Install] WantedBy=multi-user.target Enable and start the service: systemctl enable cgrulesgend systemctl start cgrulesgend After a reboot you should have the processes you configured in To check if the process actually launches in the correct cgroup after a reboot you need to use the cgroup filesystem. The file `/sys/fs/cgroup/memory/cgTeams/tasks` lists all the process ID's that run in that cgroup. I can see that it is working for teams: $ for pid in $(cat /sys/fs/cgroup/memory/cgTeams/tasks); do pgrep $pid; done remy 3305 2.4 2.0 3688372 327456 ? Sl 06:47 3:18 /usr/share/teams/teams remy 3307 0.0 0.2 189960 39656 ? S 06:47 0:00 /usr/share/teams/teams --type=zygote --no-sandbox remy 3367 0.0 0.5 1832608 87364 ? Sl 06:47 0:00 /usr/share/teams/teams --type=renderer [1]: https://github.com/rfjakob/earlyoom [2]: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt [3]: /s/inc/img/teams-ram.png [4]: /s/inc/img/ff-ram.png [5]: https://gallery.munin-monitoring.org/plugins/munin/multips_memory/ [6]: /s/inc/img/multips_memory-day.png [7]: /s/inc/img/memory-day.png [8]: http://manpages.ubuntu.com/manpages/focal/man5/cgrules.conf.5.html --- License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source. This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it. All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Just to be clear, the information on this website is for meant for educational purposes and you use it at your own risk. I do not take responsibility if you screw something up. Use common sense, do not 'rm -rf /' as root for example. If you have any questions then do not hesitate to contact me. See https://raymii.org/s/static/About.html for details.