looker: Update heap size calculation for cgroups#23
looker: Update heap size calculation for cgroups#23mattrobenolt wants to merge 1 commit intolooker-open-source:mainfrom
Conversation
When running within Docker, it's less than ideal to use /proc/meminfo since this is the memory on the entire node. Not what has been allocated to the running container. For that, we need to yank the data out of the cgroup limit instead. For inspiration, this was mostly yanked out of what the RabbitMQ docker container does: https://github.com/docker-library/rabbitmq/blob/master/3.7/alpine/docker-entrypoint.sh#L300-L316
| memLimitKb="$memTotalKb" | ||
| fi | ||
| JAVAMEM="$(awk -v limKb="$memLimitKb" 'BEGIN{ | ||
| limKb = limKb * 6 / 10; |
There was a problem hiding this comment.
I kept this math the same, but using 60% feels a bit wrong to me, if your goal is to basically be limKb = limKb - (1024 * 1024 * 1.5);
So let me know if you want me to accommodate for that instead, I just didn't wanna change behavior.
There was a problem hiding this comment.
It seems imo, if we have a cgroup, we can set the limit to exactly the cgroup (maybe subtract a few MB to keep some breathing room), and apply the 60% or some -1.5GB of the value comes from the whole system's memory. If we're running within a container, I'm not sure why we'd need to have that much breathing room.
There was a problem hiding this comment.
The JVM has support for containers now, so you can just use something like: -XX:MaxRAMPercentagerather=90.0.
There was a problem hiding this comment.
This is probably irrelevant now since as you mentioned the JVM now supports automatically setting the heap size when it detects it is running in a container.
However, when running on Ubuntu 20.04.5 LTS the /sys/fs/cgroup/memory/memory.limit_in_bytes file seems to have been replaced by /sys/fs/cgroup/memory.max.
For me, running this in docker, the memory.max file stores the max memory limit in bytes.
$ cat /sys/fs/cgroup/memory.max
4294967296
There was a problem hiding this comment.
Excuse my ignorance of cgroups, this seems to be a platform issue, when running Docker Desktop on MacOS /sys/fs/cgroup/memory.max is the file but when running on a Linux node the file was /sys/fs/cgroup/memory/memory.limit_in_bytes
3 participants
When running within Docker, it's less than ideal to use /proc/meminfo since this is the memory on the entire node. Not what has been allocated to the running container. For that, we need to yank the data out of the cgroup limit instead.
For inspiration, this was mostly yanked out of what the RabbitMQ docker container does: https://github.com/docker-library/rabbitmq/blob/master/3.7/alpine/docker-entrypoint.sh#L300-L316