I’ve worked some times ago, to troubleshoot a Nodejs microservice on an Openshift platform. This poor beast was OOMKilled by Kubernetes, due to a memory limit value exceeded. And it happened when the incoming traffic rate grown up suddenly. During this investigation I’ve learned a lot about Nodejs 12 and its V8 engine’s memory management. And I found its paradigm really interesting in a k8s cluster. So I’ve written this post, to share with you this Nodejs Kubernetes memory adventure.

Nodejs Kubernetes update version

The Docker Nodejs microservice uses Nodejs 10. Many people encountered memory issues, and demonstrate with docker containers that exceeds limit due to Nodejs hard coded heap memory size. And the main advice I can read is:

update to last NodeJS 12 version, the memory management will be improved, and take care of system memory limits.

Before Nodejs 12, versions have an heap memory size that depends on the OS (32 or 64 bits). So, following documentations, on 64-bit machines that (the old generation alone) would be 1400 MB. Indeed for my 300MB k8s limit value, it’s high.

So let’s go, developers have delivered the micro service updated. The same application with Nodejs 12. But this time can’t start. My Nodejs Kubernetes pod has been OOMKilled, due to memory limit exceed.

There isn’t precise description of how Nodejs determine its memory max, not in Node’s official documentation, nor on V8 one (Nodejs uses V8 engine to manage Javascript).

This defect 25576 give me a part of the answer:

We were delighted to hear that Node 12 would configure the limit based on available memory (as described in the blog post), so we told our users they should update to Node 12. But we weren’t aware this still meant there was a new static limit. And although the new limit is still around 50% more than before, it looks like Node 12 uses ~30% more memory than Node 10. So users with bigger projects started having memory problem just from moving to Node 12.

Filipe Silva – Roam Research developer, Angular Tools core member.

This explain why my container is OOMKilled at start. Nodejs 12 need more memory than Nodejs 10.

Nodejs memory areas

To understand why the memory limit value can be exceeded, you have to know some things about how Nodejs manage its memory.

nodejs-heap-areas

The memory consumed by a Nodejs application falls into one of three areas:

•Code
•Stack
•Heap

Code obviously is where your code is stored. Stack is where the function call stack is stored. Local variables are allocated here. Heap is what I’m talking about: that’s where dynamic allocations are stored. Here you find the strings, objects, and arrays that you work with in your JavaScript code.The Nodejs heap is composed of two sections:

•New space
•Old space

This is because allocations are divided into two generations. New allocations happen in new space, also known as younger generation. This is a small amount of memory from 1 to 8 megabytes. It’s very fast to allocate and to reclaim space. Garbage collection here happens frequently.

Allocations that survive garbage collection in new space are promoted to old space or older generation. Old space is fast to allocate, but very slow to reclaim space. Garbage collection here is slow and happens infrequently.

Objects which have survived two minor garbage collections are promoted to old-space. Old-space is garbage collected during a mark-sweep or mark-compact (major garbage collection cycle), which is much less frequent. A major garbage collection cycle is triggered when we have promoted a certain amount of memory to old space. This threshold shifts over time depending on the size of old space and the behavior of the program.

To understand more precisely I recommend to read this post.

Nodejs 12 memory management

The good news is that Nodejs is an open source project, so we can clone the repository and looking for heap memory management.

I’ve read on node’s documentation that a parameter exists to define the heap old space size. If defined the max_old_space_size parameter, will overwrite the default max heap value:

https://github.com/nodejs/node/blob/v12.x/deps/v8/src/heap/heap.cc
…
// Initialize max_old_generation_size_ and max_global_memory_.
{
…
if (FLAG_max_old_space_size > 0) {
max_old_generation_size_ =
static_cast(FLAG_max_old_space_size) * MB;
} else if (FLAG_max_heap_size > 0) {
size_t max_heap_size = static_cast(FLAG_max_heap_size) * MB;
…
}
…

Nodejs take care of constrained memory using cgroups, and if it can’t, default values will be used as Nodejs 10 do :

https://github.com/nodejs/node/blob/v12.x/src/api/environment.cc
…
void SetIsolateCreateParamsForNode(Isolate::CreateParams* params) {
const uint64_t constrained_memory = uv_get_constrained_memory();
const uint64_t total_memory = constrained_memory > 0 ?
std::min(uv_get_total_memory(), constrained_memory) :
uv_get_total_memory();
if (total_memory > 0) {
// V8 defaults to 700MB or 1.4GB on 32 and 64 bit platforms respectively.
// This default is based on browser use-cases. Tell V8 to configure the
// heap based on the actual physical memory.
params->constraints.ConfigureDefaults(total_memory, 0);
}
params->embedder_wrapper_object_index = BaseObject::InternalFields::kSlot;
params->embedder_wrapper_type_index = std::numeric_limits::max();
}
…
https://github.com/nodejs/node/blob/v12.x/deps/uv/src/unix/linux-core.c
…
uint64_t uv_get_constrained_memory(void) {
/*
This might return 0 if there was a problem getting the memory limit from
cgroups. This is OK because a return value of 0 signifies that the memory
limit is unknown.
*/
return uv__read_cgroups_uint64("memory", "memory.limit_in_bytes");
}
…

Monitor with high Kubernetes memory limit value

I Set the Nodejs Kubernetes memory request = limit = 800MB, and the half for Nodejs max_old_space_size=400MB, because usually the container consumes 400 MB. This time Nodejs start like a charme! However I don’t see a big old memory value, around 60MB only. But the container show me a high memory committed on k8s. This is the reason why I were OOMKilled at start.

But If the old size is around 60MB, why my container uses a huge memory spike at start ?

The response is : C++. As you know, Nodejs use the V8 engine to manage javascript, and the V8 engine is coded with C++. C++ heap memory consumption consists primarily of zone memory (temporary memory regions used by V8 for a short period of time). Since zone memory is used most extensively by the V8 parser and compilers, the spikes correspond to parsing and compilation events.

A well-behaved execution consists only of spikes, indicating that memory is freed as soon as it is no longer needed.(To visualize it use NodeJS –trace-zone-stats option)

V8-engine-memory

So, this is the reason why the container need more memory at start, and it depends of the application code, more libraries and dataset are imported, and more memory is needed to parse and compile.

Nodejs memory with high loads variations

Now I set the max_old_space_size=150MB (the old space memory use 60 MB without client request, so that should be a good start) and a small k8s request memory value of 200MB.

memory-heap-total

At the beginning, when the request rate growing up the old memory space is less than 150MB, and we could see the garbage collector efficiency. But when the traffic comes suddenly, the old space exceeds the max_old_space_size: that is due to GC strategy. As we have seen previously a GC threshold is calculated and shifts over time depending on the size of old space and the behavior of the program, this is a kind of machine learning. In this case, Nodejs can’t stay under the 150MB, so the application try to do the job, the GC keep on going but the memory consumption will be more than the expected one. This is what a java virtual machine can’t do, as seen in a previous post regarding java memory management in Kubernetes.

CPU is important

CPU limit here is very important. Without enough CPU time, Nodejs will stop working with SIGTERM signal, because it haven’t be able to anticipate and committed quickly enough its memory.

During high load, the GC will try to stay under max old size, so under k8s request memory, and the CPU activity will grow up. It’s a good thing regarding autoscaler threshold.

This test permit to see, that I need around 1500 mCore to absorb my scenario high load traffic during a k8s scale up phase.

Nodejs think first application response time, and not memory used. To enable optimizations which favor memory size over execution speed the V8 option –optimize-for-size permit it. (V8 options are listed here)

Nodejs and Kubernetes its real friends

To conclude, Nodejs isn’t so simple to implement in an Kubernetes environment. To prevent containers OOMKill, we have to test the application in different situations, and size it to take care of production incident’ spikes, that happened in real life, sizing the application, and the container. My configuration can now handle high load traffic variations, scaling pods, and exceed memory and CPU request value to benefit of the k8s shared infrastructure, staying under limit value.

On my Openshift environment I paid for the request value, so it’s a real advantage to stay on a low memory consumption most of the time. And access more resources occasionally, at start and during brief high loads.

Thanks for reading 🙂


0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *

Translate »