image by: https://undraw.co

You have been assigned to create a form for accepting some new metadata, in the Angular application that you had built earlier. You have designed a beautiful form with some great styling and components using bootstrap, primeNG or may be the new bulma CSS.

One particular thing you were really proud of was, subscribing to an observable from a service and handling things reactively. So you had a nice observable that you have been happily subscribing to and handling the onSuccess and onError gracefully.

One sunny day, your boss came to you and asked you to auto save the forms, as users are filling it. You like the challenge and feel upto it. You found on stack overflow that you can subscribe to the valueChanges on the angular form. Good for you. You quickly copied the code and built something like the below, and called the same save method you used earlier. You noticed all the weird things around debounce, switchMap, pipe, and so on.

this.beautifulForm.valueChanges
  .pipe(
    debounceTime(1000),
    switchMap(formData => this.save(formData)),
    takeUntil(this.unsubscribe)
  )
  .subscribe(() => this.done());

Aha !! the compilation error wouldn’t go away, any which way you try to arrange subscribe, map, observable. The save method you had earlier returns void. It probably was just eating into the response of your beautifulService and doing it’s stuff. Ok, You realised that save needs to return an Observable. But how do you do that? At the back of your mind, something is saying Pipe, Pipe, Pipe... You went to your good buddy google and asked “observable from subscribe, may be with a pipe”. Hours later, you were more confused than before. You copied some stack overflow answers, and thought let me read about Pipes a little bit.

You figured a way to make this happen. You are now returning a new observable using the “of” from your original methods, and piped on the original result doing something like below. But, but.. How do you catch the errors in pipe? Go back to searching again, Oompa !! map and catchError to the rescue. Finally you manage it, here is how it looked. So much for just a few lines of code.

return result.pipe(
  map(() => this.onSaveSuccess()),
  catchError(() => this.onSaveError())
);protected onSaveSuccess(): Observable<string> {
  this.isSaving = false;
  return of('done');
}

protected onSaveError(): Observable<string> {
  this.messageService.showError('blah blah..');
  return of('error');
}

You fired up your browser and tested it, only to realise the valueChanges have not fired unless you hit “enter” on the input fields. Bummer. Your navigator (google) was there to help again, who said that @ AngularExpert wrote an answer for similar question on stack overflow. You did find a way to notice that form fields support “ updateOn: ‘blur’ ” option, made sense. Although you probably ain’t got time for what other values are possible for updateOn. Your beautiful form now looks like this.

beautifulForm = this.fb.group({
    fullName: ['', { validators: [Validators.required], updateOn: 'blur' }]
});

You tabbed away on your form fields.. and everything seemed to be fine. You noticed that if the field requires longer input, and would require your users to spend longer time to fill, just like this medium blog (text area?), the save will not happen until they move out of that field. This could mean they could potentially loose that data. You think, what sane person would write a long paragraph on a web browser. You push it under the carpet.

So what is the moral of the story?

Save yourself some time, try to program “deliberately”.

The story line remains the same regardless of what you are trying to do. As you gain more experience, you get better at these things. You become exceptional at writing code accidentally, possibly much much faster over time. You find a way to know which google link could have an answer for your question, and which stack overflow response would most likely work. You look at the votes, tick marks, comments, and you build a pattern. Just don’t leak your secret sauce to other developers and to stack overflow, will you?

No body really has got that much time to know everything ahead of time. There are hundreds of programming languages, millions of libraries/packages, and “I don’t know the magnitude” concepts, techniques out there. You “invest the time” when something comes up. Good luck.

At Logistimo, all of our applications are containerized and run as docker containers inside Kubernetes. We had noticed lot of restarts on containers with Java apps and is quite random. Docker inspection revealed that the pod was killed by OOMKiller code:137. This meant that the application is consuming more memory than allocated to the container. It didn’t sound right, since we have limits on the Java application using -Xmx and we left about 20% buffer as Kubernetes resource limit (docker container) for the Meta space and GC data.

For e.g., 2 GB for Java process, and 2.4 GB for the Kubernetes resource.

Subsequent sections cover this problem and how to solve it in detail.

JVM memory usage

Obviously the first step was to review why the container is exceeding the said limit, clearly these are sufficiently buffered.

“ps” command confirms that the Xmx is indeed in place, and is set to max of 4GB.

Java process set to Max 4G

“top” command, however reveals that the physical memory used is 4.5 GB.

Why would Java take 500 MB more than allocated?

Since JDK 1.8.40 there is a Native memory tracker tool introduced which provides a detailed breakup of memory used by Java application with every byte accounted for. Note that NMT tool shows committed, resident might be less.

Actual usage = Heap memory + Meta space + Off heap

Off heap typically consists of class meta data, compiled code, threads and GC data. GC data is variable while rest of it should remain static for most applications. This memory is native (yes including the meta space), and JVM uses available memory on host to grow or garbage collect this data.

I would encourage you to read this excellent blog post by Mikhail to get better perspective.

Coming back to the problem at hand, JVM took 500 MB more because the underlying host had 16 GB memoy. At times this number could go higher than the buffer we had set, which would cause the container to be terminated. Shouldn’t the JVM be reading docker container’s memory limit?

Containers and Java

It turns out Java versions 9 and below do not understand containers/dockers at all (by default). It picks up available CPUs and Memory from the underlying host. Each Java app running on a host inside container relies on the host configuration. Considering that we are Kubernetes and many pods run on single node, this could lead to problems like the ones we are facing.

Java 10 supports containers out of the box, and it will lookup the linux cgroup information. This allows the JVM to garbage collect based on containers limits. This is turned on by default using the flag.

-XX:+UseContainerSupport

Thankfully, some of these features have been backported to 8u131 and 9 onwards. They can be turned on using the following flags.

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

Summary

Older versions of Java read the underlying host, and don’t understand cgroups. This would cause mismatch between container configuration and the Java process. This mismatch is on both CPU and the memory. Java has an Off heap memory component which has a dynamic GC data component, which could grow. Best way to solve this is to use the container support features available in recent versions of Java. Do not rely on buffering (It is waste of money).

Upgrade to Java 8u131+ or Java 9, if you have to remain on these major versions and turn on the experimental flags. Even better, if you can get all the container love available Java 10 onwards.

References

https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/