Does the Unix kernel version impact the JVM ClassLoader?
A practical experience from a production outage on how Java ClassLoaders behave when introducing conflicts
What exactly happened?
I have a java application that reads and writes the mongo database using the spring-boot framework and mongo client libraries. I composed the application image using docker and deployed them in three Kubernetes clusters. I noticed that the application fails to start during runtime in one of the Kubernetes clusters while the other looks fine. All three environments carry the same setup, except the kernel version of the node in the failed cluster (Linux 4.18.0–348.20.1.el8_5.x86_64 #1) differs from the other two (Linux 4.15.0–175-generic #184~16.04.1-Ubuntu).
Before apprehending the relationship between the JVM classloader and the Unix kernel, I would like to provide an overview of Java Classloaders, a description of the exception I faced, and the troubleshooting process I did to resolve the issue.
What is JVM ClassLoader?
Class Loaders are part of JRE to load classes dynamically during runtime. They are responsible for loading the java classes only when an application requires them. In Java8, there are three different class loaders:
- Bootstrap: The primordial classloader that loads classes from the rt.jar
- Extension: A child of the Bootstrap class loader that loads files from the “jre/lib/ext” directory or any different directory mentioned by the system property “java.ext.dirs”
- System: A child of the Extension class loader that loads the Application classes found in the value of the environment variable CLASSPATH, -classpath, or -cp
The Classloaders follow the Delegation hierarchy model in which Bootstrap ClassLoader has the higher priority, followed by Extension ClassLoader and then Application ClassLoader. Whenever there is a request from the application to load a class, the application loader delegates the request to the Extension class, which in turn delegates it to the Bootstrap loader. If the Bootstrap loader finds the java class in rt.jar, it will load; otherwise, it forwards the request to the Extension loader. If the Extension loader finds the java class in the external library directory, it loads; otherwise, the request will reach the System class loader. Once the system loader finds the class in all the application libraries in the classpath, the java class gets loaded. Otherwise, the loader returns a NoClassDefFoundError or ClassNotFoundException.
During this process, a class loaded by a parent ClassLoader is visible to the child ClassLoaders, but a class loaded by a child ClassLoader is not visible to the parent ClassLoaders. The loaders also ensure that the child classloaders do not load the classes loaded by parent classloaders. If the parent class loader can’t find the class, only then would the current instance attempt to do so itself.
Ok, now, What is the exception I faced?
Caused by: java.lang.NoSuchMethodError: com.mongodb.ConnectionString.getCredential()Lcom/mongodb/MongoCredential;
at com.mongodb.MongoClientURI.getCredentials(MongoClientURI.java:300)
at com.mongodb.Mongo.createCluster(Mongo.java:706)
at com.mongodb.Mongo.<init>(Mongo.java:315)
at com.mongodb.Mongo.<init>(Mongo.java:311)
at com.mongodb.MongoClient.<init>(MongoClient.java:337)
at org.springframework.data.mongodb.core.SimpleMongoDbFactory.<init>(SimpleMongoDbFactory.java:119)
java.lang.NoSuchMethodError :
The error occurs only during runtime, where the compilation is successful. When attempting to run the application, the JVM system classloader may not find the method present in the class. In my case, I have the same java class present in two different maven dependencies used by the application. I have two separate mongo clients used in the application; one uses the spring boot MongoTemplate, while the other uses the plain vanilla MongoClient from the Mongo java driver (Considering the load, the vanilla client performs better than the spring data wrapper, especially during serialization and deserialization process).
The two versions of the mongo client jar, one used by the spring-mongodb dependency and the other used by the standalone client, as mentioned in the diagram, caused this issue when starting the Spring boot application. The class com.mongodb.ConnectionString in the mongo-java-driver:3.6.3 jar has the method getCredential(), while it is missing in the same class in the mongodb-driver-core:3.4.2 jar.
What are Dependency conflicts and their implications?
Maven uses the “nearest definition” technique to resolve the version of an artifact when the POM XML has multiple versions of the dependencies. The “nearest definition” means that the version used will be the closest to the project in the tree of dependencies. If two dependency versions are at the same depth, Maven uses “first declaration,” meaning the choice will be the earliest dependency in the POM file.
Conflicts arise from dependencies when two versions of the same artifact are present in the dependency hierarchy and when the same class ( package.class ) is in the two or many dependencies. The latter is implicit and is the cause behind the issue I faced. The conflicts become problematic only when the class loaded during the runtime doesn’t have the method available in the classpath during compilation.
One can encounter the dependency tree using the below command.
mvn dependency:tree -Dverbose -Dincludes=$groupId:$artifactId
What did I do to resolve the issue?
I excluded the jar holding the conflicting class and explicitly added the mongo client dependency in the POM file. The ClassLoader picks the earliest dependency when dynamically loading the class during runtime, ultimately resolving the issue.
I appreciate the readers’ learning about whether the kernel version influences the Java ClassLoaders. While I am still trying to understand why the same application is running successfully in the other two clusters, the exception provides an opportunity to understand more about Java ClassLoaders. Feel free to add your comments.
