Managing Java dependencies for Mendix modules | CLEVR

If you have ever tried to implement a certain function as a reusable module in Mendix, there’s a good chance you’ve had to add some jar dependencies. There are multiple ways of achieving that goal. But there’s one method that I find usually works best, because it simultaneously addresses multiple issues.

But first, let me rank the game-breaking or simply annoying issues I have encountered when trying to implement a reusable module:

1. If you want to use a jar file in your module, e.g. pdfbox v2.3, but another version of this jar e.g. v1.8 is already being used in a another module of your application, you are out of luck. As far as I know, this is simply not possible. The class loader will pick up only one version, leaving one of the two modules to deal with the wrong version, which usually results in the application hanging or crashing. To make matters worse, this only occurs when invoking a specific java action with a conflicting dependency at runtime.
2. If you are maintaining a Mendix module and you need to update a jar file within it, you need to make sure everyone deletes the old jar on update. Otherwise, you end up having two jars with a different version in your classloader path, which again likely leads to the application not working. 
3. Mendix requires you to manage transitive dependencies manually. This usually means you have to run the code to see which classes are missing, then find the right jar file and add it to the project manually. Then, rinse and repeat until there are no more “NoClassFound” errors.
4. Wasting time when exporting modules is my next pet peeve. There are too many jars to include/exclude, especially if you are using something like community commons or the rest module in your project. Of course, that could be solved simply by a select/deselect button in the export dependencies dialog, but that is not the point of this post. One way or another, time is lost.

One way to effectively deal with all of these issues is to employ a build tool to produce a so-called fat jar, a single jar file that contains all your dependencies. This on its own resolves multiple issues from the previous list, including: rom the fat jar file will be updated automatically when the module is updated, thus solving point 2; the build tool can take care of all the transitive dependencies, thus solving point 3, and finally the only dependency is a single one-jar file, eliminating point 4.

The first issue we mentioned can be resolved using a technique called shadowing. Shadowing replaces patterns in class names with a given string. For example, you can replace org.json with community.commons.org.json. This lets the Java classloader load two versions of the org.json library because they have different class names.

Case study: community commons

What better module to demonstrate the techniques described above than the community commons? In its current status, it has some 20 or so dependencies (check out that scroll bar).

In this instance, I will be using Gradle. But you can opt to do the same thing using other tools e.g. Maven or JarJarLinks.

Adding dependencies to Gradle

First, I installed the Gradle Eclipse plugin after downloading it from the Eclipse marketplace. Next, I created a new Gradle project. The main file in every gradle project is the gradle build script, where many options can be specified, such as which Java version to use. The dependencies for a Gradle project are also defined in the build script. As you can see below, my first iteration of the build script is mostly standard stuff with the exception of the shadowing tool that I added:

buildscript {
repositories {
mavenCentral()//look for dependencies here
}
dependencies {
classpath "com.github.jengelman.gradle.plugins:shadow:2.0.0"
//this is the tool we will use to build the fat jar and shadow it
}
}

plugins {
id 'com.github.johnrengelman.shadow' version '2.0.0'
id 'java'
}

group 'com.mendix.community-commons'
version '1.0.0'

apply plugin: 'java'
apply plugin: 'maven'
apply plugin: 'eclipse'

task wrapper(type: Wrapper) {
gradleVersion = '3.0'
}

compileJava {
sourceCompatibility = 1.8
targetCompatibility = 1.8
}

repositories {
mavenLocal()
mavenCentral()
}

dependencies {
}

So far so good. Next, I started adding the dependencies from the community commons module:

dependencies {
// https://mvnrepository.com/artifact/org.owasp.antisamy/antisamy
compile group: 'org.owasp.antisamy', name: 'antisamy', version: '1.5.3'

    // https://mvnrepository.com/artifact/com.google.guava/guava
compile group: 'com.google.guava', name: 'guava', version: '14.0.1'

    // https://mvnrepository.com/artifact/commons-codec/commons-codec
compile group: 'commons-codec', name: 'commons-codec', version: '1.10'

    // https://mvnrepository.com/artifact/org.apache.pdfbox/jempbox
compile group: 'org.apache.pdfbox', name: 'jempbox', version: '1.8.5'

    // https://mvnrepository.com/artifact/joda-time/joda-time
compile group: 'joda-time', name: 'joda-time', version: '2.9.6'

    // https://mvnrepository.com/artifact/commons-fileupload/commons-fileupload
compile group: 'commons-fileupload', name: 'commons-fileupload', version: '1.2.1'

    // https://mvnrepository.com/artifact/commons-io/commons-io
compile group: 'commons-io', name: 'commons-io', version: '2.3'

    // https://mvnrepository.com/artifact/org.apache.commons/commons-lang3
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.0'

    compile group: 'org.apache.servicemix.bundles', name: 'org.apache.servicemix.bundles.batik', version: '1.8_1'

    // https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox
compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.3'

    // https://mvnrepository.com/artifact/xerces/xercesImpl
compile group: 'xerces', name: 'xercesImpl', version: '2.8.1'
}

I noticed that some of the dependencies are not listed in Maven Central (or at least I could not find them). No problem—I have the jar files from the community commons project on GitHub. I created a folder libs in my Gradle project, and then added the com.springsource.org.apache.batik.css-1.7.0.jar and nekohtml.jar to it. Then, I added the following line to my dependencies, which, as you might expect, adds all jar files from the libs folder to the gradle project.

compile fileTree(dir: 'libs', include: '*.jar')

Doing this, we have now resolved all Java dependencies.

Dealing with Mendix classes

My basic premise is to use the Java action to only call a corresponding function from the fat jar that I create. This is already the case for most Java actions in community commons, i.e. the actual implementation code is not inside the java action class. Instead, execution is delegatad to another class. I copied the classes from the communitycommons package where the logic is implemented, specifically ConversationLog, DateTime, ORM, StringUtils, etc., to my Gradle project. I built the project and refreshed the build path in Eclipse. Success ... at least somewhat. The dependencies are loaded and recognized by Eclipse, but I can see some missing classes. Most of the missing classes are from the Mendix API, which is used extensively in the community commons in the form of IContext, IMendixObject, Core, etc. I added them to the dependencies as well.

compile files('C:/Program Files/Mendix/7.2.0/runtime/bundles/com.mendix.public-api.jar')
compile files('C:/Program Files/Mendix/7.2.0/runtime/bundles/com.mendix.logging-api.jar')

I need to include these classes when developing in Eclipse because the compiler has to recognize them. Otherwise, the project will not compile. However, I do not want the Mendix API classes to go into my fat jar, so I excluded them.

shadowJar {
dependencies {
exclude 'com/mendix/**'
}
}

Rebuild and refresh, a lot of unrecognized classes are now ok. Here is a screenshot of my gradle project at this point:

But I can still see a few imports which are unrecognized. These were enumerations and proxy classes from the Mendix Modeler, such as system.proxies.FileDocument. I could not think of a way to add them to my list of dependencies, so I decided to wrap them. I declared an interface IFileDocument with the following code:

public interface IFileDocument {
public IMendixObject getMendixObject();
public boolean getHasContents();
}

I will only work with this interface within the fat jar. Then, I added a method to convert a system.proxies.FileDocument to an org.community-commons.main.IFileDocument. Every time a Java action needs to call some method from my fat jar file that works with files, I do a conversion, and then pass the interface.

That just leaves the enumerations and the system.proxies.Language. For the enumerations e.g. communitycommons.proxies.LogLevel, I used a similar wrapping method to the one I employed for the FileDocument. For the Language I decided to just keep the whole code (all four lines of it) in the Java action. It has no other external dependencies, so why bother? If needed, though, it can be wrapped in a similar way. I did another rebuild and added the fat jar to the project. After everything compiles, these are the dependencies that remain in the modeler.

Let me make a small digression here to talk about wrapping Mendix proxy classes. This solution is clearly not what you would normally want to do. But on the other hand, it is mostly mechanical work. The way I see things, this means there is some way to automate it. A tool can be developed that generates code for the wrapper classes. You could even go a step further and have the tool replace all usages of proxy classes with the corresponding wrapper classes.

Adding resources to a fat jar

As you can see in the screenshot, we managed to get rid of most files, with the exception of the antisamy XML files. I could just leave them as they are and everything would work fine, but they are really bugging me. For the sake of being thorough, I will demonstrate how any resource files can be included in the fat jar. But this is clearly optional.

As you probably already know, a jar file is just a zip file, which means we can include any file we want in it. In fact, Gradle automatically considers any files that are in a folder named resources as resource files and adds them to the jar. I created such a folder under src/main and copied all the XML files there. To test that the resources are really added, I did a rebuild and then extracted the jar file. I could see that all XML files are really included in the fat jar.

Next, I had to change the way the xml files are read. Previously they were loaded from the resources folder just like regular files. However, this is not possible once they are packaged in a jar file. Instead, you should use the classloader to load them as resources. I changed the following lines in StringUtils.XSSSanitize (which is part of the community commons):

String filename = Core.getConfiguration().getResourcesPath() + File.separator
+ "communitycommons" + File.separator + "antisamy"
+ File.separator + "antisamy-" + policyString + "-1.4.4.xml";
Policy p = Policy.getInstance(filename);

Instead of using a filename, I use the classloader to get the file contents as a stream:

Finally, we are left with only three dependencies:

• java that contains classes to interface with the fat jar,
• the fat jar file, and
• txt - a license file.

Because I do not see a way to reduce the number of dependencies any further, we can now move on to the next topic:

Shadowing

To keep it simple, I will just prepend the cc_ prefix to all external classes with the following code: