This is why we can't have nice things: When POM files lie

Photo by Marco Bicca on Unsplash. I really hope there's a light at the end of this sewer.

Sorry, my country is dissolving into a Nazi sewer of world-historical proportions and so I'm dealing with my otherwise fruitless rage by yelling about JVM things.

The fact that Java classes exist in a global namespace is not well-appreciated, even by vendors of major parts of the ecosystem (apparently).

Caused by: java.lang.ClassCastException: class com.google.common.graph.ImmutableGraph cannot be cast to class com.google.common.graph.SuccessorsFunction (com.google.common.graph.ImmutableGraph and com.google.common.graph.SuccessorsFunction are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @4617120)  

A confusing error at the best of times, but to a build engineer (i.e. me) trying to help out during a SEV, a despair-inducing example of how Dependency Management on the JVM is Completely Broken, What Are We Even Doing Here.

A ClassCastException always means Someone Somewhere Hates You, or at least values their own KPIs more than not polluting the entire goddamn ecosystem.

Let's debug! My first step is to navigate to the Graph class and check its hierarchy. I can confirm that, yes, in Guava 33.3.1-jre at least, that class does indeed implement the SuccessorsFunction interface. It should not be throwing a ClassCastException!

Next step. In Intellij (the vendor which is ironically the source of this amongst most of the rest of my woes), I set a breakpoint at the exception and evaluate the following expression.¹

// `this` is an instance of the `com.google.common.graph.Graph` class
this.javaClass.superclass.superclass.superclass.interfaces.first().protectionDomain.codeSource.location

That expression resolves to this:

file:/Users/<<ME!>>/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler/2.0.21/88f09afc2536e38d528e78eb8349504de10ac436/kotlin-compiler-2.0.21.jar

What the fuck! That is not the right jar!! Let's double-check:

jar tf path/to/kotlin-compiler-2.0.21.jar
…
com/google/common/graph/Graph.class

😭

Maybe it's a bug! Let's look at the latest version of the jar at time of writing, 2.2.10 (nope, same problem). Not only has Jetbrains, the inventor of the Kotlin language, released a library that is bundling Guava (among many other things!!) in a fatjar, they're not even bothering to relocate the damn packages. Ok, I'm curious, what does the pom.xml have to say for itself?

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.jetbrains.kotlin</groupId>
  <artifactId>kotlin-compiler</artifactId>
  <version>2.1.10</version>
  <name>Kotlin Compiler</name>
  <description>Kotlin Compiler</description>
  <url>https://kotlinlang.org/</url>
  <licenses>
    <license>
      <name>The Apache License, Version 2.0</name>
      <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
    </license>
  </licenses>
  <developers>
    <developer>
      <name>Kotlin Team</name>
      <organization>JetBrains</organization>
      <organizationUrl>https://www.jetbrains.com</organizationUrl>
    </developer>
  </developers>
  <scm>
    <connection>scm:git:https://github.com/JetBrains/kotlin.git</connection>
    <developerConnection>scm:git:https://github.com/JetBrains/kotlin.git</developerConnection>
    <url>https://github.com/JetBrains/kotlin</url>
  </scm>
  <dependencies>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-stdlib-jdk8</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-script-runtime</artifactId>
      <version>2.1.10</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-reflect</artifactId>
      <version>1.6.10</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>*</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.intellij.deps</groupId>
      <artifactId>trove4j</artifactId>
      <version>1.0.20200330</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlinx</groupId>
      <artifactId>kotlinx-coroutines-core-jvm</artifactId>
      <version>1.6.4</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>

There is no indication that this is a fatjar in the metadata, not to mention the hilarious fact that they are declaring a dependency on kotlin-reflect 1.6.10. But someone at some point noticed that this was Causing A Problem, so they excluded all of its dependencies. Are you kidding me. Guys, you publish a BOM, just use it!

It is entirely unclear why some dependencies are being declared and others are being silently bundled into the final artifact. Maybe I can get an answer at this issue I raised.

Why am I even using this dependency?

Well, I had been using kotlin-compiler-embeddable for source-code analysis,² which at least relocates its bundled classes. However, using that in the same project that has a dependency on the Kotlin Gradle Plugin leads to this build warning:

w: The artifact `org.jetbrains.kotlin:kotlin-compiler-embeddable` is present in the build classpath along Kotlin Gradle plugin. 
This may lead to unpredictable and inconsistent behavior.   
For more details, see: https://kotl.in/gradle/internal-compiler-symbols 

A search online for a suitable replacement for kotlin-compiler-embeddable turned up this on discuss.kotlinlang.org:

kotlin-compiler-embeddable should be used in scenarios when it’s necessary to have the compiler packaged as a single jar with no external dependencies. In all other cases, use the regular kotlin-compiler.

That answer implies that kotlin-compiler does have external dependencies (which is true), while also further implying that it abides by the standard JVM library contract by Actually Declaring all its dependencies (which is false).³

Coping

This raises the question, how can we cope with this? Well, assuming your use-case for this is in a Gradle task and your task is relatively well-written, you can migrate the task action to use the Worker API with classloader isolation. The link provided in the warning does a fair job of explaining how to do that, assuming you're ok with drawing the owl.

Ah, fuck it, let's draw the owl

Drawing the build script

// plugin/build.gradle.kts
…etc…

dependencies {
  implementation(project(":shared-lib")) {
    exclude(group = "org.jetbrains.kotlin", module = "kotlin-compiler")
  }
}

Jetbrains' docs suggest using compileOnly("org.jetbrains.kotlin:kotlin-compiler-embeddable:2.1.10"), which isn't wrong, but also won't work if that lib is actually coming from a transitive dependency.

Drawing the task

abstract class OwlTask @Inject constructor(
  private val workerExecutor: WorkerExecutor
) : DefaultTask() {

  @get:Classpath 
  abstract val kotlinCompiler: ConfigurableFileCollection

  @get:Input
  abstract val otherInput: Property<Boolean>

  @TaskAction fun action() {
    workerExecutor
      .classLoaderIsolation { spec ->
        // please note that from() adds to the
        // classpath, while setFrom() would
        // completely override it.
        spec.classpath.from(kotlinCompiler)
      }
      .submit(Action::class.java) { params ->
        params.otherInput.set(otherInput)
      }
  }

  interface Parameters : WorkParameters {
    val otherInput: Property<Boolean>
  }

  abstract class Action : WorkAction<Parameters> {
    override fun execute() {
      val otherInput = parameters.otherInput.get()
      …
    }
  }
}

Drawing the plugin

abstract class OwlPlugin : Plugin<Project> {
  override fun apply(target: Project) {
    // Use the project's version of Kotlin, if present. Else default to what we use.
    val kotlinVersion = …some method for getting the version of Kotlin you want…
    val dependencyScope = project.configurations.dependencyScope("owl").get()
    val resolvable = project.configurations.resolvable("owlClasspath") {
      extendsFrom(dependencyScope)
    }

    project.dependencies.add(dependencyScope.name, "org.jetbrains.kotlin:kotlin-compiler:$kotlinVersion")

    project.tasks.register("owl", OwlTask::class.java) { t ->
      t.kotlinCompiler.setFrom(resolvable)
      t.otherInput(true) // demonstration purposes only
    }
  }
}

Hello, owl!

I can confirm this works, it eliminated the ClassCastException. Now, a day and incredible amounts of frustration later, I guess I can go back to my original problem?

I'm just some asshole

Look, I'm just some asshole who always tries to do the right thing. But what the fuck am I supposed to do when larger and better-resourced teams just abdicate responsibility? Fuck! I guess I'll write a blog post!

Special thanks

I am much obliged Luis Cortes, who convinced me that some of my salty language was in fact acidic. All the remaining salt and/or acid is entirely of my own devising.

Endnotes

¹ Guava's Graph class has a complicated hierarchy! up

² This dependency exhibits many of the same metadata problems. up

³ To be fair, this is a very widespread problem. up