Defensive development: Gradle plugin development for busy engineers

This post is a direct follow-up to Gradle all the way down: Testing your Gradle plugin with Gradle TestKit. You don't have to read that, but I will make no effort to explain anything here that was already explained there so, you know. You were warned.

Let's assume you're busy

Since the whole premise of this post is that we're all too busy for class loader shenanigans (except me, since I'm paid to think about them, woohoo), let's cut to the chase and see the tl;dr:

If your plugin depends on the Android Gradle Plugin (AGP) (or any third-party ecosystem plugin?), you should strongly consider declaring it as compileOnly.

This is my advice for the following very simple reason:

You cannot know what your users will do, so you should assume they will do anything and everything.

Or, perhaps less opaquely:

For a variety of reasons outside of the scope of this post, AGP is critical infrastructure for a build. It is not a normal dependency that should be allowed to go through normal dependency resolution. Imagine if a plugin could silently change the version of Gradle your build ran against. It's (almost) that bad! ¹

While I think the above explanation is sufficient justification for the compileOnly advice, I can also justify it with practical concerns. The remainder of this post is a series of explorations with this idea in mind.

All of the code below (and more!) can be found on Github.

autonomousapps / blog-gradle-all-the-way-down

Companion source for "Gradle all the way down" blog post

The companion code for Gradle all the way down and Defensive development: Gradle plugin development for busy engineers

View on GitHub

A test harness

Talk is cheap, but assertions are worth their weight in gold. As you may recall from the first post in this series, Spock is my favorite JVM testing framework. Here's the first iteration of a spec for exploring our problem space:

class AndroidSpec extends Specification {

  @AutoCleanup
  AbstractProject project

  def "gets the expected version of AGP on the classpath (#gradleVersion AGP #agpVersion)"() {
    given: 'An Android Gradle project'
    project = new AndroidProject(agpVersion)

    when: 'We check the version of AGP on the classpath'
    def result = Builder.build(
      gradleVersion,
      project, 
      'lib:which', '-e', 'android'
    )

    // The output will contain a line like this:
    // jar for 'android': file:/path/to/gradle-all-the-way-down/plugin/build/tmp/functionalTest/work/.gradle-test-kit/caches/jars-9/f19c6db5e8f27caa4113e88608762369/gradle-4.2.2.jar
    then: 'It matches what the project provides, not what the plugin compiles against'
    def androidJar = result.output.split('\n').find {
      it.startsWith("jar for 'android'")
    }
    assertThat(androidJar).endsWith("gradle-${agpVersion}.jar")

    where:
    [gradleVersion, agpVersion] << gradleAgpCombinations()
  }
}

Before I take this apart for you, let's start with understanding the flow at a high level:

1. We create an Android project for testing against.
2. We run a task named which with the option -e android.
3. We assert that the jar we find via step 2 has the correct version information.
4. We run this whole thing against a matrix of Gradle and AGP versions, because we're thorough.

Here's what that spec looks like when run from the IDE:

or from CLI:

$ ./gradlew plugin:functionalTest --tests AndroidSpec
> Task :plugin:functionalTest
Running test: Test gets the expected version of AGP on the classpath (Gradle 7.3.3 AGP 4.2.2)(mutual.aid.AndroidSpec)
Running test: Test gets the expected version of AGP on the classpath (Gradle 7.4.1 AGP 4.2.2)(mutual.aid.AndroidSpec)
Running test: Test gets the expected version of AGP on the classpath (Gradle 7.3.3 AGP 7.1.2)(mutual.aid.AndroidSpec)
Running test: Test gets the expected version of AGP on the classpath (Gradle 7.4.1 AGP 7.1.2)(mutual.aid.AndroidSpec)

So that's the power of Spock. It's really easy to generate data pipelines for parameterized tests. If you're not actively afraid of this capability by the end of this post, you're not reading closely enough.

Data-driven testing

Since the data-driven aspect is so crucial to understanding the concepts we're exploring, I want to show its implementation:

final class Combinations {
  static List<List> gradleAgpCombinations(
    List<Object>... others = []
  ) {
    return [
      gradleVersions(), agpVersions(), *others
    ].combinations()
  }

  static List<GradleVersion> gradleVersions() {
    return [
      GradleVersion.version('7.3.3'),
      GradleVersion.version('7.4.1')
    ]
  }

  static List<String> agpVersions() {
    return ['4.2.2', '7.1.2']
  }
}

I know I promised in my last post that there wouldn't be any more Groovy, but the combinations() function is simply too good to ignore. Not only do I not want to have to implement it myself, I don't see why I would want to import another library to do it when the Groovy GDK is already available (which it always will be with Gradle projects).

Which AGP?

To programmatically inspect which version of AGP is on the runtime classpath of our build, I wrote a little helper script that registers a task, which, that our spec invokes. Here's how that is defined:

// which.gradle
tasks.register('which', WhichTask)

@UntrackedTask(because = 'Not worth tracking')
abstract class WhichTask extends DefaultTask {

  WhichTask() {
    group = 'Help'
    description = 'Print path to jar providing extension, or list of all available extensions and their types'
  }

  @Optional
  @Option(option = 'e', description = 'Which extension?')
  @Input
  abstract String ext

  @TaskAction def action() {
    if (ext) printLocation()
    else printExtensions()
  }

  private void printLocation() {
    def jar = project.extensions.findByName(ext)
      ?.class
      ?.protectionDomain
      ?.codeSource
      ?.location

    if (jar) {
      logger.quiet("jar for '$ext': $jar")
    } else {
      logger.quiet("No extension named '$ext' registered on project.")
    }
  }

  private void printExtensions() {
    logger.quiet('Available extensions:')
    project.extensions.extensionsSchema.elements.sort { it.name }.each {
      // fullyQualifiedName since Gradle 7.4
      logger.quiet("* ${it.name}, ${it.publicType.fullyQualifiedName}")
    }
  }
}

This task can be run in two modes:

1. ./gradlew which -e <some-extension> or
2. ./gradlew which

The first will print the path to the jar that provides the extension (such as "android"), while the second prints all of the extensions available for the given module, along with their fully-qualified types (this can be useful if you're trying to discover the types of extensions). I've been using this trick in the debugger for a long time.

Test scenarios

Let's elaborate. First, I added a flag to my plugin's build script to let me change how it is built, so that I could explore this behavior with automated tests. This is nothing something I would ever recommend in the general case—it's just for the assertions that drive the following exploratory scenarios.

// plugin/build.gradle
// -Dimpl (for 'implementation')
boolean impl = providers.systemProperty('impl').orNull != null

dependencies {
  if (impl) {
    implementation 'com.android.tools.build:gradle:7.2.0-beta04'
  } else {
    compileOnly 'com.android.tools.build:gradle:7.2.0-beta04'
  }
}

By default, we use compileOnly, but if you pass -Dimpl during a build, we'll use implementation instead. We must also update our test configuration, because we need that flag available in the test JVM (which is forked from the main JVM and doesn't get all of its system properties by default).

// plugin/build.gradle
testTask.configure {
  ...
  systemProperty('impl', impl)
  ...
}

With that small change, we can now elaborate on our spec.

Iteration 1: does it matter if the user declares AGP in the root buildscript block?

@Requires({ PreconditionContext it -> it.sys.impl == 'true' })
def "gets the expected version of AGP on the classpath for implementation (#gradleVersion AGP #agpVersion useBuildScript=#useBuildScript)"() {
  given: 'An Android Gradle project'
  project = new AndroidProject(agpVersion,useBuildScript)

  when: 'We check the version of AGP on the classpath'
  def result = Builder.build(
    gradleVersion,
    project,
    'lib:which', '-e', 'android'
  )

  then: 'Result depends'
  def androidJar = result.output.split('\n').find {
    it.startsWith("jar for 'android'") 
  }
  def expected = useBuildScript
    ? "gradle-${agpVersion}.jar"
    : 'gradle-7.2.0-beta04.jar'
  assertThat(androidJar).endsWith(expected)

  where: '2^3=8 combinations'
  [gradleVersion, agpVersion, useBuildScript] << gradleAgpCombinations([true, false])
}

In real terms, those scenarios map to these two Gradle build scripts:

// settings.gradle -- for BOTH versions of the build script
pluginManagement {
  repositories {
    gradlePluginPortal()
    google()
    mavenCentral()
  }
  plugins {
    // Centralized version declarations. These do not directly 
    // impact the classpath. Rather, this simply lets you have
    // a single place to declare all plugin versions.
    id 'com.android.library' version '7.1.2'
  }
}

// build.gradle 1
// useBuildScript = false
plugins {
  id 'com.android.library' apply false
}

// build.gradle 2
// useBuildScript = true
buildscript {
  repositories {
    google()
    mavenCentral()
  }
  dependencies {
    classpath "com.android.tools.build:gradle:$agpVersion"
  }
}

From the fact that our spec passed, we can confidently answer the question in the heading: yes, the result depends on whether the user declares AGP in the root project's buildscript block. For users unfamiliar with Android, please be aware this has been standard practice since time immemorial, and has only started changing with the latest template.

And of course this is already problematic: we have created a scenario where a plugin has managed to upgrade our build to a beta version of AGP.

Iteration 2: What happens if we turn off some TestKit magic?

There's a bit of magic in TestKit that puts your plugin-under-test on the build classpath so that you don't have to. It's very useful for simple scenarios, but I find it lacking for industrial-scale use-cases. To test these scenarios, first we must make an update to our build script:

// plugin/build.gradle
plugins {
  ...
  id 'maven-publish'
}

group = 'mutual.aid'
version = '1.0'

// Some specs rely on the plugin as an external artifact
// This task is added to the build by the maven-publish plugin
def publishToMavenLocal = tasks.named('publishToMavenLocal')

testTask.configure {
  ...
  dependsOn(publishToMavenLocal)
  ...
}

Now, whenever we run the functionalTest task, it will first publish our plugin to maven local (~/.m2/repositories).

Now make this change to Builder to let us vary this behavior:

private fun runner(
  gradleVersion: GradleVersion,
  projectDir: Path,
  withPluginClasspath: Boolean,
  vararg args: String
): GradleRunner = GradleRunner.create().apply {
  ...
  if (withPluginClasspath) {
    withPluginClasspath()
  }
  ...
}

And finally, our updated spec:

@Requires({ PreconditionContext it -> it.sys.impl == 'true' })
def "gets the expected version of AGP on the classpath for implementation (#gradleVersion AGP #agpVersion useBuildScript=#useBuildScript useMavenLocal=#useMavenLocal)"() {
  given: 'An Android Gradle project'
  project = new AndroidProject(
    agpVersion,
    useBuildScript,
    useMavenLocal
  )

  when: 'We check the version of AGP on the classpath'
  def result = Builder.build(
    gradleVersion,
    project,
    // !useMavenLocal => withPluginClasspath
    !useMavenLocal,
    'lib:which', '-e', 'android'
  )

  then: 'Result depends'
  def androidJar = result.output.split('\n').find {
    it.startsWith("jar for 'android'")
  }
  def expected
  if (useBuildScript || useMavenLocal) {
    expected = "gradle-${agpVersion}.jar"
  } else {
    expected = 'gradle-7.2.0-beta04.jar'
  }

  // Our assertion is growing more complicated
  assertThat(androidJar).endsWith(expected)

  where: '2^4=16 combinations'
  [gradleVersion, agpVersion, useBuildScript, useMavenLocal] << gradleAgpCombinations(
    // useBuildScript
    [true, false],
    // useMavenLocal
    [true, false],
  )
}

In real terms, those scenarios map to these four Gradle build scripts:

// settings.gradle now varies
pluginManagement {
  repositories {
    if (useMavenLocal) mavenLocal()
    gradlePluginPortal()
    google()
    mavenCentral()
  }
}

// build.gradle 1
// useBuildScript = true
// useMavenLocal = false
buildscript {
  repositories {
    google()
    mavenCentral()
  }
  dependencies {
    classpath "com.android.tools.build:gradle:$agpVersion"
  }
}

// build.gradle 2
// useBuildScript = true
// useMavenLocal = true
buildscript {
  repositories {
    mavenLocal()
    google()
    mavenCentral()
  }
  dependencies {
    classpath "com.android.tools.build:gradle:$agpVersion"
  }
}

// build.gradle 3
// useBuildScript = false
// useMavenLocal = true
plugins {
  id 'com.android.library' apply false
}

// build.gradle 4 (identical to 3, but recall settings.gradle
// varies)
// useBuildScript = false
// useMavenLocal = false
plugins {
  id 'com.android.library' apply false
}

Since our spec has passed, we know that yes, TestKit classpath magic influences the results of our build. Since TestKit is not in play, ever, in real builds, I prefer to not use the withPluginClasspath() method, and instead always rely on publishing my plugin to maven local, as it more closely mimics real builds.

Iteration 3: Does it matter if we use buildscript for our plugin-under-test?

@Requires({ PreconditionContext it -> it.sys.impl == 'true' })
def "gets the expected version of AGP on the classpath for implementation (#gradleVersion AGP #agpVersion useBuildScriptForAgp=#useBuildScriptForAgp useBuildScriptForPlugin=#useBuildScriptForPlugin useMavenLocal=#useMavenLocal)"() {
  given: 'An Android Gradle project'
  project = new AndroidProject(
    agpVersion,
    useBuildScriptForAgp,
    useBuildScriptForPlugin,
    useMavenLocal
  )

  when: 'We check the version of AGP on the classpath'
  def result = Builder.build(
    gradleVersion,
    project,
    !useMavenLocal,
    'lib:which', '-e', 'android'
  )

  then: 'Result depends'
  def androidJar = result.output.split('\n').find {
    it.startsWith("jar for 'android'")
  }
  def expected
  if (useBuildScriptForAgp && useBuildScriptForPlugin && useMavenLocal) {
    // our 'implementation' dependency has greater priority
    expected = 'gradle-7.2.0-beta04.jar'
  } else if (useBuildScriptForAgp || useMavenLocal) {
    // the project's requirements have greater priority
    expected = "gradle-${agpVersion}.jar"
  } else {
    // our 'implementation' dependency has greater priority
    expected = 'gradle-7.2.0-beta04.jar'
  }

  // Note that the assertion is... complicated.
  assertThat(androidJar).endsWith(expected)

  where: 'There is a truly atrocious combinatorial explosion'
  [gradleVersion, agpVersion, useBuildScriptForAgp, useBuildScriptForPlugin, useMavenLocal] <<
    gradleAgpCombinations(
      // useBuildScriptForAgp
      [true, false],
      // useBuildScriptForPlugin
      [true, false],
      // useMavenLocal
      [true, false],
    )
}

This time, rather than share all of the individual variations, I'll leave the if/else logic in place so you can use your imagination. Note that the following incorporates some pseudocode for improved readability.

// settings.gradle
pluginManagement {
  repositories {
    if (useMavenLocal) mavenLocal()
    gradlePluginPortal()
    google()
    mavenCentral()
  }
  plugins {
    id 'com.android.library' version '7.1.2'
    id 'mutual.aid.meaning-of-life' version '1.0'
  }
}

// build.gradle
if (useBuildScriptForAgp) {
  buildscript {
    repositories {
      if (useMavenLocal) mavenLocal()
      google()
      mavenCentral()
    }
    dependencies {
      classpath 'com.android.tools.build:gradle:4.2.2'
      if (useBuildScriptForPlugin) classpath 
  'mutual.aid.meaning-of-life:mutual.aid.meaning-of-life.gradle.plugin:1.0' 
    }
  }
} else {
  plugins {
    id 'com.android.library' apply false
  }
}

Aaaand I'm starting to run out of screen real estate for this spec.

We now have a combinatorial explosion 🎉 The version of AGP you'll end up with in a real build is now very hard to predict, unless you're an expert and understand how the class loader relationships work,² and how those relationships interact (or don't interact) with the dependency resolution engine.

Iteration 4: We can (and must) do better! Or, the null hypothesis.

Let's consider the following spec. At first glance, it might seem complicated, but note that the assertion is always the same. That is, no matter what combination of flags we pass (i.e., no matter what our user might decide to do), we always end up with the version of AGP the user specifies. This is the power of compileOnly.

@IgnoreIf({ PreconditionContext it -> it.sys.impl == 'true' })
def "gets the expected version of AGP on the classpath for compileOnly (#gradleVersion AGP #agpVersion useBuildScriptForAgp=#useBuildScriptForAgp useBuildScriptForPlugin=#useBuildScriptForPlugin useMavenLocal=#useMavenLocal)"() {
  given: 'An Android Gradle project'
  project = new AndroidProject(
    agpVersion,
    useBuildScriptForAgp,
    useBuildScriptForPlugin,
    useMavenLocal
  )

  when: 'We check the version of AGP on the classpath'
  def result = Builder.build(gradleVersion, project, 'lib:which', '-e', 'android')

  then: 'It matches what the project provides, not the plugin, always'
  def androidJar = result.output.split('\n').find {
    it.startsWith("jar for 'android'")
  }
  // Note that the assertion is always the same
  assertThat(androidJar).endsWith("gradle-${agpVersion}.jar")

  where:
  [gradleVersion, agpVersion, useBuildScriptForAgp, useBuildScriptForPlugin, useMavenLocal] <<
    gradleAgpCombinations(
      // useBuildScriptForAgp
      [true, false],
      // useBuildScriptForPlugin
      [true, false],
      // useMavenLocal
      [true, false],
    )
}

Q.E.D. ∎

Two roads diverged in a wood³

There you have it. Unless it's your day job to debug class loader and dependency resolution interactions, I would highly recommend you keep it simple, take the well-traveled road, and use compileOnly. No matter what, you'll get the correct result.

Bonus content

Since I don't like misleading people, I have to acknowledge that, even if you do the right thing and use compileOnly, some plugin you depend on might still use implementation and bring AGP onto your runtime classpath, whether you want it there or not. Early on in the process of migrating my large build to convention plugins, I ran into just this issue, and found myself in a scenario where two versions of AGP ended up available at runtime! 😱 To prevent that happening in the future, I added the following task to all my convention plugin projects, and run it in CI.

Here's the task implementation:

@UntrackedTask(because = "Not worth tracking")
abstract class NoAgpAtRuntime : DefaultTask() {

  @get:Internal
  lateinit var artifacts: ArtifactCollection

  @PathSensitive(PathSensitivity.RELATIVE)
  @InputFiles
  fun getResolvedArtifactResult(): FileCollection {
    return artifacts.artifactFiles
  }

  @TaskAction fun action() {
    val android = artifacts.artifacts
      .map { it.id.componentIdentifier.displayName }
      .filter { it.startsWith("com.android.tools") }
      .toSortedSet()

    if (android.isNotEmpty()) {
      val msg = buildString {
        appendLine("AGP must not be on the runtime classpath. The following AGP libs were discovered:")
        android.forEach { a ->
          appendLine("- $a")
        }
        appendLine(
          "The most likely culprit is `implementation 'com.android.tools.build:gradle'` in your dependencies block"
        )
      }
      throw GradleException(msg)
    }
  }
}

And here's how it's registered:

// register the task
def runtimeClasspath = project.configurations.findByName('runtimeClasspath')
if (runtimeClasspath) {
  def noAgpAtRuntime = tasks.register('noAgpAtRuntime', NoAgpAtRuntime) {
    artifacts = runtimeClasspath.incoming.artifacts
  }

  tasks.named('check').configure {
    dependsOn noAgpAtRuntime
  }
}

The solution, when this check fails, is to find the third-party plugin that's using implementation for AGP, and declare it as compileOnly as well.

Endnotes

¹ Gradle doesn't provide a way to model this. up
² It's always deterministic, but challenging for the non-expert to predict. up
³ The Road Not Taken, by Robert Frost. up