Configuration roles and the blogging-industrial complex

On the perils of getting what you ask for
Or, building cool stuff and making everyone think you're a wizard

Gradle 8.4 has released an exciting new feature, an API for easily creating role-focused Configurations. For those who have been around the block a few times, it is well-known that the Configuration interface is amongst the most confusing in Gradle lore, not to mention having a very overloaded name. It also has many responsibilities and a massive memory footprint.

The new API is meant as a step towards disambiguation. The new methods on ConfigurationContainer (aka configurations) are designed to simplify the creation of immutable, purpose-built Configuration instances. They do that. But they also expose complexity that was previously obscured by the prior legacy situation. On the plus side, this additional complexity is a boon to bloggers and build engineers eager to justify their (my) existence.

All of the code used in this post is available on Github.

First a warning

Most Gradle projects won’t need this kind of setup, and most Gradle plugins won’t require this elaboration of responsibilities in the Configurations they create. The primary use-case is for ecosystem plugins (aka “core” plugins), but what we’ll do here is also fairly common. Even if you don’t need this yourself, understanding Configurations more deeply can be very valuable.

What to expect if you keep reading

A brief exploration of the three kinds of Configuration types, an explanation of how to use them, and a dip of the toes in the deep waters of safe cross-project publishing and aggregation.

To guide our exploration, we will consider a concrete use-case. We want to be able to aggregate data from all subprojects at the root project (a natural location), and we want to do it in a safe way that doesn’t violate project boundaries. Here’s how we want our build scripts to look:

// aggregator/build.gradle (or root build.gradle)
plugins {
  id 'org.jetbrains.kotlin.jvm'
  id 'mutual.aid.configuration-roles'
}

dependencies {
  sourceFiles project(':feature-1')
  sourceFiles project(':feature-2')
}

and

// both *feature-1/build.gradle* and *feature-2/build.gradle*
plugins {
  id 'org.jetbrains.kotlin.jvm'
  id 'mutual.aid.configuration-roles'
}

We will apply our plugin to each of our projects.¹ It will set up publication of interesting data from the subprojects, as well as aggregation of that data in our aggregator, or root, project. The data we’re collecting for this example is information about the source files in our subprojects, hence the name of our imaginary new Configuration, sourceFiles.

What are Configurations, anyway? The three kinds

Configurations are used for…
…declaring dependencies
…resolving dependencies within a project
…sharing artifacts between projects

To create these three kinds of instances in the past (before Gradle 8.4), we had to do the following:

configurations {
  // you declare dependencies here
  create("implementation") { 
    isCanBeResolved = false 
    isCanBeConsumed = false
  }

  // plugins will resolve dependencies from this one
  create("runtimeClasspath") {
    isCanBeResolved = true  // Defaults to true
    isCanBeConsumed = false
    extendsFrom(configurations["implementation"]) 
  }

  // plugins expose artifacts to other projects on this one
  create("runtimeElements") { 
   isCanBeResolved = false
   isCanBeConsumed = true  // Defaults to true
   extendsFrom(configurations["implementation"]) 
 }
}

Note that we first create the instances, and then we mutate them in configuration blocks (yes, the word “configuration” is being used in two separate senses here).

In practice, most community plugins were not so scrupulous as to create three separate configurations for these three separate use-cases—and nor were they required to, since Gradle’s API is generally very permissive and legacy-friendly. The new APIs (see below) expose this complexity and enforce rigor… without, unfortunately, much (if anything) in the way of documentation or explanation.

Here are the new methods:

configurations { 
  dependencyScope("implementation")
  resolvable("runtimeClasspath") {
    extendsFrom(configurations["implementation"]) 
  }
  consumable("runtimeElements") { 
    extendsFrom(configurations["implementation"]) 
  }
}

We can all agree that this is fewer lines of code, but this obscures the additional rigor that is suddenly required to actually interact with these configurations. Because in point of fact, in the past, plugin authors would probably just do this:

configurations {
  create(“implementation”)
}

This creates a single configuration that Gradle will happily let you use to…
…declare dependencies
…resolve dependencies within a project
…share artifacts between projects

For reasons that are outside of the scope of this post, this is Bad™. Everyone™® agrees™®© that the Configuration API should be destroyed in a fire and/or thrown into the seam between universes, never to be seen again. The new API is meant as a step towards this glorious future.

How to do something useful and interesting with this stuff

With that out of the way, how do we actually use this API, keep up with the times, and make our plugins (slightly more) future-proof?

Let’s create a plugin! (Yes, this is my answer to everything.) The purpose of our new plugin is to aggregate information across all projects in our build in a way that is safe, that doesn’t violate project boundaries. It will demonstrate the usage of all three kinds of configuration, as well as very lightly touching on variant attributes. This is how Gradle models things like compile and runtime classpath separation, among other things, and are a very powerful (although deeply verbose and annoying) concept.

Setting up our configurations

First let’s create our three configurations:

class ConfigurationRolesPlugin : Plugin<Project> {
  override fun apply(project: Project): Unit = project.run {
    // Following the naming pattern established by the Java Library
    // plugin. More on this in a moment.
    val declarableName = "sourceFiles"
    val internalName = "${declarableName}Classpath"
    val externalName = "${declarableName}Elements"

    // Dependencies are declared on this configuration
    val declarable = configurations.dependencyScope(declarableName)
      // The new APIs return the new configuration wrapped in a lazy
      // Provider, for consistency with other Gradle APIs. However,
      // there is no value in having a lazy Configuration, since we
      // use  it immediately anyway. So, call get() to realize it, and
      // call it a day.
      .get()

    // The plugin will resolve dependencies against this internal
    // configuration, which extends from  the declared dependencies
    val internal = configurations.resolvable(internalName) { c ->
      c.extendsFrom(declarable)
      // Same as below
      c.attributes { a ->
        a.attribute(
          Category.CATEGORY_ATTRIBUTE,     
          objects.named(Category::class.java, Category.DOCUMENTATION)
        )
      }
    }

    // The plugin will expose dependencies on this configuration, 
    // which extends from the declared dependencies
    val external = configurations.consumable(externalName) { c ->
      c.extendsFrom(declarable)
      // Same as above
      c.attributes { a ->
        a.attribute(
          Category.CATEGORY_ATTRIBUTE,
          objects.named(Category::class.java, Category.DOCUMENTATION)
        )
      }
    }
  }
}

As indicated by the comments, we are following the naming convention established by the Java Library plugin.

Our declarable configuration is named “sourceFiles”, and corresponds to one of the green configurations in the diagram (let’s say “implementation” for a concrete example). It is on this configuration that we declare our dependencies. We extend this configuration with a resolvable (internal) configuration (colored blue), named “sourceFilesClasspath”,² and this is used within the project by tasks which resolve these dependencies. Finally, we also extend the original configuration with a consumable (external) configuration (colored pink), upon which we will publish artifacts for use by dependent projects.

We use the attributes to tell Gradle what kind of artifact we’ll be attaching to our configurations. In this case, we say we are publishing “documentation.” Note that this is, in a manner of speaking, arbitrary, though it is important to be consistent. If we own a Very Important ecosystem plugin such as the java-library or Android Gradle plugin, then it’s also important that our attributes be meaningful or our users will hate us. For this post, I just picked something easy and readily available without giving it a lot of thought.

Publishing a custom (non-jar) artifact

override fun apply(project: Project): Unit = project.run {
  val kotlin = extensions
    .getByType(KotlinJvmProjectExtension::class.java)

  // src/main
  val mainSource = kotlin.sourceSets.named("main")
  // src/main/{kotlin,java} as a FileTree
  val mainKotlinSource = mainSource.map { it.kotlin }

  // Register a task to produce a custom output for consumption by 
  // other projects
  val producer = tasks
    .register("collectSources", ProducerTask::class.java) { t ->
      t.source.from(mainKotlinSource)
      t.output.set(
        layout
          .buildDirectory
          .file("reports/configuration-roles/sources.txt")
      )
    }

...

configurations.consumable(externalName) { c ->
  ... as before ...

  // Teach Gradle which task produces the artifact associated with
  // this external/consumable configuration
  c.outgoing.artifact(producer.flatMap { it.output })
}

It’s a few lines of code, but what we’ve basically done is simply:

1. Registered a task that takes as input all of the source files of this project.
2. Register the output of that task as an outgoing artifact on our consumable (external) configuration.

The task that produces the artifact we’re sharing is not very interesting for our purposes here, but if you want to take a look, here it is.

Consuming the artifacts from dependency projects

Here we define and register a task that will consume the published artifacts of other projects. Recall this works because our aggregator project has this in its build script:

// aggregator/build.gradle (or root build.gradle)
dependencies {
  sourceFiles project(':feature-1')
  sourceFiles project(':feature-2')
}

This is what links our aggregator to the other projects we’re collecting data from. Now let’s register our new task:

// Register a task to consume the custom outputs produced by other 
// projects
tasks
  .register("printDependencySources", ConsumerTask::class.java) { t ->
    t.reports.setFrom(internal)
  }

and

abstract class ConsumerTask : DefaultTask() {

  @get:PathSensitive(PathSensitivity.RELATIVE)
  @get:InputFiles
  abstract val reports: ConfigurableFileCollection

  @TaskAction fun action() {
    val globalSources = reports.joinToString(separator = "\n") { 
      it.readText()
    }
    logger.quiet(globalSources)
  }
}

There’s nothing special here. The artifacts that our other projects are publishing are ergonomically available (as simple text files in this case) on our new consumable configurations, which we can feed directly and easily into our custom task, which reads those files into memory and then prints them to console for reading by your users.

Happy Gradling!

Cover image: a developer trying to find something in Gradle's documentation

Endnotes

¹ I can imagine we might want two separate plugins for this, one for publishing and one for consuming or aggregating. But this is not strictly necessary. up
² Yes, even though this won’t be a classpath in any normal sense. It’s just a convenient name. up