The thousand dollars one line mistake - SBT + PlayFramework

Nowadays everyone talks about how important it is to have a good developer experience, as it will have many good side effects, such as but not limited to:

• Development speed / Productivity

• Code quality / Maintenance

• Saving costs, etc

However, often we get ourselves working on projects that at some time in the past, a small piece of code was added to make the project faster, or even to fix something, maybe someone was trying to make the build faster, or even trying to give the engineers a better development experience. This was the case in this story.

A few years back, in a project that we worked on ( before I was even part of the company ) an issue with building SBT, Scala and play framework, was identified, where the compilation time for building the project locally was taking around 3 to 5 minutes depending the machine. An attempt to fix the issue was made. The project structure was split in 2 like below:
Before

ProjectA
  /api
 /core
 /app

After

ProjectA
 /core
 /app

ProjectApi
  /api

The following was added to the build.sbt

lazy val projectA = (project in file("."))
  .enablePlugins(...)
  .settings(commonSettings)
  .aggregate(api)
  .dependsOn(api)

lazy val api = project.settings(commonSettings)

By doing so, it did improve the compilation time, only during the build on the CI pipeline, I'm not sure if it helped during the development phase, however, it added a new and horrible bug that made developers waste thousands of hours of work.

After this line was added, developers started noticing how long it was taking just to run a simple
sbt run locally, as for every change in the codebase now, a full compilation was needed.

The journey to understand the issue

As per documented in SBT reference manual - Multiproject

Is important to note the two definitions of Aggregation and depends on

Aggregation means that running a task on the aggregate project will also run it on the aggregated projects

A project may depend on code in another project. This is done by adding a dependsOn method call. For example, if core needed util on its classpath.

After spending one day or two reading documents and many frustrated attempts to fix the issue, I ended up arriving at this Github - Spurious recompilation in multi-project build This was not the fix itself, however, gave me light by the end of the tunnel to understand the problem was indeed with the multi project setup.

And further more, I understood what was happening, and by so, now my build.sbt file was just as simple as:

lazy val projectA = (project in file("."))
  .enablePlugins(...)
  .settings(commonSettings)
  .dependsOn(api)

lazy val api = (project in file("api"))
  .settings(commonSettings)

There was a problem with how we set up projectA in SBT. We told SBT to include the project's API (which was right), but the API definition pointed to the entire project root. This meant that:

Whenever the API needed compiling, SBT would also try to compile projectA itself.
Since projectA needed the API to compile, it would trigger another API compilation.
This created an endless loop, forcing developers to kill SBT and manually clean and compile everything for every code change.
Here's what happened in simpler terms:

We told SBT to include the project's API.
The API definition pointed to the whole project.
Compiling the API triggered a full project compilation (including the API again).
This loop made SBT very slow and frustrating for developers.

The team had worked with this problem for at least 4 years...

Aftermath - fixing the issue

After I said to my teammates that I had a surprise feature already merged on master, people did not understand what was happening, however, I wanted to see the happiness on their faces, I told the whole team to pull master into any branch that they were working on, some of them did not notice anything in the first try, other started to notice that after changing any code in the codebase, it was compiling only the affected file in a matter of seconds, not minutes as before. And the best surprise was when one of the teammates noticed and said out of loud in the office.

Gust ... did you fix the compilation loop issue? I'm working in something here and I am getting instant feedback for any changes in the code.

At that time I had to admit and, share the news with all the other engineers, it made me a happier engineer as now I am happy working on this project and not waiting long period of times for a full compilation on our project.

If you feel that something is our way, whenever you are, whatever you are doing, remember you have the chance to make it better, never forget what you have started.

If you like reading this article or would like it to have more content, please let me know in the comments, I am happy to share more about this journey.

Thank you for reading.

A few stats from the project:

Project start:

• Around 4 thousand Java files

• Around 300 Twirl templates

• Compile time before improvements 3 to 5 minutes for any change in the code

• Compile time after improvements average of 1 minute and 20 seconds for full compile

• Compile time after improvements average of 5 to 10 seconds for any change with instant feedback ( the most time spent is Playframework restarting the HTTP Server )

Cover image made by AI.