Isaac Lee

Posted on Jul 20, 2023 • Edited on Jul 22, 2023 • Originally published at crunchingnumbers.live

In 1 Year

#ember #javascript #webdev #beginners

Today, I'm going to show you how to create a platform so that you and your team can write code that is more maintainable and extensible.

Since last year, at CLARK, we've been reducing our tech debt. Our main application is a monorepo with 190 packages, accrued over 7 or 8 years. We've been on Ember 3.28 for almost 2 years now. Trying to get to 4 hasn't been easy.

These issues may sound familiar to you and I think it's a good thing to know that we don't have to face them alone. The Ember Community Survey in 2022 estimated that 3 out of 4 work projects may be stuck on v3 or below. A follow-up survey in December indicated the lack of resource and outdated addons as main reasons.

The question is, why are we finding ourselves in projects that are hard to maintain and extend? What can we do differently? I want to highlight possible solutions and give practical tips based on what my colleagues and I achieved over the last year.

Keep in mind that the problems that I will describe are not specific to work or Ember v3, but more programming in general. The reason is, I want you to be able to apply these solutions in other contexts, e.g. in open-source projects like adopted-ember-addons or when it's time to update Ember from 4 to 5.

So, fangen wir mal an. Let's get started.

1. In increments

When you have to update a project that hasn't been maintained, you're going to feel, at first, overwhelmed, hopeless. How in the world do we get out of this mess? The key to doing so—and this is the most important lesson from my talk: Do it in increments.

Imagine a yarn that is wildly tangled up. The yarn is supposed to be just 1 simple thread, but it is a chaos and you can't tell where it starts and where it ends. Now, if you try to untangle the yarn by applying force in every direction, all at once, you're going to fail and make a bigger mess. But, if you give small little tugs, one at a time, sooner or later you'll find yourself that 1 simple thread.

This incremental approach is also what mathematicians favor. Global algorithms like finding the inverse of a function work well in theory but are very limited in practice. Iterative methods, on the other hand—taking a step in one direction, a step in another, and so on, until we converge to the right solution—are powerful because we can apply them to many different situations. They are also efficient and will end up saving us time.

So what does an incremental approach mean for us developers? It means two things.

First, when it's time to update a project, we have to avoid creating a plan that spans weeks or months. We also have to avoid planning down to the smallest detail. Now, I'm not saying, have no plan and just wing it. What I'm pointing out is that your code is alive. It will never stay still because someone on your team will add a feature, or some package that you depend on will have a release. The assumptions for how to update your project will change quickly, and you will have to adapt quickly as well.

Second, an incremental approach means allowing mistakes. The right solution won't be obvious from the start, especially because the project hasn't been maintained. A pull request to update the project may introduce bugs, but that's okay, we can fix them in the next one. This may sound obvious but, if we take no steps because we fear of breaking the app, then we will get nowhere. Only by taking small steps and iterating on the solution can we make impossible possible.

1a. Code metrics

You rated me a six. I was like, "Damn."

The rest of my talk is devoted to showing which small steps you can take. But, before I do that, I'll share my thoughts on code metrics: Should we measure the improvements that we make?

In math, a metric is a number that describes a phenomenon without bias. For example, you are 10 centimeters taller than I, or the train was late by 5 minutes. One day, programmers, inspired by math, came up with numbers to describe code quality. How many lines of code are there? How many components and routes do we have? And so on.

The problem is, your code is alive and the assumptions for your project change from one day to another. You have to somehow ignore the changes that weren't in your control by normalizing the metric. Maybe divide the metric by the number of lines of code, or divide by the square root or the logarithm. Whatever you can think of.

You're starting to see that we're fidgeting numbers and adding personal bias, when a metric is supposed to be objective. I claim that it's impossible to measure the change of a code metric—in other words, how much the code improved over time. We can only measure what is now, at this moment.

Long story short, don't worry about code metrics. Just do the right thing and improve that code that you've wanted to.

2. The right code

The next question is, well, what is the right thing? What does it look like? The answer is going to depend on which aspects of code we care about. To help narrow the answer, we will focus on two: maintainability and extensibility.

A code that we can maintain and extend exhibits 3 characteristics: It has a minimum API, it separates concerns, and it has few dependencies. In math, we say that these are necessary conditions. This means, if your code doesn't meet one of these conditions, it cannot be maintained or extended. Think of these as a checklist, where each item tells you which small steps you can take.

To make things more concrete, I will use reusable components, something that we are familiar with and use on a daily basis.

2a. The right code has minimum API

An API (application programming interface) defines a boundary between two code and rules for communicating with each other. When the interface is good, the code is easy to use, maintain, and extend. When the interface is bad, the code becomes a hazard and can stop your project at some point.

A common mistake that I see in reusable components is supporting too many cases. That is, the component ended up with a large API by allowing many arguments. This happens when we try to predict the future and overdesign things, or when we try to quickly fix something with an if-statement (if there is this new argument, I'm gonna do something else).

What we tend to ignore is how every case increases complexity. Not linearly but, I claim, quadratically or worse because of the combinatorial effect—how arguments interact with one another. We can assume that the higher the complexity, the more likely that the component is untested and unmaintained.

Back in Germany, we have this amazing animal that's going to sustain the future, called eierlegende Wollmilchsau. It can give us eggs, wool, milk, meat, and companionship. It can do everything that we want, because it does not exist. It's an idiom like "Jack of all trades, master of none."

So the lesson is, design simple things. Got it. But what if I already have a component with a large API? How do I simplify it?

The key to answering this question is researching your current use cases. Find out which features were almost always used and which maybe once or twice. The code for the rarely used features? See if you can delete them.

By doing so, you reduce complexity and the chance that your component will cause an issue. Also, by removing code, you can cause a chain reaction of additional refactors. Remember, the way to untangle a yarn is to give small little tugs.

Now, what if a feature was used once, but you have to support it? I'd create another component, maybe through composition. The idea is, to treat this feature as an exception and not the norm. When designing reusable components, I want you to target the 80% and not give in to the other 20%.

2b. The right code separates concerns

Recall that an API draws a boundary between two code. In the best case, each side trusts the other to perform only certain tasks and no others. When this happens, the tests for each side are simpler.

The question is, when we design a reusable component, how should we separate responsibilities? Which tasks belong to the reusable component and which to the consumer, whether that's a component or route?

In my experience, we can maintain and extend the reusable component when it handles these 3 aspects:

Accessibility (so the consumer doesn't have to be an expert in it)
Styling (making sure that things inside the container look right)
Test selectors (what should be tested and how are the selectors named?)

Meanwhile, the consumer must provide these three:

Data and translations (what should the component render?)
Margin and padding for the container (so the component can play nice with others)
Callback functions (when a user takes an action, what should the component do?)

How I separated styling—namely, the container doesn't set its margin and padding, but the consumer does with an extra <div>—may come as a surprise. I learned this from Sean Massa and Trek Glowacki a few years ago, and find that it really helps us reuse and refactor components. The rationale is, reusable components should only care about what happens inside. We encounter this idea also in container queries.

Another thing to note here is data and translations. Before Ember 3.25, we would have had to use a bunch of arguments to pass these down. Now, thanks to named blocks, the reusable component can focus on the layout, while the consumer on the content. Named blocks are one of my favorite features of Ember.

A feature of Ember that I want you to use with caution is ...attributes, as they can easily destroy separation of concerns. I'll give you 2 examples.

One time, I tried to replace flex with grid to simplify a reusable component, only to find out, many of the consumers had passed the class attribute and had overwritten the flex properties. Thanks to splattributes, implementation got leaked and the consumers are now forcing me to keep using flex.

Another problem that I observed at CLARK is too many test selectors, because many consumers had passed their own. When the same DOM element is referred to in 7 different ways, refactoring the reusable component becomes tedious. Suddenly, I have to update multiple test files in different packages.

In general, I think ...attributes is a sign that the component wasn't designed right. Maybe it should ask the consumer to, instead, use arguments to define styles, named blocks to customize content, and test helpers to write tests. I want you to use the right tools to solve the right problems.

2c. The right code depends on few

Just like arguments, every package that we install increases the chance that something goes wrong. But unlike arguments, we don't really have control over packages. If the package author doesn't do releases or they make breaking changes, our code can become stuck in time.

Now, I'm not saying, write every code yourself and have zero dependencies. Instead, ask yourself: Is their code more stable than mine? To me, stable doesn't mean, there's a 1.0 release. It means, the package is well-written, -documented, -tested, and -supported. You can install the package only if the answer is yes.

Of course, at times, you will need to install a package that is not stable. If so, try to wrap the code that you need, then write tests to document the wrapper's input and output. This way, if the package turns out to be no go, you will have to replace the code in only 1 place: the wrapper.

I'll tell you about a mistake that we had made at CLARK and how it's now affecting us with upgrading Ember. (It's a funny story and you're going to laugh, because you're not affected.)

We use ember-file-upload and I noticed, we are on 5.0.0-beta when the latest is 8-something and I can't update Ember to 4 without updating ember-file-upload first.

Well, it turns out, we came up with like 10 different ways to render <FileUpload> and, since the beta, the addon changed its API and styling drastically. So now, we have to fix deprecations, visual regressions, and failing tests for every one of these cases. If only we had come up with a wrapper component, tja.

3. Solid foundation

To help people design code right, we need a strong foundation—things like simple lint and test strategies, up-to-date dependencies, and short build and rebuild times. Replacing the foundation is a comparatively large task, but it's also something that we will have to do only once in a while.

The question is, for an existing project, how do we replace the foundation without stopping everything else? The solution, again, is in increments.

3a. Overhauling lint

A year ago, at CLARK, every package had different linter configurations, so we couldn't update packages like eslint and typescript. Furthermore, how we asked our developers to lint files was different from how we asked our CI.

Fast forward to now, our project is set up like this: There are only 3 scripts for developers to remember because every package has them: lint, lint:fix, and test. CI runs the exact same scripts so we can easily reproduce issues locally.

Second, we use the flag --cache and the package concurrently so that we can lint files faster and more exhaustively.

Lastly, we have limited resource so we rely on the default as much as possible. Things like blueprints from ember-cli and official plugins like @tsconfig/ember. We adopt 3rd-party plugins only if they are stable in the sense that I mentioned earlier.

And here is how we changed linting across 190 packages. It turns out, packages are not equal. Leaf-node packages (packages that don't depend on others) were actually seldom worked on, so we updated them all at once. Packages for a business domain belong together so they were updated together.

In a single day, we updated eslint to v8 and reset all configurations by bypassing CI. We asked lint:js to return code 0, an unconditional success. Afterwards—again, in groups of packages—we reverted the scripts, ran auto-fix, and ignored errors that couldn't be fixed.

With this divide-and-conquer strategy, it took me (one person) about 10 pull requests and no more than 5 days to introduce a change.

3b. Spotting blockers

Next, we look out for deprecations and outdated packages that can block us from updating more critical dependencies like ember-source.

To find deprecations, we can use ember-cli-deprecation-workflow and create a to-do list. The addon does require that we have enough tests to avoid false negatives. Later, I'll show you how you can write simple tests when there aren't any. Another approach is to run the app and check Ember Inspector's Deprecations tab.

Here are 3 deprecations that will likely affect many projects. You can visit deprecations.emberjs.com to learn more.

implicit-injections (v4)
this-property-fallback (v4)
routing.transition-methods (v5)

I also gathered a list of important packages and the minimum version that you want to reach. It's important to update ember-auto-import and ember-modifier now, because more addons will move on to support Embroider.

3c. Short (re)build

To help people iterate on a solution many times, we need builds and rebuilds to be fast. A forewarning: High performance optimization isn't my expertise so I cannot give you definitive answers. Nonetheless, I'll share what we did at CLARK, which seemed to help lower the times. Maybe they can help you too.

We ended up with 190 packages because of premature abstractions. By combining packages and removing dead code, we are now down to 150. In the process, we removed a few cyclic dependencies by creating a leaf-node package.

Now, if there is a component used by many packages, see if you can simplify it. You can, for example, make it template-only and replace older syntax with newer ones.

An ongoing project for us is to declare dependencies correctly so that we can adopt Embroider. Because we use yarn to manage the monorepo, many packages that had been created by copy-paste listed wrong dependencies. You can find unused dependencies by searching code for how they would have been used.

Finally, if you have ember-cli@3.15 or higher, there's a hidden feature that can make rebuilds faster. In ember-cli-build.js, simply set BROCCOLI_ENABLED_MEMOIZE to true. This happens by default in Embroider projects.

4. Solving together

My colleagues and I are in this lucky situation, where I can maintain code full-time, but that's still a bus factor of 1. It's important that I share knowledge and get more people involved. This year, we began to tackle tech debt together. Each quarter, we discuss ideas and decide what to work on.

To show that every one of you has the power to make change, first, I will cover 5 techniques for refactoring. These are accessible and can be used on a daily basis. Next are codemods, something that's more advanced and takes time to do. Finally, we will think about how our interpersonal skills affect how we collaborate.

4a. Refactors

There is this book on refactoring that I hate. It gives us 70 techniques and the examples are academic, so I could never tell which are actually important.

I claim, we just need 5 to survive: Write tests, rename things, make early exits, extract functions, and remove dead code.

Write tests

If your project doesn't have tests, you can use Ember CLI to write the simplest test, a tautology (true is equal to true). For example, you render a component and write assert.ok(true), or you look up a service and assert that it is truthy.

Even these placeholder tests provide two valuable information: the minimum data needed to initialize your object, and a guarantee that the object won't cause issues when you use it.

You can learn more about how to write tests from my talk at EmberFest 2019. The lessons from back then still apply today.

Rename things

When you don't understand what a variable, condition, or function does, there's a chance that other people won't either. Once you understand the code better, give names that are descriptive.

Make early exits

Nested conditions (in general, indentations to the right) are a recipe for disaster. They encourage us to keep nesting to handle new exceptions.

You can fix this by making early exits. If there is code that happens when a condition is true, instead, you exit immediately when false, using return, break, or continue. Early exits help us simplify logic and move code to the left.

Extract functions

At times, you will find a function that has many lines of code, but is actually performing a few key steps in sequence. If so, create a function for each key step and give a name that describes the step.

This process of breaking a large function into smaller ones is called extraction. Once you extract functions, you have the option to move them to app/utils (“utilities”) and write unit tests.

Remove dead code

Finally, if you see code that isn't used, delete it. By removing code, you can simplify assumptions, remove dependencies, and allow further refactors.

To find dead code, you can use git grep or your code editor's Find tool. For searches to be accurate, though, your code has to be written well (a Catch-22) and, ideally, be statically analyzable. It's hard to match names that are dynamically generated.

4b. Codemods

When you want to update many files to follow a new format, you might ask, should I write a codemod, a program to update the files for me? It depends.

When you write a codemod, you pay the costs upfront. You have to first create and configure a package, write code, then write tests to test that code. This can take days or weeks. But once you have a codemod that is backed by tests, the returns are manifold. The codemod can update your project in a second and can be reused to help other projects migrate.

Here's the crux: When your project has many variations in code because it hasn't been maintained, updating it by hand will be faster. A codemod will run into edge cases that may or may not occur in other projects, and every edge case that you handle is extra code that you have to maintain.

Nonetheless, the ability to help others just might be the deciding factor for you. As a rule of thumb, consider writing a codemod if you can cover the usual 80%.

To get started, I would've recommended two years ago Robert Jackson's codemod-cli, but this package is unmaintained, doesn't support TypeScript, and makes a divide-and-conquer strategy—taking small steps—hard to achieve.

So I created @codemod-utils, a set of tools and conventions for writing codemods. I use it to power all of mine:

And just last week, I published a CLI. You can use it to create a modern project that comes with lint, test, CI, and documentation out of the box.

My hope is, we can lower the barrier enough that, if a person can write a function in Node.js, then they can start writing a codemod. I'd love to see more people writing one, given that Polaris is coming up, and Glint and <template>-tag can use a higher adoption. Who knows? Maybe, some day, I will give a talk on codemods.

4c. Different perspectives

While solving problems together, I want you to think about your interpersonal skills and how they affect how you collaborate.

Each of us, with a unique background, has a different way of thinking and verbalizing how we perceive the world. These differences surface when we discuss ideas and review each other's code. The more we are competent in our interpersonal skills, the better we can appreciate the differences and appreciate one another for who they are.

We say interpersonal skills because they are something that we can learn by practice. I myself learned through Toastmasters, where I met many good people. Nowadays, with more information online, I recommend that you branch out and see what interests you.

Back in Germany (and this time, it's real), you can stream shows called Sag's mir, Unter Anderen, and 13 Fragen, where 2-6 people with opposing ideas carry a conversation that is personal and civilized. The goal isn't to win the argument, but to listen to each other and come up with a compromise. I really like these shows because they can teach us what makes a discussion go well.

5. Future is now

Last but not least, I want to show you that you can move towards the future even when your project is behind. For example, you can install polyfills to start modernizing syntax, and update ember-cli to the latest, independently of ember-source:

If you maintain a v1 addon, you can enable two ember-try scenarios (embroider-safe and embroider-optimized) so that you can discover issues early and create a plan to adopt the v2 format. You can also support Glint and <template>-tag users, even before you migrate to v2.

This April, at CLARK, we started using a private package registry. It helped us extract linter configurations from the monorepo, so that we can reuse them in other projects and standardize how we write code. The registry also helped us set up another monorepo (with pnpm), where we extracted addons, converted them to v2, introduced embroider-css-modules, supported Glint and <template>-tag, and wrote test apps to make sure that we are ready for the future.

By thinking a bit outside of the box, you just might be able to solve many problems at once.

6. Closing

In conclusion, when your project is currently hard to maintain and extend, you may feel overwhelmed and hopeless, but please—don't give up. By taking small steps and iterating on a solution, you can introduce change that will help you and others. And you don't have to do it alone. Ask your team and the community that we have for help.

In 1 year, when we are back at EmberConf, I'd love to hear how your project is doing. Maybe you'll strike a conversation with me, in the hallway or on Discord, or present a talk like I did today. Until then, mach's gut. Take care.

DEV Community