Preface
In 2023, Typescript is rarely questioned as an important tool for modern JavaScript developers, but one of its biggest limitations is the lack of added runtime type safety, particularly when dealing with IO at the boundaries of your application.
To solve this problem a number of popular runtime validation and type-safety tools have popped up, usually competing on runtime parsing speed, API expressiveness, and a TypeScript vs JSON-schema based core.
Compile-time and Editor Performance Pain
What seems to be degrading is performance of the developer experience. TypeScript drives the code intelligence of many popular editors, and as you add type complexity and size to an application or monorepo, code-completion and type-checking starts to draaaaag. This is because driving features like autocomplete means compiling your code on the fly repeatedly, and the more types & files needed to populate autocomplete for a line of code, the longer it takes to get a response.
This is compounded by the growing movement of fully and deeply typed libraries like tRPC, and Tanstack's Query & upcoming Router, which utilise new layers of generics and mapped types over your DTOs, deepening your types, and surfacing compile-time performance issues which weren't as noticeable before.
What's prompted me to look at this is my tRPC+Zod editor experience at work has become unusable. 2-3 seconds to just get autocompletion options on a tRPC path, and then another 2-3 for the next path, repeat. When investigated using TypeScript's tracing tools the data entirely points back to my team's Zod DTOs. What I learned is that Zod's performance is okay at the start, but when you start using methods like .extend/.pick/.omit (and so on) the performance regresses in the order of a magnitude. Rather than making this into a "Zod considered bad" post, I wanted to investigate how the alternatives which can be integrated with tRPC fare, and see whether I can do better.
Analysis Setup
For these benchmarks I'm using a Macbook Pro with M2 Pro and 32GB of RAM.
I've worked on 2 codebases in an Nx monorepo to set up this experiment.
- Light Type - an experimental runtime type-checker with feature parity in a number of key areas (and an inspired API) to Zod
- Benchmarks - a tRPC API and React SPA which uses a number of popular and tRPC-compatible type-checkers
The type-checkers under test are:
- Zod 3.20.6
- Yup 1.0.0
- Superstruct 1.0.3
- TypeBox 0.25.21
- Light-Type (Experimental Version)
There may be more libraries which can be integrated with tRPC, but the first 3 are recommended by the project, and TypeBox has had some effort made to integrate it. Light-Type is designed to fit the necessary interfaces of Zod so it's also compatible natively.
The Benchmarks are each made up of a tRPC router with types co-located, and a React Component using tRPC's @tanstack/query abstraction which consumes the router.
TypeScript is pretty good at optimising its compiles. It caches where it can and where there's a single path back to a type it doesn't spread itself wider. So when performance tracing this setup I've seen that compile-times for each router/component don't bleed onto the others, making benchmarking the compiler much easier.
The Actual Types
Each library has two benchmarks, a 'simple' one, and a 'complex' one which adds usage of the equivalent to extend/omit in each library. Except Yup which doesn't have the necessary features to join the complex round.
Simple Router Example
// simpleZodRouter.ts
import { z } from 'zod'
import { publicProcedure, router } from '../trpc'
const PersonDto = z.object({
id: z.number(),
firstName: z.string(),
lastName: z.string(),
tel: z.string().optional(),
})
const CarDto = z.object({
id: z.number(),
name: z.string(),
age: z.number(),
brand: z
.union([
z.literal('Volvo'),
z.literal('Mercedes'),
z.literal('BMW'),
z.literal('Ferrari'),
z.literal('Bazmus'),
])
.optional(),
previousOwners: z.array(PersonDto).default([]),
})
type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']
export const simpleZodRouter = router({
list: publicProcedure
.input(z.object({ count: z.number().default(100) }))
.output(z.array(CarDto))
.query((opts) => {
return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
return {
id: idx,
name: 'Foobarmo',
age: 3,
brand: 'Bazmus' as Brand,
previousOwners: [
{
id: idx * 100,
firstName: 'Bob',
lastName: 'Owneverythingman',
tel: undefined,
},
],
}
})
}),
get: publicProcedure
.input(z.object({ id: z.number() }))
.output(CarDto)
.query((opts) => {
return {
id: opts.input.id,
name: 'Foobarmo',
age: 3,
brand: 'Bazmus' as Brand,
previousOwners: [
{
id: opts.input.id * 100,
firstName: 'Bob',
lastName: 'Owneverythingman',
tel: undefined,
},
],
}
}),
create: publicProcedure
.input(CarDto)
.output(CarDto)
.mutation((opts) => {
return opts.input as DbCar
}),
})
Complex Router Example
// complexZodRouter.ts
import { z } from 'zod'
import { publicProcedure, router } from '../trpc'
const EntityDto = z.object({
id: z.number(),
})
const PersonDto = EntityDto.extend({
firstName: z.string(),
lastName: z.string(),
tel: z.string().optional(),
})
const CarDto = EntityDto.extend({
id: z.number(),
name: z.string(),
age: z.number(),
brand: z
.union([
z.literal('Volvo'),
z.literal('Mercedes'),
z.literal('BMW'),
z.literal('Ferrari'),
z.literal('Bazmus'),
])
.optional(),
previousOwners: z.array(PersonDto).default([]),
})
type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']
export const complexZodRouter = router({
list: publicProcedure
.input(z.object({ count: z.number().default(100) }))
.output(z.array(CarDto))
.query((opts) => {
return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
return {
id: idx,
name: 'Foobarmo',
age: 3,
brand: 'Bazmus' as Brand,
previousOwners: [
{
id: idx * 100,
firstName: 'Bob',
lastName: 'Owneverythingman',
tel: undefined,
},
],
}
})
}),
get: publicProcedure
.input(z.object({ id: z.number() }))
.output(CarDto)
.query((opts) => {
return {
id: opts.input.id,
name: 'Foobarmo',
age: 3,
brand: 'Bazmus' as Brand,
previousOwners: [
{
id: opts.input.id * 100,
firstName: 'Bob',
lastName: 'Owneverythingman',
tel: undefined,
},
],
}
}),
create: publicProcedure
.input(CarDto.omit({ id: true }))
.output(CarDto)
.mutation((opts) => {
return opts.input as DbCar
}),
})
Running the benchmark
# Set up the monorepo
yarn install
# The app/api can served as a test of the integration
yarn light-type-benchmark:start
# Run a traced compile of the repo
yarn tsc:trace
> tsc -p tsconfig.base.json --noEmit --incremental false --jsx react-jsx --allowSyntheticDefaultImports --generateTrace ./.trace
tsc:trace produces ./.trace/trace.json which can be analysed in your browser's dev tools or Perfetto.
This does produce a lot of tracing output which we don't need, but since the benchmarks are isolated in their own projects TypeScript produces reliable results for them. I've edited down the trace file just to the files we're concerned with.
Results
And here are the results:
You can download and explore the trace yourself here: https://gist.github.com/Nick-Lucas/ee964bf004f40ea6601bd2543869efee
While compile-times do vary slightly from run to run, the scale of difference between each library remains the same.
| Library | Test | Router ms | Component ms | Note |
|---|---|---|---|---|
| Zod | Simple | 24ms | 10ms | |
| Zod | Complex | 281ms | 17ms | 281ms is not a typo |
| Yup | Simple | 22ms | 9ms | |
| Yup | Complex | Could not join the fun | ||
| Superstruct | Simple | 10ms | 11ms | |
| Superstruct | Complex | 42ms | 18ms | |
| Typebox | Simple | 28ms | 11ms | |
| Typebox | Complex | 38ms | 9ms | |
| Light Type | Simple | 8ms | 25ms | |
| Light Type | Complex | 22ms | 11ms |
Conclusions
The big outlier here is Zod, which seems to have some structural issues with its types, ballooning in cost by 10+ times after just 2 .extend and 1 .omit call. All the other libraries perform pretty well in the same situation, though Superstruct seems to balloon by about 4x, and TypeBox is slower to begin with but then degrades less poorly. Yup doesn't compete on complex features but performs very well at what it's designed for.
It's clear that all the type processing for the libraries is happening in the processing of router files where those types are located, and those are quite varied in cost. These results are reproducible over and over again too, it's not a die-roll for the big differences. The react components themselves seem to be isolated and very fast to compile.
-
Light Type is consistently the fastest across multiple benchmarks. This is going to be down to a couple factors:
- It has fewer features than the other libraries. I would predict it will slow slightly over time as I continue developing it.
- Light Type has a lot of focus put into producing the simplest possible types at every stage. Which for example Zod doesn't seem to prioritise.
Notes specifically on Zod: That big block in the flamegraph is Zod's Complex test. Basically don't try to use any features outside of primitives, object, array, or you may immediately gain 10x 🤒 to that type.
- The sum total of all type compilations is your editor experience, and 281ms is already about as high as you probably want your entire library of types to take for autocomplete.
- Those brown tags in the top of the flamegraph are labelled
recursiveTypeRelatedTo_DepthLimit, I'm really not an expert on TypeScript's insides, but that sounds like it's trying to tell us something important, and Zod has been guilty of causing "Type instantiation is excessively deep and possibly infinite" errors. - Zod right now, looks to be a big footgun and in need of rethinking its types, at least when combined with other modern tools like tRPC. If you're starting a new project it might one to avoid for now, though there's really not much alternative if you love Zod's API
Top comments (11)
Hey there @nicklucas ! I'm on the core team at ts-rest.com. We've actually done a very similar analysis here: github.com/ts-rest/ts-rest/issues/162. We use Zod for a lot of our implementation. Curious to see if you've enabled
strict(more specifically,strictNullChecks) in all of yourtsconfig.json/compilerOptions?If not, I implore you to follow the steps here ts-rest.com/docs/troubleshoot#why-... (you don't have to use
ts-rest) and see if it improves your TS performance.strictNullChecksis actually required by Zod github.com/colinhacks/zod#requirem..., and the reasoning is here: github.com/colinhacks/zod/issues/1750I suggest rethinking posting this article if you haven't enabled that compiler option for every
tsconfig.jsonin your monorepo, as that would lead to an unfair comparison against the other libaries.Thanks for reading!
Hey Michael, yes these benchmarks were all done using strict=true and it's the same at work. Strict mode just makes TypeScript that much better. Good question though as I didn't think to bring it up :)
Have you tried SureType?!
npmjs.com/package/suretype
We maintain a whole repo of runtime benchmark tests here:
github.com/moltar/typescript-runti...
Didn't know about Light Type, added an issue here:
github.com/moltar/typescript-runti...
Looks fantastic! Any plans to add comparative benchmarks for type-checking in the spirit of this article?
Light-Type is very much still an experiment, but I'd love to see it incorporated, especially if it goes to a release 😄
Do you mean this:
If so, there are a few issues and discussions surrounding that. No clear consensus yet on how to achieve parity across all packages. Basically there is a debate whether we should have a common denominator or if we should open up to different test cases. Opening up to different (per package) test cases can open a can of worms.
You can find an awesome, and very thoughtful response by @sinclairzx81 (the author of typebox) here:
github.com/moltar/typescript-runti...
Ah I mean, my understanding from looking through your results is it's about run-time performance of the parsing of data. This piece here is about language server / compile-time performance of typescript itself. I believe they're both very important in their own ways.
Zod has put a lot of effort into being pretty fast at parsing and that shows in your suite, but it's demonstrably bad at producing performant typescript inference. I've worked on a benchmark suite lately which demonstrates this and could easily be extended to more libraries
Ok, I understood your point. Yes, I thought about TS parser performance. I think it would be a great addition to the suite. Please create an issue with any thoughts or suggestions you may have to foster a public discussion. We would appreciate your existing knowledge transfer so we can make the benchmark suite more robust.
Feel free to submit a PR. Use a previous PR for submission as guidance. Even if it's not fully ready, it's ok. As long as it's a published package, it's fair game. At least that will give you some feedback how it stacks against the rest of the ecosystem!
Issue: github.com/moltar/typescript-runti...
I highly encourage folks to checkout the little talked about lib SureType... it is amazingly fast and a pleasure to use.
npmjs.com/package/suretype
How does this affect time-to-autosuggest (considering TS caches a lot of stuff)?