Let's start with my chosen definition of reliability. It is quite academic but don't worry, we will dissect and explain it in a human and business-centric way.
Reliability is the probability of failure-free software operation for a specified period of time in a specified environment.1
Reliability is fundamentally about correctness. Time and environment are considered for practical reasons — to make reliability quantifiable and measurable. Companies typically assess reliability based on the environment (development, staging, production) and specific timeframes (e.g., the last 7, 30, or 90 days). It is important to note that reliability is not the same as availability — software can function correctly yet still be unavailable.
Now that we know what reliability is, we can discuss why it is important to the business even at the risk of stating the obvious. When people buy software products and services, they place their expectations on them. These expectations are influenced by the provider's claims of quality. If these expectations are not met, they may walk back on their purchase, ask for a refund, and eventually, turn to the competition for alternatives. A low level of reliability may be one of the reasons for not meeting their expectations. Other reasons include availability issues, low performance, poor user experience, and so on.
How does unreliable software lead to customer churn? When people use our software, they pursue a goal. If they cannot achieve their goal, there is no point in using the software in the first place. To achieve these goals, the software must work correctly as expected by the user. Often incorrectness manifests itself in obvious ways such as internal server errors. If these errors occur intermittently, it is not necessarily a reason to leave for the competition. People have learned to live with occasional issues and intuitively refresh the page when they occur. However, if it happens often enough, it becomes frustrating, compounds over time, and leads to customer churn. The good news is that software providers can easily detect and measure these errors. At other times, incorrectness is very subtle and difficult to spot. For example, mistakes in financial data analytics. People base some serious decisions on this data. If the software presents the wrong numbers, it may have a catastrophic impact on the company using the software.
Does this mean that every company aims to develop equally reliable software? No. While reliability is important for all companies, its significance varies depending on their stage of development.
- Startup: Reliability is moderately important. While the highest priority is rapid iteration, a base level of reliability is essential to gain customer trust. The company must ensure that core features function as expected, critical bugs are identified and fixed quickly, and third-party vendors are reliable to minimize infrastructure risks.
- Scale-Up: At this stage, poor reliability can stall growth. The challenge is to scale rapidly while minimizing reliability issues. The more reliability suffers, the longer it will take to reach the growth targets. Conversely, excessive focus on reliability can also slow growth. Striking the right balance is crucial. Key investments should include automated alerting and monitoring to detect and resolve failures quickly, as well as redundancy and failover mechanisms to handle traffic spikes.
- Corporate: Reliability now directly impacts customer retention, operational efficiency, and revenue. The primary goal is to maintain customer trust. Key investment areas include high availability, multi-regional architecture, strict SLAs to uphold reliability commitments, and service incident post-mortems to prevent recurring failures.
- Enterprise and Mega-Corp: At this scale, mission-critical failures can lead to financial, legal, and operational crises. The main objective is to mitigate risks. Key investment areas include disaster recovery plans, predictive analytics, DDoS protection, and a zero-trust security architecture.
Okay, we know what reliability is and what it means to the business. How does Effect improve the reliability of the produced software? Two major features of this framework directly impact reliability: Error Management2 and Configuration Management3. I want to focus specifically on these two features for a reason. From my experience in the industry, escaped defects (colloquially known as bugs) and configuration errors cause over 40% of all incidents. Improvements in these two areas only can yield up to twice as reliable software. Can Effect help us get rid of all the bugs? Absolutely not. Can it significantly reduce the surface area of possible bugs? Absolutely. Let me show you how.
Error Management
Error management can be divided into two main concerns:
- Error discovery
- Error handling
Error discovery
In conventional programming, where errors are not represented as values and are not statically typed and hence known at the time of development, error discovery is a painful and lengthy process. Generally speaking, there are a few approaches to it:
- Reading: the engineer would inspect the implementation of each action to an arbitrary level of depth. Remember, in the real world, our programs tend to be very deep. This is because business domains tend to get more complex as time passes, as well as the increased utilization of internal/external libraries to reap some productivity benefits. It is unrealistic to expect all possible errors to be discovered this way.
- Trial: the engineer would try different inputs, and simulate different states of the environment (network failures, permutations of stored data, etc.) to discover how the program can fail. It is unrealistic to expect all possible errors to be discovered this way, either.
- LLM: the engineer turns to their favorite LLM for advice. The machine activates all its artificial neurons to ask humanity's collective consciousness what to tell the poor soul. Still, we cannot realistically expect to discover all possible errors this way, either; especially in large codebases consisting of hundreds and thousands of files.
As demonstrated, conventional techniques are either labor-intensive or limited in effectiveness. A common approach is to identify obvious errors early while allowing unforeseen issues to surface in production — essentially "taxing the early adopters," who are often paying customers. Some may see this as a disservice.
TypeScript and Effect offer a superior alternative — automating the error discovery phase using the compiler. This process takes only an instant to a few seconds.
type CreateArticle = Effect<Success, Failure, Requirements>
Both success and failure can be encoded in the type system. The industry calls this a "result type"4. The Effect type5 extends this concept by also encoding dependencies — more on that in a future post.
Error handling
To demonstrate error handling, let's implement a function called createArticle that takes two arguments:
- input: The request body data for creating an article.
- currentUser: The user making the request.
The function must meet the following requirements:
- Only users with
READ_CATEGORYandWRITE_ARTICLEpermissions can create an article. - The input has an unknown structure and must be parsed.
- The article must belong to an existing category.
- A category has a limit on the number of articles it can contain.
We’ll explore three different approaches to error handling.
Try-catch approach
const createArticle = async (input: unknown, currentUser: User) => {
try {
authorize(currentUser, ["READ_CATEGORY", "WRITE_ARTICLE"])
const { categoryId, ...article } = parseInput(input)
await enforceCategoryExists(categoryId)
await enforceArticleLimitInCategory(categoryId)
const persistedArticle = await pRetry(
() => persistArticle(article),
{ retries: 1 }
)
return { status: 201, article: persistedArticle }
} catch (e) {
if (e instanceof /* ??? */) {
// ^^^^^^^^^
// no knowledge of what can go wrong
}
}
}
The try-catch block is readable and easy to understand but ineffective for error discovery. Since the caught error is typed as unknown, the IDE cannot provide intelligent hints about possible error types. Even with LSP6 assistance, there is no guarantee the listed errors will occur at runtime.
Go-style approach
const createArticle = async (input: unknown, currentUser: User) => {
const [_, unauthorizedError] = authorize(currentUser, ["READ_CATEGORY", "WRITE_ARTICLE"])
if (unAuthorizedError) {
return { status: 403 }
}
const [parseResult, parseError] = parseInput(input)
if (parseError) {
return { status: 400, errors: [{ id: parseError.name, details: parseError.message }] }
}
const { categoryId, ...article } = parseResult
// etc.
return { status: 201, article: persistedArticle }
}
This approach requires functions to return tuples, ensuring errors are explicitly handled. For example, parseInput could be typed as:
type ParseInput = (input: unknown) =>
| [CreateArticleInput, false]
| [void, ParseError]
This forces error handling in most cases. However, since authorize returns void regardless of success or failure, the compiler cannot enforce error handling for it. Additionally, this approach is verbose and negatively impacts code readability compared to try-catch.
Result type approach
Another alternative is to use a basic result type with a tagged union7.
type ParseInput = (input: unknown) =>
| { _tag: "success", value: CreateArticleInput }
| { _tag: "failure", error: ParseError }
Some developers prefer the explicitness of this type. However, it makes the resulting code even more verbose than the Go-style error-handling approach.
const createArticle = async (input: unknown, currentUser: User) => {
const authResult = authorize(currentUser, ["READ_CATEGORY", "WRITE_ARTICLE"])
if (authResult._tag === "failure") {
return { status: 403 }
}
const parseResult = parseInput(input)
if (parseResult._tag === "failure") {
return { status: 400, errors: [{ id: parseResult.error.name, details: parseResult.error.message }] }
}
const { categoryId, ...article } = parseResult.value
// etc.
return { status: 201, article: persistedArticle }
}
Compared to the previous approach, the reliability gains of this method are minimal at best.
Effect approach
Is there a way to combine the readability of try-catch with the reliability of Go's style of error handling? Yes. Effect’s error handling is just as readable as try-catch while providing even stronger reliability guarantees than the Go approach.
Don't worry if the following code seems complex — we'll break it down step by step in the next section.
import { Effect, Schema } from "effect"
import { ArticleBody, ArticleTitle, authorize, CategoryID, enforceArticleLimitInCategory, enforceCategoryExists, persistArticle, type User } from "./domain.js"
import { exhaustiveErrorCheck } from "./helpers.js"
class Input extends Schema.Class<Input>("Input")({
categoryId: CategoryID,
title: ArticleTitle,
body: ArticleBody
}) {}
const parseInput = Schema.decodeUnknown(Input)
const createArticle = (input: unknown, currentUser: User) => Effect.gen(function*() {
yield* authorize(currentUser, ["READ", "WRITE"])
const { categoryId, ...article } = yield* parseInput(input)
yield* enforceCategoryExists(categoryId)
yield* enforceArticleLimitInCategory(categoryId)
const persistedArticle = yield* persistArticle(article).pipe(
Effect.retry({ times: 1 })
)
return { status: 201, article: persistedArticle }
}).pipe(
Effect.catchTags({
"ArticleLimitExceededError": ({ name: id }) => Effect.succeed({ status: 400, errors: [{ id }] }),
"CategoryNotFoundError": ({ name: id }) => Effect.succeed({ status: 400, errors: [{ id }] }),
"ParseError": (e) => Effect.succeed({ status: 400, errors: [{ id: e.name, details: e.message }]}),
"UserUnauthorizedError": () => Effect.succeed({ status: 403 }),
"SqlError": (e) => Effect.die(e),
}),
exhaustiveErrorCheck
)
Let's focus on the business logic first. The program executes the following sequence of actions, any of which can fail in some way and short-circuit the flow:
-
authorize: makes sure that the user has the specified permissions needed to create an article. This action may fail with
UserUnauthorizedError. -
parseInput: takes some data of an unknown shape and parses it into data of an expected shape. I chose to type the input as
unknownto model the system boundary. The request body of an incoming HTTP request is usually a stream of characters. When this function receives the input, though, we expect it was already parsed partially, e.g. usingJSON.parsewhich usually returns an unsafeanytype. Typing it asunknownleads to better reliability by forcing us to properly parse the data. This action may fail withParseError. -
enforceCategoryExists: makes sure that
categoryIdpoints to an existing category. Assume that our domain requires all articles to be created under a specific category. This action may fail withCategoryNotFoundError. -
enforceArticleLimitInCategory: assume that our domain places a limit on how many articles can be placed in a given category. This function makes sure that this limit has not yet been reached before continuing with the rest of the flow. This action may fail with
ArticleLimitExceededError. -
persistArticle: attempts to save the article in our persistence layer. This action may be retried exactly once in case of failure. It may fail with a rather obscure
SqlError. In practice, this error could be modeled as a union of specific SQL errors to distinguish between retriable and non-retriable errors.
const createArticle = (input: unknown, currentUser: User) => Effect.gen(function*() {
yield* authorize(currentUser, ["READ", "WRITE"])
const { categoryId, ...article } = yield* parseInput(input)
yield* enforceCategoryExists(categoryId)
yield* enforceArticleLimitInCategory(categoryId)
const persistedArticle = yield* persistArticle(article).pipe(
Effect.retry({ times: 1 })
)
return { status: 201, article: persistedArticle }
})
The above excerpt of our earlier code example only deals with the happy path. In Effect, this would be a valid program that compiles. TypeScipt is fully capable of inferring the type of this program thanks to the use of generators8. The signature would look similar to the following:
type Success = { status: 201; article: Article }
type Failure =
| ArticleLimitExceededError
| CategoryNotFoundError
| ParseError
| UserUnauthorizedError
| SqlError
type CreateArticle =
(input: unknown, currentUser: User) => Effect<Success, Failure>
As we can see, Effect is extremely composable. TypeScript's inference capabilities coupled with Effect's composability help us write reliable software. It also provides flexibility in error handling. It does not force us to handle all expected errors. We have the freedom to handle as many or as few of the errors as we see fit. It even lets us enforce exhaustive error handling. To do just that, we can implement a simple helper function.
import type { Effect } from "effect"
export const exhaustiveErrorCheck = <
Success,
Requirements,
E extends Effect.Effect<Success, never, Requirements>
>(effect: E) => effect
The above is an identity function ((a) => a) that enforces that the input is an effect that can never fail. Let's see how to use this in practice.
pipe(
Effect.catchTags({
"ArticleLimitExceededError": ({ name: id }) => Effect.succeed({ status: 400, errors: [{ id }] }),
"CategoryNotFoundError": ({ name: id }) => Effect.succeed({ status: 400, errors: [{ id }] }),
"ParseError": (e) => Effect.succeed({ status: 400, errors: [{ id: e.name, details: e.message }]}),
"UserUnauthorizedError": () => Effect.succeed({ status: 403 }),
"SqlError": (e) => Effect.die(e),
}),
exhaustiveErrorCheck
)
This excerpt from our original code example demonstrates error handling which is visually and logically separated from the happy path with a pipeline9. Effect.catchTags10 is a way to define multiple error handlers at once. exhaustiveErrorCheck makes sure that all expected errors have a defined handler. If we were to remove the SqlError handler, the program would compile with the following error:
Argument of type '<Success, Requirements, E extends Effect.Effect<Success, never, Requirements>>(effect: E) => E' is not assignable to parameter of type '(_: Effect<{ status: number; article: { title: string & Brand<"ArticleTitle">; body: string & Brand<"ArticleBody">; id: string & Brand<"ArticleID">; }; } | { status: number; } | { ...; } | { ...; } | { ...; }, SqlError, never>) => Effect<...>'.
Types of parameters 'effect' and '_' are incompatible.
Type 'Effect<{ status: number; article: { title: string & Brand<"ArticleTitle">; body: string & Brand<"ArticleBody">; id: string & Brand<"ArticleID">; }; } | { status: number; } | { ...; } | { ...; } | { ...; }, SqlError, never>' is not assignable to type 'Effect<unknown, never, unknown>' with 'exactOptionalPropertyTypes: true'. Consider adding 'undefined' to the types of the target's properties.
Type 'SqlError' is not assignable to type 'never'.(2345)
As with most TypeScript errors, the key information is found at the bottom of the message: Type 'SqlError' is not assignable to type 'never'. This means we did not handle the case of SqlError. Additionally, exhaustive error checking also helps when new error cases are added, for example, by augmenting the persistArticle action to also emit a Kafka message and possibly fail with KafkaProducerError.
Notice that all errors, except for SqlError, are handled by resolving the effect successfully using Effect.succeed11. That does not mean that the article was created successfully. It simply means that the operation of creating an article is executed according to expectations. That is, when the user is not authorized to create an article, the function successfully prevents them from creating one.
The handler for SqlError resolves the effect using Effect.die12. If the code execution gets to this point, it means the retry of persistArticle failed. We could retry more times, add an exponential backoff schedule13 with jitter, etc. but if all attempts fail, we would still have to handle the no longer retriable SqlError. The only thing left for us to do at that point is to handle the case as an unrecoverable error14.
After we handle all expected errors, either with Effect.succeed or Effect.die, TypeScript's inference understands that the handled errors should no longer appear in the inferred type of the effect. The new signature of the createArticle function would look similar to the following:
type Success =
| { status: 201; article: Article }
| { status: 400; errors: ArticleError[] }
| { status: 403 }
type CreateArticle =
(input: unknown, currentUser: User) => Effect<Success, never>
TypeScript and Effect provide a world-class developer experience in terms of error handling. The IDE15 is our friend and suggests what error cases to handle as we implement the block of code in Effect.catchTags.
Configuration Management
All programs run in a certain environment. Configuration can be viewed as the API between the environment and the program. The program declares its requirements, and the environment tries to satisfy them. This establishes a consumer-provider relationship between the program and its environment. This fact has a serious reliability implication in practice. A program and its environment can change independently of each other. Since the requirements are defined in the program but the configuration values live in the environment, the possibility of the two drifting, especially in complex systems, is very high. That can, and in practice often does, cause service incidents. To prevent that from happening, we have to carefully design our software as well as the deployment pipeline. Should the environment not fully satisfy the program's requirements, its deployment must fail; keeping the healthy version of the software and its environment running. Effect can help us design such software.
Effect is fully aware of this dual nature of managing configuration. This is why it models it as such. The Config module implements the tools necessary to declare the configuration to be consumed by the program, while the ConfigProvider module implements the tools necessary to provide, or extract, the configuration from the environment (environment variables, file system, network,...).
The following example is a simple program declaring some configuration it requires to run correctly.
import { Config, Effect } from "effect"
import { MysqlConfig, RuntimeEnvConfig } from "./config"
const program = Effect.gen(function*() {
const config = yield* Config.all({
mysql: MysqlConfig,
port: Config.number("PORT").pipe(
Config.withDefault(3000)
),
runtimeEnv: RuntimeEnvConfig("RUNTIME_ENV")
})
yield* Effect.log(config)
})
Notice, that the code in the example does not say anything about how to extract the configuration from the environment. It does not say if they should be read from the environment variables, the file system, or some other place. That is the responsibility of the ConfigProvider. Let's see how that fits in the picture by demonstrating how to run this program.
import { NodeRuntime } from "@effect/platform-node"
import { program } from "./main.js"
NodeRuntime.runMain(program)
By default, Effect uses its built-in ConfigProvider that extracts the configuration from the environment variables. We can fully modify this behavior, and we'll do so very soon. But first, let's take a look at the errors produced by this program when the environment does not satisfy the requirements.
Error: (Missing data at MYSQL.USERNAME: "Expected MYSQL_USERNAME to exist in the process context")
and (Missing data at MYSQL.PASSWORD: "Expected MYSQL_PASSWORD to exist in the process context")
and (Missing data at MYSQL.HOST: "Expected MYSQL_HOST to exist in the process context")
and (Missing data at MYSQL.PORT: "Expected MYSQL_PORT to exist in the process context")
and (Missing data at RUNTIME_ENV: "Expected RUNTIME_ENV to exist in the process context")
I formatted the above message to slightly improve its readability. As we can see, the following environment variables are missing: MYSQL_USERNAME, MYSQL_PASSWORD, MYSQL_HOST, MYSQL_PORT, RUNTIME_ENV. Notice that Effect collects all unsatisfied configuration errors, instead of failing fast. This massively improves the developer experience when faced with multiple missing environment variables. What we can also notice is that the program does not complain about the PORT configuration because it was instructed to fall back to a default value in case it was missing.
So, what if we wanted some of the configuration to be read from the file system instead of the environment variables for the reasons of enhanced security? We can do so by providing an additional ConfigProvider that will read the configuration from the file system if it's not available in the form of an environment variable.
import { PlatformConfigProvider } from "@effect/platform"
import { NodeContext, NodeRuntime } from "@effect/platform-node"
import { Effect } from "effect"
import { program } from "./main.js"
program.pipe(
Effect.provide(PlatformConfigProvider.layerFileTreeAdd({
rootDirectory: "/secrets"
})),
Effect.provide(NodeContext.layer),
NodeRuntime.runMain
)
If we run the program again without satisfying the requirements first, we'll see the following error.
Error: (Missing data at MYSQL.USERNAME: "Expected MYSQL_USERNAME to exist in the process context")
or (Missing data at MYSQL.USERNAME: "Path /secrets/MYSQL/USERNAME not found")
and (Missing data at MYSQL.PASSWORD: "Expected MYSQL_PASSWORD to exist in the process context")
or (Missing data at MYSQL.PASSWORD: "Path /secrets/MYSQL/PASSWORD not found")
and (Missing data at MYSQL.HOST: "Expected MYSQL_HOST to exist in the process context")
or (Missing data at MYSQL.HOST: "Path /secrets/MYSQL/HOST not found")
and (Missing data at MYSQL.PORT: "Expected MYSQL_PORT to exist in the process context")
or (Missing data at MYSQL.PORT: "Path /secrets/MYSQL/PORT not found")
and (Missing data at RUNTIME_ENV: "Expected RUNTIME_ENV to exist in the process context")
or (Missing data at RUNTIME_ENV: "Path /secrets/RUNTIME_ENV not found")
Notice that the error is more verbose now. We can see that it tried looking up the configuration both in the process context (environment variables) and the file system.
Let's see what the output looks like when we do satisfy all the requirements. There is one small problem, though. The example runs in a closed sandbox environment. We cannot provide any environment variables, nor can we write to the file system. Don't worry because Effect got us covered! We can provide the configuration explicitly from the code itself using ConfigProvider.fromMap.
import { PlatformConfigProvider } from "@effect/platform"
import { NodeContext, NodeRuntime } from "@effect/platform-node"
import { Effect } from "effect"
import { program } from "./main.js"
program.pipe(
Effect.provide(PlatformConfigProvider.layerFileTreeAdd({
rootDirectory: "/secrets"
})),
Effect.provide(NodeContext.layer),
Effect.withConfigProvider(
ConfigProvider.fromMap(
new Map([
["MYSQL_USERNAME", "abcde"],
["MYSQL_PASSWORD", "12345"],
["MYSQL_HOST", "dummy host"],
["MYSQL_PORT", "3306"],
["RUNTIME_ENV", "development"],
]),
{ pathDelim: "_" }
)
),
NodeRuntime.runMain
)
I might be a bit ahead of myself but this way of providing configuration significantly improves the testability of all application code written in Effect. Notice how all the values supplied are strings. The Config module knows exactly how to parse and validate configuration of any complexity. This is what the output looks like:
{
mysql: {
username: 'abcde',
password: <redacted>,
host: 'dummy host',
port: 3306
},
port: 3000,
runtimeEnv: 'development'
}
Summary
Let's review the key points we learned about reliability:
- Reliability is a measure of correctness.
- Correctness is crucial for software users, as it enables them to achieve their objectives.
- When software operates correctly (as expected), it reduces customer churn, leading to more customers with higher lifetime value16.
- Escaped defects and configuration errors are the primary causes of service incidents.
- Effect's error management and configuration management features help build more reliable software.
- The unique combination of TypeScript’s inference and Effect’s composability provides a clear view of edge cases in any part of the program and equips us with the tools to handle them efficiently.
- Configuration errors arise due to the inherent disconnect between where configuration values reside (the environment) and where configuration requirements are defined (the program).
- Effect’s configuration management is designed to address this disconnect.
Even with Effect’s enhanced reliability guarantees, escaped defects can still occur. These may result from unexpected errors not captured in our type system or from a misalignment between our assumptions about customer expectations and reality. That’s why robust observability practices are essential — they provide visibility into these issues.
In the next post, we’ll explore observability and how it helps monitor both errors and software performance. See you there!
Top comments (0)