Spore Drive: Building a Cloud Platform in a Few Lines of Code

Building your own cloud platform has many benefits, including cost savings, ownership, sovereignty, and most importantly, the ability to make your solution self-hostable. However, constructing a cloud computing platform is a long, complex, and expensive endeavor—a problem partially solved with Tau. Why only partially? Simply because, though Tau is a simple-to-deploy single binary, you still need to get that done, configure Tau, and ensure dependencies like Docker are installed. Not to mention, you have to repeat this on each host for every Tau update. If you ask me, it's not developer-friendly just yet!

Introducing Spore-Drive

One option is using Infrastructure as Code (IaC) tools like Terraform or Pulumi, which come with their own complexities and learning curves—not to say these aren't developer-friendly tools. What we need is a tool that's specific to Tau, much like macOS is specific to a MacBook, delivering the best results and user experience. This is why I built Spore-Drive, a tool that allows you to build, expand, and keep a cloud up to date with just a few lines of code.

The idea is that given a configuration—which is generated or edited with code—we can determine and execute a deployment strategy. The first part relies on the concept behind go-seer, which essentially treats a folder containing YAML files as a database. This helps keep the state in a human-readable format, and we can also commit it to a Git repository. The second part consumes the configuration and deployment parameters to build a deployment plan (called a course) and then executes it, resulting in the deployment of Tau.

Architecture

Let's consider our goals:

• Support for multiple programming languages
• Web-ready
• Enterprise in mind
• No full rewrite for each language

With these goals in mind, we need to have Spore-Drive run as a service, with clients written in many languages. Because it has to be web-ready, we'll use ConnectRPC to implement the API.

The core can be written in Go—not only is it my language of choice, but many of the components we'll be using, like go-seer and mycelium, are written in Go. Moreover, we intend to have Spore-Drive be part of Tau's CDK, which means it needs to be embedded in Tau as well.

Using Spore-Drive

As of this article, I have only managed to create a TypeScript client. You can use Spore-Drive directly with Go, but a client is next on my list. So for this article, I'm going to use the TypeScript package.

Context

I have created droplets on DigitalOcean, and I'd like to build a cloud with the main domain being pom.ac, hosted by Namecheap.

I need to fetch droplet information to create or update my configuration, deploy with Spore-Drive, and then update my DNS records on Namecheap.

Create a Project

Let's create a TypeScript project:

$ npm init -y
$ npm install typescript tsx --save-dev
$ npx tsc --init

Edit your tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "ESNext",
    "moduleResolution": "node",
    "declaration": true,
    "outDir": "dist",
    "rootDir": ".",
    "strict": true,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "skipLibCheck": true,
    "lib": ["ES2020"]
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules"]
}

Add the following to package.json:

"scripts": {
    "displace": "tsx src/index.ts"
},

Finally, install Spore-Drive:

$ npm install @taubyte/spore-drive

Create a new file src/index.ts and import Spore-Drive:

import {
  Config,
  CourseConfig,
  Drive,
  TauLatest,
  Course,
} from "@taubyte/spore-drive";

Configuration

Start by creating a configuration:

const config: Config = new Config();

If you'd like to open or sync your configuration to disk, pass a path like so:

const config: Config = new Config(`${__dirname}/../config`);

Then, initialize it:

await config.init();

This will have the Spore-Drive service create an instance of your configuration.

A cloud built with Tau requires some minimal configuration:

• Root domain: Identifies the cloud. For me, it's pom.ac.
• Generated domain: Can also be a subdomain.
• Domain validation key
• Swarm key

Let's start with these:

export const createConfig = async (config: Config) => {
  const domain = config.Cloud().Domain();
  await domain.Root().Set("pom.ac");
  await domain.Generated().Set("g.pom.ac");
  await domain.Validation().Generate();
  await config.Cloud().P2P().Swarm().Generate();
}

We need to define a signer for SSH. Here, I'm using username and password, but you can also use an SSH key.

export const createConfig = async (config: Config) => {
  // Previous code...
  const mainAuth = config.Auth().Signer("main");
  await mainAuth.Username().Set("root");
  await mainAuth.Password().Set(process.env.DROPLET_ROOT_PASSWORD!);
}

Hosts can run multiple instances of Tau; each is called a node, and each node can run multiple services grouped under what we call a shape. Here, I'm defining a shape called all, which includes all services except gateway.

export const createConfig = async (config: Config) => {
  // Previous code...
  const all = config.Shapes().Shape("all");
  await all
    .Services()
    .Set(["auth", "tns", "hoarder", "seer", "substrate", "patrick", "monkey"]);
  await all.Ports().Port("main").Set(BigInt(4242));
  await all.Ports().Port("lite").Set(BigInt(4262));
}

Next, we're going to add our hosts. Here, I'm using a helper (see do.ts) that lists all my droplets. If the host does not exist, I add it:

const bootstrapers = [];

for (const droplet of await Droplets()) {
    const { hostname, publicIp, tags } = DropletInfo(droplet);
    if (!hosts.includes(hostname)) {
      const host = config.Hosts().Host(hostname);
      bootstrapers.push(hostname);
      await host.Addresses().Add([`${publicIp}/32`]);
      await host.SSH().Address().Set(`${publicIp}:22`);
      await host.SSH().Auth().Add(["main"]);
      await host.Location().Set("40.730610, -73.935242");
      if (!(await host.Shapes().List()).includes("all"))
        await host.Shapes().Shape("all").Instance().Generate();
    }
}

await config.Cloud().P2P().Bootstrap().Shape("all").Nodes().Add(bootstrapers);

Note that I'm adding new hosts to bootstrapers because the next step is making sure all these hosts can bootstrap each other. This is not a requirement—this example has two droplets—but in a larger setup, you'd like to have bootstrap nodes that do not run any service.

Finally, we commit the changes:

await config.Commit();

Add the call to the createConfig function after initialization:

await config.init();
await createConfig(config);

Drive

Now that we have our configuration loaded, we can create a Drive. It takes a reference to our configuration instance and an optional Tau version or binary. Here, I'm going to make sure hosts are running the latest Tau:

const drive: Drive = new Drive(config, TauLatest);
await drive.init();

The next step is to plot a course:

const course = await drive.plot(new CourseConfig(["all"]));

Spore-Drive will determine the best strategy to deploy the latest version of Tau, while installing dependencies and making sure configuration for each node is generated.

To start the deployment:

await course.displace();

This starts the deployment (or displacement, as I call it). The process is run on the service side and can be queued (in the future).

To get updates on the process, you can call progress, which will feed you a progress stream:

for await (const displacement of await course.progress()) {
  // Handle progress updates...
}

Each element of the stream is DisplacementProgress and has:

• path: /<course name>/<host name>/<task name>
• progress: 0 to 100
• error: Error message, if any

Check out displayProgress if you'd like to display the progress as progress bars.

Use your cloud!

Your next step is to use your newly deployed cloud. Check tau.how for more details on that.

Full Code

The full code for the pom.ac cloud can be found here.

In action

What's Next?

In my effort to close the gap between software development and cloud computing, I'm planning to build a similar interface for taucorder. Ultimately, both will become part of the Tau CDK.