How to build a Web IDE like codesandbox?

#webdev #node #react #vscode

Introduction

I first came across Web IDE’s during my college days through online competitive coding platforms like leetcode and hackerrank, and since then the concept of web based IDE’s or Editors has always intruded me and this being relatively very niche topic I could not find much resources on its workings. However by the end of my final year I came up with a prototype web IDE that lets you build react applications in the browser, the prototype had a file explorer, working terminal, code editor and a live preview window. Back then i was too embarrassed about my code quality and architecture that i did not post or open-source the project. Fast forward to today, I revisited the project again, laughed at my mistakes and ended up improving the IDE to my best knowledge and now i have decided to open-source it. If you find my content interesting, leave a 💖 and follow me for more.

Features:
Code editor (Monaco)
File explorer
Xterm.js terminal
LSP (Language server protocol) Support

Tech stack:
Node.js
Docker
React

Available Implementations
Before I explain my implementation of online code editor, I would like to mention few other available approaches/websites such as:

VS Code.dev: Core VS Code is open-source (other than extensions marketplace) and since it is built with web technologies like HTML, CSS and JS through electron framework, all most all of the editor front end is compatible to run inside a web browser, but for editor backend it makes use of remote servers or docker containers.

StackBlitz: Uses WASM and WebContainers to power their code playgrounds that run entirely inside a web browser.

My Build:

Since I wanted to build (or at least assemble) my own editor VS Code wasn’t the approach that i was interested in. The StackBlitz is cutting edge and a great piece of tech but there is limitation to amount of stuff that can be done through WASM.

So my approach was to take all the available open-source tools and stitched them together to run as a web editor.

FrontEnd: I used monaco and xterm.js as my editor and terminal respectively (VS Code also uses these). I wrote a small recursive file explorer component for viewing, creating and deleting files/folders. The entire editor is responsive thanks to split.js.

The frontend is build with react and deployed as a static site inside my S3 bucket.

The link to frontend repo

Backend: The backend is written in node.js and it runs inside a docker container. The backend is responsible for serving project files and writing content to the files and also spawning a terminal instance for the user (with non root privileges).

The link to backend repo

Enough talk show me code…!

For Editor there are many react monaco wrappers out there, but after trying out a few of them the best one I found is from typefox called monaco-language client. The great thing about this is, it seamlessly integrates with almost all vs-code syntax highlighting extensions and also it provides a way to integrate Language servers over web sockets (this is the coolest part, more on this in later section).

The coolest part of this repo is that it strips down and makes use of existing VS Code extensions for providing language specific syntax highlighting (check line no 1). Checkout Monarch if you want to build syntax highlighting by yourself.

Also here you can see the golang lsp configuration, but what is LSP ?

LSP stands for Language Server Protocol. It’s basically a special language that lets code editors and development environments (like Visual Studio Code) to talk to language servers.

These language servers are like mini programs that understand the ins and outs of a specific programming language. The editor can ask the server questions about your code, and the server can provide cool features like auto-completion, finding errors, auto importing packages, or jumping to where a function is defined.

Before Microsoft introduced the LSP protocol the language developers used to write language support plugins for each and every editor out there, but now thanks to this protocol the language server is implemented once and used everywhere.

When a language support extension is installed in VS Code, it basically gets the syntax highlighting and sets up the language server. The VS Code runs the installed language server as a process and communicates through Json-RPC calls. But for our web editor we cannot run these language servers in browser, so we run the language server inside the docker container and transfer all the JSON-RPC calls made by the editor to the language server via a web socket connection. You can think of this as a way of piping | the RPC calls over sockets.

{
    serverName: "GOPLS",
    pathName: "/gopls",
    serverPort: 80,
    runCommand: "gopls",
    runCommandArgs: ["serve"],
 }

Checkout the backend repo for complete code.

Let’s talk about the terminal

VS Code uses Xterm.js for its integrated terminal. Xterm.js is a terminal emulator with rich plugin/add-on ecosystem. Since our entire user workspace is inside a docker container and only the front end is being served through web, we need a way to attach our Xterm.js terminal emulator to a process that in turn interacts with pty on docker.

To achieve this I made use of node-pty, xterm.js attach add on. The node-pty spawns a child processes that can interact with the operating system like you would on a real terminal, but we need a way to transfer the commands and responses from web to backend running inside the docker container, for this I made use of attach add-on for xterm.js. This works similar to the LSP setup i.e, each keystroke in terminal (xterm.js) is transferred to backend via socket connection and then its fed into the node-pty process and response generated in turn is brought back.

Backend code for terminal:

This concludes the terminal section of the web IDE.

Additional features:

We have a file explorer and a terminal, since files or directories could also be modified via terminal we need a way to communicate those changes to frontend. For this I made use of Chokidar, this package lets us setup file system watcher (nodemon also makes use of chokidar) and through following callbacks we can communicate file system changes.

The above piece of code listens for new file additions and sends a socket event to frontend so that UI stays in sync even if the files are being created from the terminal or any other process.

Security and deployment:

Through a web IDE we are basically giving people on the internet access to free cloud compute and chances of malicious actors taking advantage of such system is high. Since the backend is in docker, we could easily limit the CPU and Memory usage of each container and instead of giving root access to the user we can create a low privileged user and spawn the terminal through this user.

Dockerfile:

Run the docker container with command