Building a custom HTTP server from scratch

I've built APIs and played around with virtual machines for a long time now, and if you've done similar things
You know It's always followed by some protocol: http, web-sockets, ssh, ...

So today I wanna breakdown some mysteries on how things work behind the scenes just a little bit. Covering one of the most used protocols, HTTP

For context: a "protocol" is just like a dialect between computers, so different systems can talk in a common manner
.

References (sources you may wanna check after reading this):

• High Performace browser networking (book)
• HTTP 1.1 RFC (documentation on how HTTP is implemented)
• Github source code (implemtation in golang)

.

Inside the engine - TCP

Now networks are unreliable, data may be lost in between, or take too much time to do so.
TCP was built for this, to provide a reliable connection on a unreliable network

Suppose there's two devices on a network, these devices have some address (their IP), and a process running (on a port), that's all you need for TCP.

It let's the devices:

• send data in order (in chunks called "segments") --> like a queue.
• if any data is load, retransmission is performed (ensures data reaches)

How do devices know if the other has received or not? Simple, just send an "ACK" (acknowledgement) to confirm, if there's no response after some time, assume it is lost.
Anyways, this in itself is a large topic, so we'll get to HTTP

Now, like HTTP, TCP also has underlying client-server model

And here's how the process kind of turns out to be:

1. So let's say server has an address (2.2.2.2) & running a TCP service on port (3000).
2. Client sends a TCP request here, first these devices has to do a "handshake" process (before they start actually sending data)
3. The handshake process goes as follows: client --> server(ACK) --> back to client (they exchange info needed during actual transmission of data later).
4. Once client receives data back from server, it starts sending data to server.
5. Every chunk of data after that, is a client --> server and followed by an ACK received back.

.

A look at HTTP

Now we've understood TCP a bit, let's look at the data being sent after the handshake.

HTTP is nothing more than just a format on how data should be present (same goes for most protocols).

Example:

When you use something like express, fastapi, flask to make a backend service, their implementation may be different but all of them expect the request in this format!

So the idea is to parse them in order that they are supposed to be and process them, then send a response, which also is in the same format.

Try using the following command in your terminal/command prompt (check if you have wget or not, most systems always has it beforehand):

 wget --server-response --debug --post-data="test" --header="Content-Type: text/plain" https://jsonplaceholder.typicode.com/posts

This will retrieve some json responses.

Note: In the example, it's a simple "Hello world" (a string), but you can send a lot more like: html, json, assets, blobs, etc

.

Honestly, these are more than enough to start building a HTTP server.
Like I said, the implementation can be different, but the protocol is the same, so here's some guide I'll give you to follow:

1. First try establishing a TCP connection, send some dummy data back & forth (you'll need a TCP server & a TCP client - check your language's docs - ex: go - net & nodejs - Net)'

2. Seperate into modules & test indivually:

   • "Parser" --> breaks down request into info, header & body.
   • "Router" --> like express, fastapi, or any server side framework handles "routing" i.e: check type of request (POST, GET) & path to assign a callback function
   • "Composer" --> after processing the callback function, get response from the handler (callback function), and respond to client.

.
This was just a basic overview, this is definitely not something you wanna be using in prod.

But making things from ground up does give you a really good perspective on how things are, and change the way you look at problems, so you know not only how to do it, but also why.

You can follow my implementation at Github (it's in golang, but still you can use as reference to implement in your favourite language).

(There's more to be added in the future in this implementation probably, so I'll share updates here if I do make some changes)