Just months ago the Commons Host static hosting CDN launched with a single edge server. Today there are over 20 edge servers around the world. The majority are inexpensive ARM-based micro servers hosted by volunteer contributors on commodity Internet connections, often Gigabit fibre. Others are virtual machines in cloud data centres which offers similar performance.
Illustration: Map of Commons Host CDN edge servers (live & WIP)
Because We Can
Kenny and I worked diligently on deployment automation. This allowed scaling the edge server count from single to double digits.
With these tools in place, we decided to build and deploy a completely new service in parallel on the same edge server network.
We chose to implement DNS over HTTPS, or DOH for short. DNS resolving is perfectly suited to the advantages and constraints of the Commons Host server network. Low latency due to global coverage, and minimal hardware requirements.
DNS, meet HTTP. HTTP, meet DNS.
Building an HTTP CDN requires learning about the Domain Name System (DNS). DNS is ancient by Internet norms; many years older than the World Wide Web or HTTP.
Standards like HTTP or DNS are the work of the Internet Engineering Task Force. This organisation provides an open, vendor-neutral discussion platform through public mailing lists. IETF also runs conferences, held 3 times yearly, rotating through the Americas, Europe, and Asia.
Photo: A session at the IETF 100 conference
At the 100th IETF conference, in Singapore where we live, a draft called DNS-over-HTTPS was presented and intensely debated. Attendees packed the conference hall. This was a meeting of worlds between DNS and HTTP experts. Even DOH's authors are respected leaders from both DNS (Paul Hoffman, ICANN) and HTTP (Patrick McManus, then Mozilla now Fastly).
I was fortunate to attend IETF 100 last year. The humbling experience left a deep impression. Implementing DOH would also be a personal tribute to this community.
How Hard Can DOH Be?
HTTP servers exist. DNS servers exist. So we just duct tape the two together? Well, basically, yes.
Driven by curiosity, Kenny wrote the first DOH implementation while brushing up on Node.js and reading the DOH draft specification, tabula rasa.
Over the next few days we rewrote and refactored the code. In the end we built a middleware called for Node.js web servers called Playdoh.
Screenshot: Playdoh GitHub repository
Playdoh relays raw UDP messages between a DOH client like Firefox and a traditional DNS server. Playdoh is 150 lines of DOH duct tape, with 300 lines of tests to make sure it sticks.
Deploying a DNS Resolver
To offer a global DOH service, each edge server needs to run its own caching DNS resolver. A resolver processes the DNS query and caches responses so that users benefit from faster future lookups.
We learned of Knot Resolver by talking to friends in the CDN industry. Knot Resolver is open source software developed by the Czech Republic DNS registry (CZ.NIC). Fun fact, Knot Resolver also powers the Cloudflare 1.1.1.1 public DNS service. Others recommended Unbound or BIND as resolvers. We may yet run those in a mixed network for heterogenous resilience.
It took a few days to tune the Knot Resolver configuration and automate its deployment. This involved remotely upgrading the operating system across all edge servers. A risky proposition involving custom vendor kernels for the ARM servers. With overseas unattended physical machines there is no option to press a reset button or flip a power switch. The only solution was to perform tireless careful testing in staging environments, using Vagrant/Virtualbox or on spare hardware. Eventually we ironed out subtle differences between the various server configurations.
This was much more time consuming and technically challenging than coding Playdoh. Our knowledge and experience continues to grow, as documented in the merge request description. Next time this will be easy.
So how is DNS traffic served by a DOH CDN?
Bootstrapping DOH CDN: Anycast IP vs Geo DNS
Users need to be able to easily configure their DNS settings and connect to a nearby DNS server for low-latency lookups.
Traditional public DNS services make use of an expensive Anycast IP network. Users are routed to one of many edge servers worldwide. They share the same IP address but announce different routes using BGP at Internet Exchanges. ISPs will route users via the shortest path. Sadly this is not easily accessible due to cost and administrative overhead.
They also opt for memorable IP addresses. Google owns 8.8.8.8, Cloudflare owns 1.1.1.1, Quad9 (IBM) owns 9.9.9.9, and so on. Their IPv6 addresses are less human friendly but the principle is the same.
With DOH the DNS resolver address is a familiar URL instead of an IP address. This URL contains a domain name so that the connection may be secured using a signed TLS certificate. E.g.: https://commons.host
So DNS itself is used to direct traffic to a DNS over HTTPS service. Chicken or egg? Not quite.
DOH works by bootstrapping the initial DNS lookup of the resolver's hostname. This DNS lookup is still handled by traditional DNS servers like those of an ISP or a local server. The HTTPS connection is then secured with a signed TLS certificate for that domain. Any tampering by a malicious (or faulty) DNS server at the ISP would simply result in a failed connection attempt. So there is no risk of exposing to the DOH client to tampered responses.
Diagram: DOH bootstrap sequence
Bootstrap procedure:
- Browser performs DNS lookup for the Commons Host DOH server hostname using a standard, potentially untrusted, DNS server.
- DNS server responds with the IP address of the nearest Commons Host edge server.
- Browser establishes a HTTP/2 connection with the edge server. TLS certificates ensure an encrypted and authenticated connection.
- Subsequent DNS lookups are tunnelled inside the HTTP/2 connection to keep them safe from snooping or tampering by third parties.
* Shown IP address is an example. Actual address is based on location and other performance metrics to determine the optimal edge server for a particular user.
What About Security?
Running a public DNS service is typically fraught with security problems. An open resolver, one that accepts DNS queries from anyone on the Internet, is a convenient traffic-amplifier for DDoS botnets and other malicious actors.
Amplification attacks work by spoofing the source address on a small DNS query, so that the large DNS response gets delivered to an unfortunate target. Attackers use public DNS servers to generate a multiple of their own bandwidth and aim it at a target while hiding themselves as originators. Most people would never want to run such a service, and in fact many ISPs block inbound UDP traffic on port 53 for this reason.
DOH eliminates the spoofing problem. The HTTPS connection requires a secure handshake so traffic can not be spoofed or misdirected. Any responses are always delivered to the correct source, making DOH safe from amplification attacks.
Living on the Edge
Running a public DOH service is much easier than a traditional DNS open resolver. Expect many organisations to offer such services.
One benefit of the Commons Host network is that anyone can sponsor and host an edge server. This brings the CDN edge on-premises, and a great way to run a sub-millisecond latency (i.e. LAN) DOH server. Doing so improves your DNS lookup speed while serving your local community. Get in touch if you are interested.
Using Commons Host DOH
Currently Firefox is the easiest way to use DOH.
Screenshot: Firefox Network Settings for DOH
- In the Preferences screen, open the Connection Settings dialog.
- Turn on the checkbox: Enable DNS over HTTPS
- Enter the URL:
https://commons.host
More DOH browser/OS support and bridging solutions will hopefully follow soon. Chrome seems to have a DOH implementation on the way.
One Last Thing: Custom Domain DOH Resolver
Commons Host supports DOH service on custom domains!
Deploying a custom domain on Commons Host is as easy as pointing a CNAME DNS record to commons.host using your domain name management provider.
Commons Host uses Geo DNS to point the domain commons.host to the edge server best suited for any user worldwide. Every edge server has its own public IP address. The edge web servers run the Playdoh middleware which processes DOH requests based on HTTP headers, while regular web requests pass through. The same domain and the same edge server can serve both web and DOH traffic.
Simply deploy a website with a custom domain and use your personal URL as the DOH resolver endpoint. For example: https://www.$yourdomain.com
The same goes for choosing a specific Commons Host edge server as your DOH endpoint. Each one is directly addressable by country code, airport code, and incrementing counter. For example: https://us-lax-1.commons.host connects directly to the Los Angeles edge server.
Top comments (11)
There are polyfills but they are non-trivial. Huge size and too many edge-cases that defeat the purpose of open standards based development.
FWIW the dashboard is very limited in functionality at the moment. You may be better off using the CLI toolchain. help.commons.host/cli/
The development flow is typically: design/docs -> api -> cli -> web.
Are you on Firefox? The Commons Host dashboard uses Web Components, a standard that is only supported by the very latest versions of Firefox. Try Firefox 63 or later. Currently that would be Beta/Developer (63) or Nightly (64). IIRC the same requirement applies for DNS over HTTPS.
See: twitter.com/justinfagnani/status/1...
PS: I find Web Components are great to work with as a frontend developer and well supported by Chrome and Safari. Edge just announced their implementation is in-progress.
Interesting. That's probably not a Firefox issue since it looks like this for me with 62.0.3 (current stable for MacOS from mozilla.org).
I attempted to avoid loading a web font and constructed a CSS font stack that would use a native condensed ("narrow") type face on as many platforms as I could test.
Which OS/WM are you using? Is there any font installed on your system, by default, with a condensed or narrow variant?
Congratulations on your work! This platform is Super cool! Keep doing well, and i got some errors while deploying prnt.sc/l4rpj3
Hi Yoginth! Thank you, that means a lot. :)
Let me take a look, should be a simple fix on my end. (Famous last words...)
Update: Should be fixed now. Disabled auto-renewal of the API server's certificate a while ago. Just did a manual renewal. Will automate this asap.
Sorry to Say! Still it's not working :( prnt.sc/l4v71q
Ah, that's a different issue. The deployment actually was successful. I see the static files in the backend (object store). But the API call from the CLI timed out because processing and deploying the files took too long.
So while the API was doing its work, and taking a minute or so, the CLI client got impatient and showed you a
FetchError.I am planning to replace the synchronous deployment API with an asynchronous design. That should prevent this confusion.
See: gitlab.com/commonshost/core/issues/12
Thanks for the feedback btw! Really helps me focus on solving these friction issues. ☺️
PS: If you deploy Markdown (
README.md) it is not automatically converted to HTML. Run a local build (or in CI) before deploying.Sure, would love to learn how it goes. Maybe we can connect your nodes to Commons Host DOH somehow? Been thinking of a way to build a secure global DOH network using volunteer hosted RasPi-like devices.
Email: sebdeckers83@gmail.com
Thanks, I'll check it out. Second Arch user showing me this font issue this week.
@artemix Follow up: I think the fonts should be slightly improved on Arch now. Worked for the other person. Their Arch system includes "Nimbus Sans Narrow" and "DejaVu Sans Condensed" fonts.
Link: help.commons.host
I just saw your GitLab and the ACDN project. Very nice! Want to DM about it via Twitter or email?