After just under 2 weeks of work; I finally reached the first major milestone for Go-DOM.
Now, the browser will download and execute remote JavaScript when building the DOM tree.
A Brief History
The project started as a crazy idea; seeing that Go and HTMX is a stack that is gaining some popularity;
Go already has all the tools you need to test pure server-side rendering. But when adding a library like HTMX, the behaviour of the app is the result of a choreography between the initial DOM; the attributes on interactive elements; the HTTP endpoints reached, and the content delivered by those endpoints; both response headers and body. To verify the behaviour from the user's point of view, you need a browser; or at least a test harness that behaves ... not entirely unlike a browser.1
Searching for "headless browser in go" only led to results suggesting to use a real browser in headless mode. This combination that has a huge overhead; discouraging the fast and efficient TDD loop. Relying on a real browser will typically slow you down instead of speed you up.2
So the idea was sparked; write a headless browser in pure Go as a testing tool for web applications;
The first uncertainties to address was the parsing of HTML; as well as script execution. I managed to quite quickly; within 2 days to address both of these. I had a very rudimentary HTML parser; as well as I had integrated v8 into the code base3 and make Go objects accessible to JavaScript code.
The HTML parser was later removed, as go x/net/html already implements an HTML parser; dealing with all the quirks of HTML parsing. Parsing a well-formed document isn't a terribly difficult problem to solve. It's gracefully dealing with malformed HTML where it becomes tricky.
After some time, I also managed to get inline script execution to run at the right moment; i.e. the script is executes when the element is actually connected to the DOM, not after the full HTML has been parsed.
Working with HTTP requests
After being able to process an HTML document with inline script; the next step was to actually download scripts from source. This needed to integrate an HTTP layer; such that the browser fetches content itself; rather than being fed content.
The http.Client also allows you to control the actual transport layer using the http.RoundTripper interface. Normally you start a server; which will listen for requests on a TCP port. In this context TCP serves as the transport layer; but is itself irrelevant for the processing of HTTP requests. Due to simplicity of the standard HTTP stack in Go; an entire HTTP server is represented by a single function, func Handle(http.ResponseWriter, *http.Request).
The headless browser can completely bypass the overhead of the TCP stack and call this function directly using a custom RoundTripper.
Now the browser can perform HTTP requests, but the browser code itself is ignorant of the fact that the HTTP layer is bypassed. And with this came the ability to download the script during DOM parsing.
Example Code
Let's explore a simple test, as it looks now in the code base (the code uses Ginkgo and Gomega, a somewhat overlooked combination IMHO)
First, the test creates a simple HTTP handler which serves two endpoints, /index.html, and /js/script.js.
It("Should download and execute script from script tags", func() {
// Setup a server with test content
server := http.NewServeMux()
server.HandleFunc(
"GET /index.html",
func(res http.ResponseWriter, req *http.Request) {
res.Write(
[]byte(
`<html><head><script src="/js/script.js"></script></head><body>Hello, World!</body>`,
),
)
},
)
server.HandleFunc(
"GET /js/script.js",
func(res http.ResponseWriter, req *http.Request) {
res.Header().Add("Content-Type", "text/javascript")
res.Write([]byte(`var scriptLoaded = true`))
},
)
// ...
The intention here is merely to verify that the script is executed. To do that, the script produces an observable side effect: It sets a value in global scope.
To verify that the script was executed is just a matter of examining the global scope, which is done by executing ad-hoc JavaScript from the test itself; verifying the outcome of the expression.
The code to create the browser, load the index file, and verify the observed side effect
browser := ctx.NewBrowserFromHandler(server)
Expect(browser.OpenWindow("/index.html")).Error().ToNot(HaveOccurred())
Expect(ctx.RunTestScript("window.scriptLoaded")).To(BeTrue())
Test execution is also pretty fast. The part of the test suite involving JavaScript execution currently consists of 32 tests running in 23 milliseconds.
Next milestone, integrate HTMX.
As the project was initially conceived while trying to verify an HTMX application, a reasonable next goal is to support just that case. A simple HTMX application with a button and a counter, which increases when the button is pressed. This will drive the need for:
- An
XMLHttpRequestimplementation needs to be in place. Work is underway for that. - An
XPathEvaluator. I believe this can be poly-filled to begin with. - Event propagation. Only
DOMContentLoadedandloadevents are emitted right now. Elements need to support more events; such asclick; as well as methods to trigger them.- This might also require proper event capture and bubbling.
And then ...
Following that, more advanced user interaction; proper form handling, e.g., input handline (e.g., pressing enter in an <input> field submits the form. This typically also involves some kind of URL redirection; which drives the need for a history object, etc.
Integration external sites
With the ability to control the transport layer; we can provide tests with unique abilities; we can mock out external sites that the system would depend on at run-time. E.g., for a given host name, the test could provide another Go HTTP Handler simulating the behaviour.
The most obvious example is the use of external identity providers. The test could simulate the behaviour of a login flow; not having to force you to create dummy accounts in an external system, have test failures because of outages in an external system, or simply being unable to automate the process at all because of 2FA or a Captcha introduced by the identity provider.
Another use case is the use of API-heavy libraries, like map libraries, that incur a usage cost. Mock out the external site to not receive an extra bill for running your test suite.
But given the fact that you can pass a custom http.RoundTripper; it should be easy enough to create an implementation that forwards the call depending on the host.
Usability over Compatibility
Creating a 100% whatwg standards compliant implementation is quite an endeavour; I didn't fully comprehend the scope until I actually started reading parts of the whatwg specifications. The goal is to create a tool helping write tests for web applications. Full compatibility is a long-term goal; but until the project has reached some level of usability, I will not start filling out holes of missing compliance.
For that reason; features that are more likely to be used in actual applications are more likely to be prioritised. A feature request pointing to an actual test giving the wrong result is likely to be prioritised. A feature request for the implementation of a specific standard is likely to be rejected.
Spread the word
I believe this can be a very useful tool for many developers, so if you read this, let your coworkers know it exists. This is so far a spare time project, of which I have plenty of at the moment; but that will not be the case forever.
If you want to see this live, spread the word ...
Perhaps you would even sponsor this? Do you have a large company building web applications in Go? Feel free to contact me.
Find the project here: https://github.com/stroiman/go-dom
-
Kudos if you managed to catch the homage to a popular BBC radio play. ↩
-
This is based on personal experience. Doing TDD right will speed you up due to the fast feedback cycle. The overhead of a real browser tends to make you write tests after the production code; loosing their benefit of the feedback loop a fast test suite gives you. ↩
-
Groundwork had already been laid by the v8go project. However; not all features of v8 are exposed to Go code; including necessary features for embedding native objects. I was able to add those in a separate fork; which is still WIP. ↩
Top comments (1)
Absolutely incredible! 😻.