didi-ops
Recently the didi repository has gone a bit quiet, this in part is due to the frustrations I have been having around monorepo tooling and the enormous headache that comes with stitching it all together. I had trouble with:
- developing typescript packages.
- linking the distributed code all together and keeping the versions in sync.
- preprocessing release output so I could vet the contents of package.json's
- adding boilerpalte without copy and pasting over and over and over (adding new packages became a drag)
I wont name names but let's just say even with the popular tools out there, this pipeline is hard to stitch together. What I wanted was a tool that could generate my packages with sensible defaults and know the distinction between libs and clients (functionality and consumers of functionality), an important architectural decision didi subscribes too.
I wanted to use just NPM scripts to do all the heavy lifting, but it became crazy complex pretty quickly, scripts all over the place, 9 or so individual scripts per package, a ton at the root as well. I then stumbled upon something tucked away in the NPM docs.
process.env.npm_lifecycle_event // magic
With this, I can detect the key of the NPM script and place a big switch statement in a single file, and so didi-ops was born. didi-ops has been quite a lot of work but the result is pretty great, let me show you.
In one package.json at the root of the didi mono-repo we have just the following scripts:
Some of you will know that $npm_package_config is just a package.json variable.
"config": {
"ops": "node ./ops/ops.js"
},
So you might be thinking, hold on, all those scripts point to the same file? Yep, I hid the mess in a massive node script 537 lines of dev-ops quality scripting (in other words, its enough to work). How then does one script do all this for any package I have added to didi or will add in the future?
Prompts, lots of prompts.
So how does it know what package is installed?
There is a .json manifest keeping track of clients and libs, this means that there can be several differences in the prompts and wizards depending on the package type.
Here I'll show you what sort of things are tracked.
But its not just tracking, didi uses template-dirs to generate all the files and folders a client or a libs package needs, all the test setup, the tsconfig, the package.json, README and more, even the comments inside these files.
Additionally bases files have been used where possible to inherit tsconfig and others, this keeps package configuration lean.
Docs
So now didi has its own package generator, is this enough to attract developers? I mean, it's about as easy as it gets to create new packages? No, this is not enough, because docs.
That leads me to the other reason I have been so quiet, I have been exploring static documentation generators, given that I can already generate the API Reference using typedoc, I really needed a more uniform way of integrating this with something I could write guides for. I tried Docz, It was promising but buggy and just took me to long to work with. you just know when something isn't working its time to stop. So I stopped and looked around again, I found Docusaurus 2. Firstly its fitting for a dinosaur like didi! But most importantly it was working very well out of the box! I had a lot more to show for myself in 5 mins of Docusaurus.
Docusaurus sadly doesn't really know the distinction between API References and Docs, so I had to do some acrobatics to make this work. Then tie this up to didi-ops and now I have a wizard to select which packages to build docs for, using typedoc and the markdown plugin.
Oodles of content!
(it's a bit rough around the edges, we have an issue open for that)
Is this work upstream yet? Not yet yes take a look at master branch, I have done so much of this work off of tickets and branches that I now need to get integrate this back into an issue, this will be a challenge, but it was worth it. Then I can finally say that didi's infrastructure is done and get some features sorted beyond its current feature set.
adam-cyclones
/
didi
Convert a project from common JS to ESmodules, with included bundler-like / task runner behaviour.
didi
A transpiler for JavaScript and Typescript, transforming CommonJS modules into distinct ES Modules
Who uses didi?
Frontend and deno developers will find didi useful.
Try it out.
didi is not ready for production however you can still take didi for a spin!
npm install -g @didi-js/client-didi-cli
# or
yarn global add @didi-js/client-didi-cli
Create a project an entry file, some installed node_modules and then require the browser dependency into your entry file using the base specifier.
const colorThief = require('colorThief'); // base specifier example, no paths needed
run didi path/to/example-project
The result should have output a new target directory within this example-project and also a server should have started on http://localhost:8086.
You may see some console errors in the browser, this is normal for this stage.
You may also notice that your import looks like this:
import colorThief from "color-thief"; // still no path?…
http://didi.land < website coming soonish.
progress report
Top comments (8)
Totally relate to pain of setting up a project from scratch figuring out the the pipelines for everything. What I have found helps is keeping a list of all technical debt. Acknowledge you have took a shortcut somewhere for reason X and move on. One day you will get back to it but not today, otherwise the little things can really bog you down! Having said that it looks like your keep track of work with issues in your repo which is cool.
I'm keeping an eye on this project, it looks really promising. would love to help out even if it is just docs. On that note if you're still evaluating documentation tools I've been hearing good things about VuePress which is a static site generator designed for writing documentation vuepress.vuejs.org/
Thank you Alex, yes tracking in issues has been enormously helpful, I can divide my thoughts and put everything upfront in the public eye.
I saw VuePress, it also looks good, I am a bit agnostic about the frameworks but I figure that React would be a good place for a wide range of developers to help out with Docusaurus, even if Vue is a little easier. The main reason was that the Typedoc markdown plugin had the support for Docusaurus 2 which meant that I could get all the routing generated without having to do any further haccking.
There is a new challenge to fix now, Docusaurus 2 does not anticipate generated markdown - it expects the author to write everything and make it styleable. So styling the API Reference will require some either via typedoc plugins to add classes, or remark plugins on the docusaurus side, Then there is the theme for didi which will likely be like deno but with more green. As soon as I push this up I will raise a further issue.
I really appreciate you can see the potential and the offer of support, your most welcome to help!
Yeah makes sense! it funny how sub parts of the project can stem a whole new set of problems that need to be resolved, the fun never ends! 😅
I think the ESM landscape is fascinating right, feels like there is loads of innovation happening. May I suggest you add a section like "Motivations for creating Didi" in your project Readme? I think this is a good way to explain why one would use this and how it resolves current pain points of using ESM.
Hit the nail on the head with that.
The motivations are missing, because I need to boil down several of my posts that lead to this point (100% want to add this section ASAP). What do you enjoy about ESM the most?
I'm not really using ESM as much as I would like to be. But what they promise is why i want to use them more.
About the same as me. I will have to do a
why didi is a thingpost in a short format.Loving the journey you are sharing.
I'm very much enjoying the process as well, the mistakes, failures and successes. I'm learning so much. It's all about the foundations, I think, if the developer is my customer of sorts, I should try to offer the best service I can. After all, It's not just the technology I have to win people over on, it's the passion of the idea, a future with JavaScript, but without bundles. I guess I can't do that without total transparency and really low barriers to entry. Thank you once again Ricardo.