Graphology: Writing a graph of nodes UI in HTML5 (Part 1)

I like Blender's compositor nodes.

It doesn't mean I know how to use them well, but I like the UI and I think it works very well in that context.

I also kinda like the JACK Patchbay, because it shows you exactly where everything is connected - and it uses a similar interface.

We will not talk about the ALSA Modular Synthesiser here.

I like these interfaces because they show you how everything is connected in a complex graph, where you can zoom in on a particular part and change some settings of a single node, or zoom out and get a good feel of how the data (be it pixels, vector data, sound, whatever) flows through the graph.

So I want to write a general-purpose graph interface for working with graphs of complex nodes. And I want to use a data-driven approach so that, if you want, node UIs will be generated automatically.

So far

A scattershot experimental approach. Here is the sort of UI I'm after. It's pretty horrible at responding to screen sizes.

Nodes UI example

Here's a simple demo of one of the interactions I'm after: dragging around Bezier curve connectors.

Note that this may very well be a horrible and hackish approach. It uses SVG to render the connector, and changes the path live as you drag its free end with the mouse.

Existing software

d3 - Seems like it might help in some respects but I couldn't find a demo that captures what I want. Also I want this UI to be compatible with React (more or less) which d3 isn't.

jsPlumb - This demo captures an essence of what I want to do. It's ugly as a naked mole rat, and the library itself is priced in $k's per year. I want my UI library to be open source (with reasonable commercial licences should this concept catch on).

Basic data model

I am assuming the following things for the following example:

1. Nodes are representations of functions
2. A node can have many inputs and outputs

import {Node, Input, Output} from 'nodes-ui';
import {Colour} from '../image/Colour';

export const ColourMixer = Node({
  type: 'ColourMixer',
  name: 'Colour Mixer',
  inputs: {
    colour1: { name: 'Colour 1', type: Colour },
    colour2: { name: 'Colour 2', type: Colour }
  },
  settings: {
    mixType: { name: 'Mix function', type: String }
  },
  outputs: {
    output: { name: 'Output', type: Colour }
  }
});

That would be an individual node's definition. Perhaps a special case might be necessary for nodes that provide external input ("sources") or describe external outputs ("sinks") in the graph. The UI will take in those node definitions do some magic to construct graphical representations of those nodes, letting you add and remove them and link them together into graphs. Finally, it will give you back a graph of objects like this (where the #'id' notation should be thought of as "a reference to a Node object with this ID):

{
  id: 'ColourMixer-35',
  type: 'ColourMixer',
  settings: {
    mixType: 'multiply'
  },
  inputs: {
    colour1: {from: #'Image-24', output: 'colourData'},
    colour2: {from: #'Colour-5', output: 'colour'}
  },
  outputs: {
    output: {to: #'Display-1', input: 'colourData'}
  }
}

(And I guess at the top level, you'd have to have an object like {sources: [...], sinks: [...]})

From there, you'd use this graph representation to construct your internal computation graph or whatever it is you want, and then run some computations on that structure.

What do you think of this idea? Am I reinventing the wheel? Is there a library that can already do this? Or do you have a suggestion for improvements? Let me know in the comments!

Comments

[TBD]

• SVG and Canvas example of nodes with zoom and add new nodes features
  static.laszlokorte.de/graph/

• also check Laszlo implementation of Bret Victor's talk "Drawing Dynamic Visualizations"
  github.com/laszlokorte/reform-swift

• interactive math expressions visualizer
  thetamath.com/

• implementation of your mockup
  codepen.io/xgundam05/pen/KjqJn

• another implementation that looks good
  github.com/sketchpunk/NEditorJS

[Chuck Woodraska]

I have been looking for something like this too build interfaces with. Saw a cool video recently for visual programming: [Jonas Gebhardt - Evolving the Visual Programming Environment with React at react-europe 2016]: youtube.com/watch?v=WjJdaDXN5Vs