Intro
How do we communicate mental models? If I have an understanding of some system, how do I communicate that understanding to other people?
Well, we have various forms of media for doing this. You can write an article, draw a picture, give a presentation, make a physical model, write a computer program.
All of these communication media are also thinking media. So I can use these media to communicate with myself. The process of making any one of these things develops my own understanding further.
Lately, I've been really interested in an emerging medium of interactive diagrams. Interactive diagrams are pictures that move, pictures that respond to a viewer's interactions. And I like these because they can help you develop a geometric intuition for the system that you're studying. [more]
They can help you explore a parameter space.
Interactive diagrams can help you understand causality in a system. [more]
They can help you understand relationships within a system. [more]
How does one part affect another part? If I change one thing, how does that affect all the other things?
Interactive diagrams can explain a process. [more]
And interactive diagrams can help you see connections between multiple ways of looking at a system. [more]
Today, people create interactive diagrams by writing code. I claim that this is problematic for several reasons.
First of all, there's context switching. So when I'm writing code, I'm thinking symbolically. But when I'm designing a diagram, when I'm sketching it on paper, when I'm playing with the diagram I've made to see if it looks the way I want it to look, to figure out what I need to change, all of the design of the diagram, I'm thinking spatially.
So in this process, you're constantly switching back and forth between thinking symbolically and then looking at your diagram and thinking spatially. And I think this context switching could be better supported.
Second, there's a big disconnect between viewing and authoring. So if I find a static picture that I like and I want to modify it or build off of it, it's easy for me to draw on top of it or collage it in Photoshop and make a new picture out of it.
But if I find an interactive diagram on the web that I like, well, maybe I can download the source code. But now I'm faced with this huge obstacle of understanding the source code.
And finally, writing code is difficult and it takes a very long time. Often hours or days.
So, in this talk, I'll be presenting some new work, Apparatus. Apparatus is a hybrid direct-manipulation graphics editor combined with a dataflow programming environment. [more]
And in this combination, Apparatus supports both spatial thinking and symbolic thinking. And it also helps both of these modes of thought support each other.
There's a gradient between viewing and authoring. The interactions that you use as a viewer interacting with the diagram, drag and drop, are the same as the interactions you use as an author editing a diagram, also drag and drop.
And finally, Apparatus aims to bring the timescale for creating these interactive diagrams down to minutes rather than hours.
Editing
So here I have Apparatus.
It's a direct manipulation graphics editor, so I can have shapes and move them around, resize them.
The shape that's selected, I have an inspector that shows me all of the attributes of this shape. And I can change these manually within the inspector.
And I can also write expressions in here. So I can say 1 plus 1. And then Apparatus says that's 2, and it makes the "scale x" of the circle 2.
Expressions can also reference other attributes. So I can say 1 plus "scale y". Now when I change "scale y", "scale x" changes accordingly.
These references can go across shapes. So I can take this rectangle and say I want "scale y" to control the rotation. So if I change "scale y", the rectangle will rotate.
So this is dataflow. It's like a spreadsheet. And like a spreadsheet, it has two really nice properties.
For one, there is no hidden state. So all of the state is there. It's always visible.
And second, this editor is live, meaning that there is no separate edit mode versus run mode. There's no compile button. There's no play button. You're always both editing and playing.
And if you think about it, a graphics editor also has these two properties. There's no hidden state. And there are no separate edit mode versus run mode.
So I'll do an example.
Variables
So I've drawn this face.
And this outline shows me all of my shapes. I can label them and reorder them, things like that.
So we've seen how shapes have these attributes. We can also create custom attributes. These are called "variables".
So I'll make a variable called "surprise". And I'll use that to control the mouth's "scale y".
And let's also have that control the size of the eyes.
So, what we've done here is essentially created a program that's procedurally drawing a picture, and it's drawing it based on this parameterization of this variable called "surprise".
Incidentally, this is a structured editor, so you can, for example, rename any of these variables and everything else just renames accordingly. In a text editor, of course, you'd have to manually somehow change all these references.
Interactivity
So we've seen how to procedurally draw a picture. Now, how do we add interactivity to the picture?
Well, the picture's already sort of interactive in the editor in the sense that if I were to click and drag on a shape, it moves around.
So that's a form of interactivity. But this is just the default interaction.
You'll notice when I put my mouse over the shape, the "x" and "y" attributes are highlighted in red, because it says these are the attributes that are going to change when I start dragging.
What I can do is click on this control circle, and now when I highlight over the mouth, this "wow" variable is highlighted in red, and when I drag on the mouth, that variable will change.
So this control mechanism is very flexible.
So here I have an example that has three variables that are controlling the shoulder, elbow, and wrist joints of this arm linkage. I can select the hand and then have it control any of these variables.
So I can make it control "wrist", and then when I drag it, "wrist" will change, or I can make it control "elbow".
When I drag it, the elbow variable changes. Or shoulder. Or I can make it control all three of these.
So what Apparatus is doing, the way this is implemented, is with numerical minimization.
Essentially when I click and drag on a shape, Apparatus says, okay, I'm allowed to change these three numbers. How can I change them so as to minimize the distance from the current mouse position to the shape that's being dragged?
As I drag, it's solving this minimization problem, and then adjusting the attributes.
So this is very flexible.
For example, I can make all of these variables have the same value, and then when I drag, the solver just knows what to do.
Next I'm going to talk about reuse.
Reuse
In code, we can reuse a chunk of code by wrapping it in a function and then calling that function in other places. Or we can use objects and classes. So we can put a method on a class and then all of the objects will inherit that class.
This is how you do that kind of reuse in Apparatus.
You'll notice that all the diagrams we create, there is a little icon in the create panel that's made for every diagram.
So if I make a new diagram to reuse this face, I can just drag it right out.
And each of these instances is sort of like a copy of this master shape. The instances don't affect each other, so I can change the "surprise" of one, and it won't affect the other one. They also don't affect the master.
I can even open these up. So I can take this face and open it up and see. Maybe I'll change the head, the color of it, and I'll make it like that.
So like copies, you can change anything about them. It's like you copy and pasted some code.
However, changes to the master get inherited by the instances. So if I go back to the master and I take these eyes and I make them white instead of gray, then the instances inherit that change.
Structural changes get inherited as well, so I can add some more shapes. I'll make some pupils for the eyes.
And then both of all those changes get inherited by the instances.
Changes don't get inherited if they're essentially overwritten. So it's like prototypal inheritance. If I change the "surprise" of the master, then that's not going to affect the instances, because I've overwritten "surprise" in each of the instances.
Likewise, if I change the color of the master, this one inherits that change, because I haven't overwritten the color. But this one keeps its color, because I have overwritten the color.
So this form of reuse is very handy for creating reusable widgets. So if I wanted to make a slider, well, a slider is just a track and a handle.
And the slider has a parameter that's its value. So I'll set the handle's "x" position to be that value. And I'll make the handle a controller for values.
And now I can drag the handle back and forth, and the value will change.
And then I can reuse that slider. So I'll put three of them out here, and I'll make a color picker.
So a color picker just has three sliders. It has a red slider, green, and blue. And it has a swatch that's going to show the current color.
It has a variable called color that's an RGBA value of red, green, blue, alpha component of one.
And now I'll use that color for the fill of the swatch. And now I have a color picker.
So then I can use these anywhere I want. So maybe I'll bring a slider out and have it control the "surprise" of the faces.
Maybe I want one face to be surprised and the other one not to be surprised. So I'll do one minus.
I can bring out a color picker, and use it to change the fill color. And then a color picker for that face over there.
So, a future trajectory, I think, of Apparatus is moving towards bootstrapping the editor within itself. So this drag and drop, you can make sliders which control, maybe this is a thing that's for interacting with the diagram as a viewer of it, but these things are also useful as an author of the diagram.
I can use these interactive widgets to edit the diagram itself.
So this is what I'm talking about. There's no separation between the viewing experience and the authoring experience. There's a gradient between viewing and authoring. Viewing is just dragging fewer things than authoring.
Now I'm going to talk about looping in Apparatus.
Spreads
If I have a rectangle here, I've got these attributes that I can change. And attributes can be singular values, like zero, or they can be spreads. I'll show you what a spread is.
If I say "spread(0, 10)"...
...what this does is it makes essentially ten different values. Rather than "x" being just one value, "x" is now simultaneously 0, 1, 2, 3, 4, ..., up to 9, and Apparatus will draw the square for each of these things.
So then I can reuse that. I take this "x" and I'll make it control the "scale y", and then you get something that looks like that.
So there are ten "x" values and ten "scale y" values that come in pairs.
You can say "x * 2", and it'll do what you expect. Or "sin(x)".
Spread also has an increment, optional third parameter. So I can say "spread(0, 10, 0.5)", and then that'll say 0, 0.5, 1, 1.5, etc.
I'm going to break out that increment variable. And I'll set the "scale x" to it.
So this is how you'd make a bar chart.
An interesting thing about spreads is how they combine. I've used one spread and I'm using that to control the "x" of the rectangle and also the "scale y". So it's essentially there's one "for loop", you could say, that's controlling both "x" and "scale y".
If I write in another spread, if I say I want "y" to be "spread(0, 4)", this will do the cross product of the spreads. You can think of it like two nested "for loops".
So that means that there's a difference between referencing a spread and creating a new spread.
For example, if I want to make a grid, I can say "spread(0, 5)". And that'll make my rows.
And then I can say "spread(0, 5)" for "y" also. So that makes a grid.
But that's different. Having these two separate spreads is different than reusing that spread, because then this will do the component-wise version of it.
So spreads were created to answer this question of, sometimes you have some variable, and you scrub it, and you say, oh, that's really interesting. It's really interesting how the picture changes as I move that variable around. And you want to see all of the versions of that.
So you just take that variable, and you say "spread". And then you can see multiple versions of it. So it's sort of implicit looping.
Spreads can also be used to plot functions.
So this rectangle, if I open it up, I can see that it's actually made up of four anchor points that I can move around.
So really, "rectangle" is just a path of anchor points.
I can delete those, and I have a triangle.
So if I want to graph a function, I can use spreads. Essentially, what I'm going to do is I'm just going to spread an anchor point.
So I'll take this anchor point, and I'll say, I want the "x" to be spread 0 to 10. That's going to make 10 anchor points.
And then I'm going to set the "y" to be "sin(x)".
This is taking the 10 anchor points that are distributed along the x-axis, and moving them to be sine of that value.
I can turn off "close path" and get rid of the fill.
Maybe make it a little thicker.
And then if I want more precision, I just put in an increment. So now I have 100 anchor points going from 0 to 10 that are graphing this function.
So that's how you would do a Cartesian graph in Apparatus.
Examples
Those are the fundamental features of Apparatus. Now I'd like to show some examples that were created in Apparatus.
Last week at my lab, we had a little micro jam. About a dozen of us sat around a table for two hours and played with Apparatus to see what we could make. So I'm just going to show a few of the things that came out of that session. [more]
Bret Victor made this one.
I think it shows the expressivity you can get out of Apparatus, if that's what you're going for.
Josh Horowitz made this diagram of a cycloid. Do you know what a cycloid is? So if you take a bicycle wheel and you attach an LED to one of the edges and then you roll the bicycle wheel, then the LED will trace this path.
I like this one because I think it's a really good example of how spreads can be used.
Josh had started with this diagram that just does that.
And then he said, well, I want to see what happens when that gets traced. So the answer is, well, you just use a spread. You take that progress and say, well, "spread(0, 10)".
So now I can see ten different values of that and change the increment.
A few modifications and you get it to this.
Paula Te made this diagram of a bike. You can pedal it and it moves.
You can also change the gear ratio. So you can change the size of the gears and then as you pedal it, it will move differently.
So Paula made this and then Josh saw it and he said, oh, that reminds me of a puzzle.
So this is the puzzle. So you have a bike, you tie a string to the pedal. And now you're going to pull on the string, you're going to pull backwards on the string, and the question is, does the bike move backwards or does it move forwards?
I'll give you a moment to think about it.
The options are it moves backwards, it moves forwards or it doesn't move at all. So how many people think it moves backwards? How many people think it moves forwards? And how many people think it doesn't move?
So, a range of responses. Most people think it moves forwards. So we decided to try it out with our model.
So we made the pedal a controller. Now I can take it and I can drag it backwards and the bike moves backwards.
I'll do it again. Take the pedal, drag it backwards and the bike moves backwards.
If you're not convinced, I encourage you to try it with a real bike, which is what we did at the lab.
There's also this video.
So it's sort of this balance of forces. You think that you're pedaling the bike clockwise, so you think that you normally go forward when you do that. But you're also pulling the bike backwards, so which of these two forces is acting in a stronger way?
It turns out that the gear ratio is actually important for this problem.
If you're in an extremely low gear, like you're going up a really steep mountain, then when you pull the pedal backwards, the bike actually moves forwards.
So it snaps around because the numerical solver is finding solutions over there.
But anyway, you can do this, and it turns out that what matters is the gear ratio on the bike compared to the ratio of the wheel versus the pedal. So the radius of the wheel compared to the radius of the pedal.
And so when the gear ratio exceeds that other ratio, the bike gets pulled backwards, otherwise it goes forwards.
We can actually see that happening. I made this other model where I just took the bike diagram and made a spread of it.
So I've got six bikes here, and you'll notice that this back gear is getting bigger and bigger as I go down. So I'm testing out a bunch of gear ratios all at once.
I can pedal them all simultaneously.
So you'll see that the bicycles with the higher gear ratio move forward more with the same amount of pedaling.
I've drawn this red reference line here that shows the starting position of all the pedals. And then what happens is if I drag this just a little bit, so if I just pedal just a tiny bit, you can see that for the bike in the lower gear, the pedal is now behind that original starting place.
Whereas for the bike in the higher gear, the pedal moves forward more.
So they all started here. And then the lower gear ones, they move behind.
The one at 0.5, that's where the ratio of the pedal to the wheel is exactly equal to the gear ratio. In that case, the pedal moves straight up. So in this one, the pedal moves straight up on the initial pedaling.
Conclusion
So to sum up, Apparatus is a graphics editor combined with a programming environment. You use the dataflow programming environment of it to procedurally draw the picture. So that's a case of the symbolic thinking controlling the spatial thinking. So you're using algebra to drive geometry.
Then you also set up these controllers where you can now drag a shape. And then that moves your diagram. So if the algebra is setting up the state space of your diagram, it's setting up all of the possible states that your diagram could be in, then when you drag the geometry, it's moving through that state space. So that's geometry driving algebra.
And this is how it unifies the symbolic and the spatial way of working.
You can do reuse. I guess I didn't mention that this form of reuse is coming from Jonathan Edwards's original Subtext paper. I think it's called "call-by-copying" in the paper. [more]
And it's really nice because it's essentially like you're copying and pasting. You can change anything that you want about the instance, but it still inherits changes you make to the master.
And you can't do that with textual programming languages. You really need a structured editor.
So imagine if you had your code, and you copy and pasted one bit of code and put it somewhere else, and you made some modifications to it, and then you changed your original code. It would be really messy to try to automatically propagate those changes to your instances, to the version that you pasted.
But you can do it if the editor is sufficiently aware of the structure of your program.
And then this reuse enables you to make all these widgets that you can reuse. And you create this working environment where viewing the diagram feels the same as authoring the diagram. So there's just a gradient between viewing and authoring.
For looping, we use spreads, which enable us to see any given variable or attribute across a range of values.
And finally, I showed some examples, which I think demonstrate that Apparatus is not only a communication tool, but it's also a thinking tool.
So you can use Apparatus to simulate systems. You can perform experiments in it. Apparatus not only is useful for communicating the answer to a problem, but it's also useful for generating questions, as we did with the bike diagram.
Apparatus is free, open source. It is on the web as of late last night. [more]
It's on GitHub. [more]
Apparatus has gone through many iterations over the past year or so. So I'm really excited to finally be releasing it publicly.
I'd love to hear your thoughts on it. I'd love to see diagrams that you create in it. There's a mailing list. There's a Google Group for it. [more]
And if you're interested in working on the editor, I would love to collaborate on GitHub.
Thank you.