Intro
I was asked to say a few words about the thread of research that my group and I have been involved in for several years.
It actually extends from a thread of research going back 60 years. And it's kind of hard to fit 60 years into a 30-minute talk. So hopefully I'll be able to hit some of the main themes.
Everybody here is familiar with the processes of commercial product development. But I've found that there tends to be less widespread understanding about just what research is and what it's for. So I want to talk a bit about that.
And also, in particular, a special form of research called "bootstrapping", which has been unusually fruitful in creating everything that we do every day.
And then I'll talk a little bit about what my research group has been up to, and how we fit in with that tradition.
My research group is called Dynamicland. We're a nonprofit. We are four people at the moment. And we and some friends have set up a little exhibit in the hallway down there if you want to take a peek afterwards.
Research
To get started talking about research, let's take a look at a user interface.
Here's one. It's a good one. I like it.
And there's a number of really deep, powerful ideas that you can see in this user interface.
One you'll see right here is that there's text on the screen.
So there's text on a screen. That's actually a really incredible idea.
And it's interactive text, which means that as the data changes, the text can change.
You can select it with a pointing device. That's a pretty cool idea. You can copy and paste it. Those are pretty incredible ideas.
You can use your pointing device to click on buttons and things happen.
You don't have to remember commands to type in, because there's a menu.
And the information space can be larger than the physical space, because there's a scrollbar.
These are literally world-changing ideas exhibited here in this interface. This was designed at Stripe, of course, but they're in every interface I've ever designed, and every interface that all of us use every day.
And as designers, we were very fortunate that we did not have to invent these ideas.
But somebody did. And those somebodies were researchers working in research labs. So, to call out a few of my personal favorites...
Here's Ivan Sutherland in the early 60s, inventing interacting with dynamic graphics on a screen. [more]
And in the late 60s, we have Doug Engelbart's project at Stanford Research Institute, where he's inventing working with large amounts of textual information on a screen, selecting with a pointing device. He invented the pointing device. Sharing that with other people who have their own screens. Clicking on links. It all came out of NLS. [more]
And then here in the early 70s is the Smalltalk interface at Xerox PARC, where you see those buttons and menus and scrollbars and all those wonderful things. [more]
And it's important to recognize that what you're looking at here, at the time, this was just complete bizarro. Nothing looked like this. The state of the art in interactive computing was this.
It was basically a printer and a typewriter. And this was the hot shit. Normal computing was this. [more]
It was a computer center that you didn't even have direct access to. [more]
So this was some really far off, crazy stuff.
And I've called out three projects up there. There was maybe a hundred, maybe a couple hundred people at the time working on this sort of thing. And that's in comparison to the thousands of people working this way.
And the hundreds of thousands of people who were at companies that were using computing in this fashion.
So these are very small communities working on these things.
And proportionally, a very small amount of the money in the industry was going into these things that actually ended up mattering.
So, that stuff didn't reach the general audience. One of the reasons being that computing hardware was too expensive at the time.
And so it took basically a decade of Moore's law, it took microprocessors and cheap memory, before there could be
a feasible commercial product. [more]
And then another ten years to ubiquity. [more]
So we're drawing a timeline here on the scale of decades. One milestone per decade. And all that had to happen before we could have Stripe.
Apple and Microsoft, of course, have done very well for themselves. And they did well for themselves by taking these ideas that were invented in research labs and implementing them on newly cheap hardware.
And that's a pretty common pattern. You take any number of examples. Let's look, for example, at machine learning.
A little bit of machine learning going on here at Stripe.
Neural networks were invented in the 40s by neuroscientists who wanted to understand how the brain worked. [more]
And their computational properties were worked out over the next couple decades. [1] [2] [3]
And all that had to happen before decades later, after decades of Moore's law, we could get the deep learning billboards on 101.
So research is planting seeds that, decades later, turn into entire industries. You don't have the research, you don't get the industry.
And maybe the closest-to-home example might be the internet.
So, Stripe runs on the internet. Stripe would be inconceivable without the internet.
Patrick and John didn't have to invent the internet.
But somebody did, and those somebodies were researchers.
Largely rounded up by Licklider and Bob Taylor at the Advanced Research Project Agency. They were working out of industrial research labs and academic research labs. [1] [2] [3]
And worked out the infrastructure of the internet and its core protocols over the next few decades. [1] [2] [3]
And again, all that had to happen before Stripe would be conceivable, or any of Stripe's customers.
So, Patrick and John didn't have to invent the internet. And they didn't have to invent scrollbars, or files, or file systems, or for loops.
Invented by Heinz Rutishauser in 1950, and he was on the Algol committee and got them in. [1] [2] [3]
Stripe uses for loops.
What if Stripe paid Rutishauser a fee for every for loop, in the same way that Stripe's customers pay Stripe a fee for every transaction?
That's not how the world works, and I don't think that is how the world should work. For loops were Rutishauser's gift to humanity.
And that's true for most of these research projects. It's just easy to not realise the weight of all these gifts, and how much we rely on them.
You can think of it as: the researchers are planting seeds which then, over time, businesses arise to harvest the results and turn them into products.
And that's a fine system, as long as the system supports those seeds being planted in the first place. It needs that to be sustainable.
One of the reasons I'm here talking about a bunch of stuff that happened in the 60s was because that's when there was sufficient public funding for these long-term ambitious projects. [1] [2]
That disappeared a little bit later, and most research funding was corporate funding after that, which didn't work nearly as well because the incentives were misaligned. [more]
And then today, for the sort of bootstrapping projects that I'm going to be talking about in this talk, there is no funding.
And that means these projects aren't happening. And what that means is that the Patrick and John equivalent 20 years from now isn't going to have a whole lot to work with.
They're probably not going to see it that way. They're going to see it as, progress has stalled because all the low-hanging fruit has been taken.
But that's not how it works. There's an infinite amount of fruit in computing. You just need the right kind of research environment to cultivate it.
Bootstrapping
And the right kind of research environment that has cultivated an enormous amount of fruit is what Engelbart called a bootstrapping research environment.
And so that's the Engelbart who invented working with text on the screen and mice and hypertext and video conferencing and all that. The agenda of his research lab was not actually to invent any of that stuff. The purpose of the research lab was to find ways for his own research lab to be more effective at inventing stuff. [more]
Bootstrapping is a process where you're not trying to make any particular product or technology to put into the world. You're actually trying to create your own little world. You're trying to create an environment around yourself where you're using tools and processes that are qualitatively different from those outside.
Chuck Thacker at Xerox PARC who invented the Xerox Alto put it as, we were spending a lot of money to simulate the future. [more]
So you can think of it as, the laboratory itself is the experiment. The goal is to create a social environment in which people are living in the future. And that serves as a prototype for how all people may someday be able to do that.
And these environments, these worlds that are being created, are often illegible to those looking at them from the outside. I'm going to talk a little bit about that.
And the purpose of all this work is usually a very holistic, humane goal.
It's not about technology. It's not really a technical goal. It's about enabling all people to see and understand and communicate in qualitatively different ways.
And it's often cataclysmically motivated, in that the researchers feel like if we don't find new ways of dealing with the world, we're all fucked.
So what you see here is Engelbart's research group. They're just having an ordinary staff meeting. They each have their own monitors, and they're looking at text on the screen, and they're kind of going through their notes together. [more]
And again, at the time, nothing looked like this. This was just its own world. This bore no relation to what either computing looked like at the time or to what work, what office work, looked like at the time.
It was this completely bizarre phenomenon. But for them, what you see in that photo is not a demo. That was just their day to day. That was just how they did their work.
Another example is Xerox PARC in the 70s. What you're seeing here is Marion Goldeen. [more]
She is teaching a class to some slightly younger kids.
And so all the kids have their own personal computer, and they're all programming on the computer and creating graphics on the display.
This looks familiar today. At the time, nothing else in the world looked like this at all.
And so people who looked at this didn't really have any idea of what they were seeing.
We can see that in a report from a Xerox executive.
A team of executives came down to PARC to see what the researchers were up to. They were shown the graphical interface, and what they saw was "the capacity to draw squares, circles and connected lines." [more]
"The product-focused officers were not impressed. The chief of staff wrapped up the show by commenting this was a hell of an expensive Etch-a-Sketch toy for his grandchildren."
So what this executive was shown was this.
And this executive knew what business looked like. Business looked like this thing on the left.
The thing on the right, he didn't really have a frame of reference for. And so he thought it was a toy.
Nowadays, of course, business looks exactly like that thing on the right.
And that thing on the left looks completely antiquated. And Xerox, which was once the top technology company in the country, is now completely irrelevant.
So it takes something really special to even be able to look at research and see it. You have to get enough out of your own context to be able to see that the research isn't solving the problems of your context. It's actually bypassing your problems by creating a new context of its own.
And maybe it's a rule of thumb that if what you're looking at doesn't actually look a little bit silly, then maybe you're looking at something incremental.
In a few minutes I'm going to show you some silly-looking things that I and my team do every day.
Again, the goal of all this work is not any technical goal. It is this grand vision of lifting all of humanity to enable us to think and communicate in new ways. [more]
For Doug Engelbart, he put it as, he wanted to augment the human intellect to enable groups of people to work together to solve humanity's complex urgent problems.
This is not a statement about operating systems or programming languages or mice. This is a statement about how human beings communicate with each other and how human beings see complex situations.
The operating systems and mice were a byproduct of chasing this goal.
Alan Kay, who was one of the principals at Xerox PARC and was responsible for Smalltalk, he wanted to create a new form of literacy in which all people could model and simulate the complex systems of the world. [more]
"Understand systems are too complicated to think about in classical ways." The programming languages and operating systems, again, were kind of a byproduct of what was ultimately a completely educational goal.
And here's an illustration.
What you see here on the left is the first instance of the pop-up menu. This is in Smalltalk in the early 70s. What you see on the right is the first commercialization of the menu. [more]
These two objects have completely different purposes. These two objects have nothing to do with each other.
But they look alike. And because we've grown up with descendants of that thing on the right, we can't actually see the thing on the left for what it actually is.
To understand the thing on the left, you have to understand that Alan Kay was trying to create a medium in which all people would be literate in modeling and simulating the complex systems of the world.
And he thought a good way of organizing those models might be as these computational objects that are sending messages to each other. So that's where "object-oriented" came from.
And so, how do you send a message to an object? Well, one way is to write it out.
You can give the name of the object and give the message. And that was the Smalltalk programming language. And that's a powerful context because you can send many messages, you can abstract over them, you can define your own messages.
But Alan Kay was trying to create a new form of literacy, which meant that it had to extend down to the world of young children. Young children had to grow up immersed in this computational medium in the same way that children today grow up immersed in a world of written text. And young children aren't particularly fluent in abstract language. But they are good at using their hands.
Well, is there a way to send a message to an object using your hands? Well, maybe you use this mouse thing to click on the object, and the object then reveals its vocabulary to you. "Here are the messages that I know how to deal with."
And these are the exact same messages you can then use in the programming language. They're the same vocabulary, whether you choose from the menu or whether you write it out.
And so when a child clicks on that menu on the left, they're doing two things. One is that they're actually invoking that action.
But the second thing is that they're learning this vocabulary. They're learning how to communicate with this object, and they can then use that vocabulary later, when they're ready, in the programming context, to make things of their own, to go beyond what's on the menu.
So that thing on the left is educational scaffolding. The thing on the left is designed to teach children to program.
The thing on the right is access to functionality. You click on the thing on the right, it runs an assembly language subroutine written by a priesthood of programmers who have decided what you can and cannot do. And if you want to do something that's not on the menu, you buy another computer.
So that thing on the left is designed to expand your world, expand your possibilities. The thing on the right constricts your possibilities.
The thing on the left is designed to teach you to become self-sufficient, to go beyond the menu. The thing on the right teaches you to be dependent on a corporation for all your needs.
And that ideology on the right has become so ubiquitous and so widespread that we do live in a world where, if you want to do anything on a computer, you use an app to do it.
And the idea on the left, this amazing idea that the purpose of a user interface is to teach you to go beyond the interface, that idea has been buried.
So that's the world that we lost.
Dynamicland's research
Some of us still want that world, so we're making it.
I got infected by Kay and Engelbart's ideas many years ago. And in 2011 or so, I was in a position to do some work, and I felt like the direction in which I could most usefully extend their ideas had to do with the use of space.
This was very much influenced by Edward Tufte, who demonstrated really powerful forms of understanding that could come from spatial arrangements. [more]
And the two aspects of space that felt really promising to me were using space to show context. To show lots of things, see the entire system and the parts and the whole and the connections between them. [1] [2] [3]
And the other aspect was to use space to move things around, so you could work with ideas geometrically, not just symbolically. [1] [2] [3]
This was not bootstrapping research because that requires a special kind of environment.
And so I was very fortunate a couple years later to be involved in the founding of a new research lab, where we could actually start to do some real work. [more]
For the first year or so, the group continued making these screen-based tools because that's kind of what we knew how to do. [1] [2] [3] [4] [5] [6]
But the purpose of the lab wasn't to "do computers". The purpose of the lab was to enable deep forms of understanding from context and spatial manipulation.
And so we found ourselves designing these big posters where you could actually see the entire system, and the parts, and the whole, and the connections between them. [more]
And you could walk around this poster world with other people and point to things, and be immersed in this space of structured information. And it was persistent. It didn't blink out when you switched to another tab.
Then for manipulation, we found ourselves making things like dioramas and physical models and board games. These things where you could actually use your hands instead of clicking. [1] [2] [3]
Because, if you're serious about space, this is the amount of space that you get on a computer screen.
And this is the amount of space there is.
And that's also where all the people are. They're out here.
And it just felt more and more absurd to be trying to design these expansive spatial interfaces in that much space.
It just seemed like, well, if walking around these poster worlds is so great, maybe what we should do is paint computation onto the posters and interact with those, and just forget about the screens entirely.
And so that led within a few weeks to our first spatial programming system, which was nicknamed Hypercard in the World, where you'd use this laser pointer to circle physical objects in the world and then you could write code that ran on them. [more]
All the objects that you "blessed" in this way were all running their own little program, and they can all see each other and communicate with each other, and you can interact with them with this laser pointer. [more]
You could put them anywhere in the building. [more]
So all the objects in the building were part of a single space, which meant that you could design activities that spanned the entire space.
And so that meant you could make these posters where you could select parts of them and bring up details, and you could search for things in these posters. [more]
And you could browse around this structured space of information. [more]
We got to experience what it's like to read a book while being able to see a map of the entire book, and see where you are in the whole, and see the connections between all the parts. [more]
You could also draw on the wall, I guess. [more]
This was a programming system, and it was implemented in itself as a big poster made out of living physical things. [more]
And that meant anyone could just walk up to it and start playing with it, and see the data flowing through the system, and understand how it all worked.
And everybody did understand how this worked, even if they didn't care, because it was just furniture and they walked past it every day.
So this was making some progress on context, but we still wanted to use our hands. And so that led to a series of prototypes around putting computation onto physical objects that you can manipulate with your hands. [1] [2] [3] [4] [5] [6] [7] [8]
And then a few more complete computing systems in which you work with objects with your hands. [more]
Or objects could move themselves around. [more]
And all of this culminated a couple of years ago in Realtalk.
So Realtalk was the first system in which we could actually seriously start bootstrapping. It was the first system that was designed so that we'd be able to put our laptops away and actually start working entirely in the system itself.
So in Realtalk, programs are physical objects. [more]
They're just things that are lying there and you can pick them up with your hands and move them around, and they can draw pictures on each other or affect the world in whatever ways you want.
The program is on the object. That means anybody can walk up and read any object.
In order for the object to be running its program, it has to be out, has to be visible, which means that has to be visible to everybody else, which means that anything that's happening, it is there somewhere. You can come up to it and look at it and manipulate it.
And you can change the code on anything by poking this keyboard at it. [more]
You change the code, and now it's doing that thing instead of what it was doing before.
There are also non-keyboard ways of programming. [more]
The programming model is designed for really rich interaction between the objects. And so you can get really interesting things by making these small, simple objects that combine with each other in very rich ways. [more]
The objects don't care who made the object. So I can make some things, and you can make some things. And we can put our objects together, and they'll work together.
Because they're all physical objects, we can arrange them spatially. We can put them on the wall. We can put them in boxes. We can put them on shelves. We can put them in our pocket. [more]
We can rip them, tear them, throw them away, doodle on them with markers. [more]
All these organizational features are zero lines of code, because you don't have to program a simulation to move objects around. You just move the objects around.
And the idea here is that this enables you to hold abstract ideas in your hands. [more]
You can work with abstract concepts out there on the table, and with other people who can also get their hands in there and work with abstract ideas.
People like to make books. A book takes zero lines of code and a hole punch. [more]
Books can contain video.
Books can contain simulations. Maybe you have a different simulation on each page, or a different version on each page. [more]
Books can affect the space around them. So they can use as much space as they need to convey whatever it is they want to convey.
Again, it doesn't take any coding work to make a book. You just stick things into a binder.
And like everything else, books are editable. [more]
You can take pages out, put them in, reprogram any pages. Take pages out and make your own book out of them.
Every book in Realtalk says, you can add to me.
You can edit me. You can edit me while you're using me. There's no distinction between author and reader.
And you see the same sort of thing in presentations, as well. [more]
A presentation can be co-created with the audience, where the presenter is putting things out into the audience, the audience is putting things into the presentation while it's going on.
And these things, again, are programs. They're living things. So the presentation can turn into a conversation, or play, or...
We don't even know what to call this. It's like painting a living mural together.
Discussions can take place around a table where you actually have datasets just right there on the table. [more]
So you can explore data while talking to other people and looking them in the eye.
Everybody can reach in and work with the computational model. Or you can change the computational model. [more]
You can program as you're having a discussion.
So you can explore ideas that can only be expressed computationally.
And again, the purpose of all this is an authoring system. What you do in Realtalk is, you make things.
You don't download apps. You make things out of stuff that other people have left lying around.
And there's intended to be this very gentle slope from, you start out playing with things that are lying around. But then you can easily just make little tweaks. [more]
So you change a color from green to red. Or you put your name in there. Or you just grab two things other people made and put them together. [more]
Then it's this gradual progression from tweaking and remixing,
to taking on more substantial projects. [more]
And that extends all the way down to the system itself,
which is just as visible and tangible as everything else. So anybody... and people do. The community just comes up and puts things on this whiteboard. [more]
There's no distinction between OS and user. It's just basically the stuff up here has a wider scope than anything else.
But anybody can go up and grab the compiler and change the compiler. And everybody is now using your new compiler.
Dynamicland as bootstrapping
So as I mentioned, bootstrapping is not about putting any particular product or technology into the world. It's essentially about creating a new kind of social environment. People are working and thinking in a new way.
And so we knew as we were making Realtalk, we also had to create that environment. We had to not just create the operating system, but create the place that is the operating system.
And so that was the idea behind Dynamicland. Dynamicland was designed to be both a research lab and a community space at the same time. [more]
And we've had thousands of people come through Dynamicland and experience working like this. [more]
But the most fruitful interactions have come from the hundreds of residents that we've had.
Of all kinds. Students and teachers and scientists and artists and activists and all kinds of people who are
working on their own projects in this new context. [more]
And it's hard to describe what it's like
to create software in an environment where everything's visible, and everything's grabbable, and everything's remixable. [more]
And the sorts of spontaneous connections and collaborations that just happen all the time.
Like literally, every single day you'll be walking by and you'll see somebody working on something. And you'll sit down and you'll start working on it with them.
Or you'll grab something else and put it into that. Or you'll see something that someone else left lying around.
As a joke, you'll do a little riff on that and put something else next to it. And then someone else will riff on that.
That is how things progress in Dynamicland. That's how projects happen. Many hands, spontaneously.
This is from our first game jam. There's a bunch of people sitting around this table, and they're all making their little objects. [more]
But then they found it was more fun to make objects that messed with other people's objects.
And then people reprogrammed those other people's objects, and it became this kind of communally-authored stew of software, where nobody even remembered who made what, because everybody made everything. This kind of software that none of us had seen before. [1] [2] [3]
Some people at the game jam wanted to make this Mario-style platformer game. The way that this happened was, [more]
somebody was playing the game using this crafted joystick, [more]
which was a lump of clay on a spring. Somebody was playing the game [more]
while somebody else was programming it. There's two sheets of 8.5x11 paper on the wall there that are the code for the game. The game is two pages of code. [more]
And so somebody was playing the game, while at the same time somebody was editing it, while at the same time somebody was drawing artwork.
And the artwork, again, is drawn on paper and is projected. While at the same time people were rearranging the level, again by literally grabbing the platforms and moving them around.
And so you had playing the game, while programming the game, while drawing the artwork, while doing level design, all completely simultaneously.
The game designers afterwards were just like, I've been in this business for twenty years, I've never worked with somebody else at the same time.
Some grad students and their professors from UCSF's Bionano lab came by, and in about an hour we had all together collaboratively live coded this environment for working with DNA. [more]
And it was made by everybody sitting around this table and grabbing things and moving them over here and reprogramming this and reprogramming that and shouting like no, DNA doesn't work like that, it has to be over like this.
And it was a lot of fun. And also the students afterwards were just like, we wish we could work in this environment instead of the fancy commercial tool that we pay the big licensing fee for.
And then we ourselves as the researchers, we are currently building the next system. So we get to use old Realtalk as the development environment for the new thing. [more]
And so we get to work in the same way as well, [more]
where we're literally immersed in the system that we're building. [more]
And putting it up all around ourselves, and handing things to each other, and working at the same time,
and using all the space that we need. [more]
We can bring people over and show them, here are our processes.
We'll bring up the timeline, and you can see the mutexes locking and unlocking.
We can hold blocks of memory in our hands
and apply our algorithms to it by hand, and see what they're doing, down to the bytes. [more]
I've been working with bytes for 35 years, and I've never been able to hold them in my hands before.
And this is just how we work. This is our day-to-day. We use laptops to browse the web, but we don't actually use laptops for our real work. We do it all in this manner.
And we're kind of just getting started. All of these bootstrapping projects that I talked about took a decade to get to where they needed to go, and we're just a few years into that.
An experiment in lifting humanity
I mentioned some of the ideals of Kay and Engelbart that motivated them, and our ideals are continuous with theirs.
I feel like if humanity is to have a future, we must become able to see, understand, and discuss the vast, invisible, complex systems that shape our world.
Computation is an essential ingredient in that medium. Computation is too powerful a tool to be trapped inside tiny devices and be restricted to professional programmers.
So what if computing was like electric light? It was infrastructural. It was built into every built infrastructure. You knew you could walk into any building and just put down a piece of scratch paper and start scratch programming.
Instead of what we're doing today with our cell phones, which is essentially carrying flashlights in the dark.
What if programming was something that everybody did? In the same way everybody knows how to read and write, everybody would be able to use computation to explore the ideas and problems that are meaningful for them.
What if an operating system was something that a community built for itself, from scratch, for the particular problems and needs of that community?
What if we worked in computing environments that were expansive enough that you could see whole systems and deal with them as a whole, not just looking at little bits and letting the complexity build up and overwhelm us?
And what if computing actually helped us work together? And together doesn't mean email and Slack channels.
Together means actually together, talking with each other, looking each other in the eye, all getting our hands on the same thing.
For us, these what-ifs aren't hypothetical. We don't ponder them and then go write a whitepaper.
These what-ifs are how we live every day. They're just how we do our work. And we hope that in however many decades, everybody in the world will be able to live like this as well.
Thank you.
