Intro
Since this is SPLASH, this is a programming conference, I want to start by reassuring you that this talk is, in fact, about creating programs.
I want to start with that because it might not seem like that's what I'm talking about.
I'm not going to be talking about writing code, I'm not going to be talking about building software in the way we normally think about it. But this is very deeply about creating programs.
Kind of in the same way that we create sounds when we talk, and we create letters when we write. This is about creating programs as a means of person-to-person communication, as a means of representing thought.
This is a personal talk in certain ways, and so I wanted to start out with just a little bit of personal background, so you can kind of see where I'm coming from.
I got my start, like many people, making games and apps and whatnot. I think found my stride designing creative tools, and made a number of musical instruments, visual design tools, that sort of thing. [more]
That started getting me interested in UI design. And after studying that, I went to Apple, inventing and prototyping UI concepts for experimental projects, which ended up influencing these things and perhaps other things yet to come.
I was also very interested in information design. I helped Al Gore make a book app about climate change. I designed a series of interactive information graphics, to enable readers to understand things in ways that would be difficult to understand in text. [more]
And then I got on a train, and started living on the train as a wandering research hobo. [1] [2] [3]
Among other things, designing new kinds of reading material, where it's not the author lecturing at the reader, but the author is guiding the reader through exploring dynamic models, simulations that the author has provided. The reader has the agency to try different things out, to say "what if", to critique the model, to be more active, be a participating reader.
So, making these things, and then also prototyping these different creative tool environments for dynamic things.
So, tools for working with mathematical systems, for making software, for making electronics, animation. [1] [2]
Tools for making things that have behavior, that do stuff.
And my focus always seemed to be on the representation of that behavior. How could authors see what the thing they were making is actually doing?
And what are powerful ways of seeing,
so they can deeply understand what the thing is doing, and understand it in many different ways. [1] [2]
This notion of representation, representation of dynamic behavior, seemed to be very central.
So, I made all these things, made all these prototypes, and this hobo period was just a couple years or so. [1] [2]
And then over the last year, I've just been reflecting on all this raw material, and trying to hear what the prototypes are trying to say.
Because all those things that I showed, each one of them was kind of individually useful in its own little domain, but that was never really the point of any of this.
All of those were a vantage point to get a glimpse of something much bigger. This new way of working and thinking in the dynamic medium. And that bigger picture is what I want to try to talk about today.
This talk has two parts. The first half is about the past, and the second half is about the future.
In the first half, I'm going to try to make the argument that representations, by which I mean the ways we externalize thought, have been responsible over the last 2000 years, in large part, for the intellectual progress of humanity. Enabling us to think thoughts that we couldn't think before.
But while these media that we've invented have empowered us in certain ways, they've also crippled us in other ways. And we now have the opportunity to design a new medium of thought that undoes some of that damage, that's both humane and empowering.
So that's the first half. In the second half, I'm going to try to sketch out what that medium might look like. What it might be like to communicate, to work, to think, in a humane dynamic medium.
Representations
We'll start with representations.
What I mean by representation is the physical form of a thought. It's something out there in the world that you can sense with your senses and manipulate with your body.
As an example, I want to tell you about one of my favorite forms of representation of all time, which was invented by William Playfair in 1786. [more]
William Playfair was writing a book about England's trade with other nations, and he had a lot of data. [1] [2]
And back then, data was always expressed, it was always represented, in tables of numbers. If you were doing economics or business or science, you had these tables of numbers, and you had to learn how to read and write them.
"Data" and "tables of data", they were the same concept. You couldn't tease them apart. The data, and the way they were representing the data, people couldn't think of them separately.
Playfair had a very interesting thought. He realized that there's a certain form of understanding that we bring to tables of numbers. But there's a very different way of understanding that we bring to things like maps.
There's a different set of perceptual and cognitive capabilities that we bring into play when we're looking at a map.
Playfair had the unprecedented thought of making a very peculiar kind of map, where instead of left-right being west-east, it would be earlier-later. And instead of up-down being north-south, it would be more money, less money.
So he drew this funny kind of map, which was the first data graphic, the first time anybody had plotted data.
And people immediately caught on to, wow, this is a really powerful way of understanding data.
As these things happen, people got it, but it didn't actually catch on as a thing for another hundred years or so. But eventually it did.
And today, of course, the data plot underlies almost all of modern science and engineering. Most branches of science and engineering would be almost unthinkable without the plot.
It's just totally fundamental to everything we do, to the extent that it almost seems obvious. And it's hard to realize that it was only invented a couple hundred years ago.
The big point here, and this is what I want you to hold through this entire talk:
What Playfair did was, he realized that human beings have a certain capability for understanding maps, for understanding this kind of cartographic abstraction. And this capability was not being used when they were trying to understand data.
So he invented a representation which represented the data in such a way that this latent capability could be brought into service.
And that's what representations do. That's how they work their magic.
They draw on certain latent innate capabilities that we have, capabilities that originally we evolved in a hunting-gathering context, and bring these latent capabilities into service, repurposing them for a more abstract purpose than they were originally intended for.
You can see this sort of pattern in play in many of the great representations throughout history.
In mathematics, of course, one of the first amazing ones was the invention of Arabic place-value numerals. [more]
You can't do arithmetic in Roman numerals. You can't think arithmetic in Roman numerals. You have to go over to your abacus. [more]
Place-value Arabic numerals were the first user interface for numbers that allowed calculation to be done on paper. This was totally transformative. [more]
Same sort of story with the invention of algebraic notation as opposed to writing out imperative knowledge in big blocks of text. You could write things declaratively in a small number of symbols. Absolutely transformative. [more]
Leibniz's notation for the calculus. Leibniz was the UI designer of the 17th century. He was obsessed with notation, always trying out different notations, always talking with his friends about notation. Because he realized that a lot of the power in an idea lies in the form in which it's expressed, because that's what allows people to think it. [more]
And in almost any given field, you can go in there and find a particular form of representation which transformed or ignited the field.
The periodic table, for example, Mendeleev's periodic table was basically the beginning of a theory of chemistry. That's where it all started. [more]
Bohr's model of the atom, which of course was originally inspired by Copernicus's model of the solar system. [more]
Faraday's representation of magnetism as lines of force, which influenced Maxwell to make his theory. [more]
Although, Maxwell did not write Maxwell's equations. Maxwell wrote 20 equations and 20 unknowns, scattered throughout his paper. It was absolutely a brilliant theory, but there was something much deeper there that nobody could see. [1] [2]
And it was Oliver Heaviside who invented the language of vector calculus specifically, for the sole purpose, of being able to write Maxwell's equations in that compact four-line symmetric form. [1] [2]
Of course, that representation of Maxwell's equations has been very influential, but more so, Heaviside's representation of "the vector" basically powered a hundred years of physics.
When you're thinking of the great ideas of history, often what you really want to be thinking about is the great representations that enabled people to think those ideas.
So, ideas live in representations, and representations, in turn, have to live in a medium.
There are different media of thought. There's some ideas that you can talk about in speech. There are some ideas that are carried well in song, or theater.
But the biggie, of course, was print, the invention of the printing press 500 years ago, which basically made all that stuff on the left possible.
All that stuff on the left was basically designed for the medium of print. With the possible exception of the numerals, but even those, they'd been around for a few hundred years, but they were basically kind of a niche thing until print came along, and that's when they started becoming widespread.
So, a powerful medium is what enables powerful representations to be used and seen and spread.
Ascent
You can think about the invention of powerful representations and the invention of powerful media to host powerful representations, as being one of the big drivers, over the last 2000 years, of the intellectual progress of humanity. Because each representation allows us to think thoughts that we couldn't think before, and we continuously expand our thinkable territory.
You can think of this as tied to the grand metanarrative of the ascent of humanity, moving away from myth and superstition and ignorance, and towards a deeper understanding of ourselves and the world around us. [more]
I bring this up explicitly because I think it's good for people to acknowledge the motivation for their work. And this story of the intellectual progress of humanity is something that I find very motivating and inspiring, and it's something that I feel like I want to contribute to.
But if you take this as your motivation, you have to be honest with yourself.
There definitely has been ascent. We have improved in many ways. But there are also other ways in which our history has not been an ascent.
We invent technology, we invent media technology, to help us make this climb.
But every technology is a double-edged sword. Every technology enables us, or has the potential to enable us, in certain ways, while debilitating us in other ways.
And that's especially true for representations, because the way that representations work is they draw on certain capabilities that we have. So if we go all-in in a particular medium, like we did with print, the capabilities that are not well-supported in that medium get neglected. And they atrophy. And we atrophy.
I wish I knew who drew this picture, because it's a wonderful depiction of what I'm trying to express here.
And even a little misleading, because the person in the last stage there, hunched over his tiny rectangle, we reached that stage a couple hundred years ago.
We reached that stage with the printing press and cheap paper, book-based knowledge, the invention of paper-based bureaucracy, paper-based working.
We invented this lifestyle, this way of working, where to do knowledge work meant to sit at a desk, and stare at your little tiny rectangle, and make little motions with your hand.
It started out as sitting at a desk, staring at papers or books, and making little motions with a pen. Now it's sitting at a desk, staring at a computer screen, making little motions on a keyboard. But it's basically the same thing.
This is what it means to do knowledge work nowadays. This is what it means to be a thinker. It means to be sitting and working with symbols on a little tiny rectangle.
To the extent that, again, it almost seems inseparable. You can't separate the representation from what it actually is.
And this is basically just an accident of history. This is just the way that our media technology happened to evolve. Then we designed a way of knowledge work for that media that we happened to have.
Inhumane
I'm going to make the claim that this style of knowledge work, this lifestyle, is inhumane.
I mean something very precise by that, so I'm going to try to explain with an analogy.
Imagine you adopt a puppy.
So you adopt this puppy, and you name him Puddles. You take Puddles home, and you give him a little snack, and you stick Puddles in a cage, and you lock the door forever, and never open it again, and Puddles lives out the rest of his poor life trapped inside this little cage.
So, that's wrong. I think most of us would agree that's sadistic, that's obviously inhumane. So let's ask, what exactly is wrong about that? What is inhumane about sticking Puddles in the cage?
Well, we kind of have a notion of what it means to live a full doggy life. Dogs have to run around. Dogs run around, and they sniff other dogs, and they pee on things, and that's kind of what it means to be a dog.
They've inherited this set of capabilities and traits from their wolf ancestors, and we recognize that a dog has to be allowed the full free expression of its entire range of capabilities.
By sticking him in the cage, you're constraining his range of experience. You're not letting him do all the things that dogs can do.
And this is exactly what we've done to ourselves. We've invented media that severely constrain our range of intellectual experience.
Of all the many capabilities that we have, all the many ways of thinking that we have, we've constrained ourselves to a tiny subset. We're not allowed to use our full intellect.
Capabilities
To see that, we need to take a look at: what is that full range of capabilities? What are all the different ways we can think about things?
And it turns out it's a very vast and amorphous space, our modes of understanding.
There's been a number of different attempts to add structure to that space, to carve it up. One way of looking at it is by looking at the different sensory channels that we use to make sense of things.
So as an example, if I say music, you'll probably think, oh, that's something that we understand aurally.
But there's another representation of music which is visual, which is sheet music, which allows us to understand music in a very different and very powerful way.
If you're making music, then you're often thinking about it tactically. It's a tactile understanding. It's an interaction between your hands and the instrument.
You could define dance as an understanding of music which lives in the body, which lives in the movements of the body.
And if you've ever played in an orchestra or a band, you can appreciate a kind of spatial understanding of music.
Where the guitarist is over there, the drummer is over there. You're thinking about music in a spatially-distributed way.
And music is just an example. There's any number of things which can be represented in any or all of these forms, these channels.
And it's a "more than the sum of their parts" thing. When you have multiple representations, they compound and reinforce each other in incredibly powerful ways, and allow for understandings that would not be possible in any single channel.
So that's one attempt to draw basis vectors into this space of understandings.
Another useful one was presented by Jerome Bruner, kind of adapted from Piaget's theories. [1] [2]
He talked about the action-based, image-based, and language-based understandings. [1] [2] [3]
Riding a bike is something that you understand by doing it.
You can't talk about how you ride a bike. The understanding lives in the performing of it.
There's an image-based understanding.
You can look at the drive train of a bike, and see how the sprocket is pulling the chain over the derailleur, and understand this complex mechanical system in this image-based, wordless way.
And then, of course, there's language-based understanding, understanding you can articulate.
You can talk about parts of a bike. You can talk about gear ratios, and calculate them.
I think most of us in this crowd tend to gravitate towards the symbolic. So I think it's important to remember that speech is purely symbolic.
And the most important invention in the history of humanity was the invention of writing, where we took speech and turned it into an image-based, iconic, second-channel form. [1] [2]
So that's what happens when you get multiple channels involved, it leads to understandings that would not be possible with any single one.
I'm going to put up a couple others, just to give you a sense of the breadth of this space.
There's Howard Gardner's multiple intelligences. [more]
Some people abuse this to say things like "Johnny is visual" and "Sally is verbal" or whatever. Don't do that.
The way to use this is to realize that every human being has all of these capabilities, and we can design representations across them.
I especially like Gardner's because he explicitly calls out the empathetic forms of understanding: the way that we understand other people, understand ourselves, understand nature. Designers can use all of these capabilities.
My current favorite right now is Kieran Egan's framework. He was kind of inspired by Vygotsky. [1] [2]
It's a little too subtle to talk about here. But if you're interested in this sort of thing, I recommend you read Egan's book, The Educated Mind, because it's fantastic.
So, the point of all of this is really just that the space of the ways we understand things, the space of our cognitive capabilities, is so vast and so diverse that even people who have devoted their entire careers to trying to reduce it down to a neat little theory have come up with very different answers, because there's so much there, it's such a rich space.
Let's just focus on these two for now.
Basically, every circle up there is a superpower. Every circle up there is a capability that we've been honing for hundreds of thousands of years, that we use in innumerable ways, and can combine with the other ones in really powerful ways.
And what happened with the invention of the printing press, and the invention of tiny-rectangle-based knowledge work, is this.
All that stuff drops out, and we're working with visual symbols. We're reading visual symbols, we're manipulating visual symbols. That's what it means to do intellectual work nowadays.
And you might think, okay, maybe that was true with paper and books, but with computers, it's getting better, right? And no, with computers, it is getting worse.
With a book, a book is at least a physical object which exists in the world. There's some amount of tactile response. You can hold it and move it around.
With a book, you can make a shelf, which is a spatial representation of knowledge. It's not a very good one, but it's at least something that you can understand spatially.
When you're writing with ink on paper, you can move freely between drawing imagery and writing in language, because the paper doesn't really care what kind of marks you make on it. So you have the freedom, at least, to move between those two modes.
But then we invented these things.
They have these flat, glassy screens that have no tactile response. And, that drops out.
And they're expensive little screens, so they take up a little tiny portion of your field of view. And the spatial stuff drops out.
And they have these keyboards. Basically, the only convenient thing to do is punch in symbols.
Especially for us - anything from writing an email to writing a computer program, anything other than just typing in letters is incredibly cumbersome. Regardless of what you're trying to express, you express it in symbols, because that's what the interface encourages you to do.
So, that drops out.
And so this is the cage that we have trapped ourselves in.
This is the way in which we have constrained our range of experience.
In which we have created a tiny subset of our intellectual capabilities, and restricted ourselves to this tiny subset, and have forbidden ourselves to use our full intellect.
There are two things wrong with this. One is that it's inhumane.
In the same way keeping Puddles in the cage is inhumane, because he can't do all his doggy things, using media that restrict our thinking to this tiny subset is inhumane, because we can't do all our thinky things. We can't think in all the ways that human beings are able to think.
So there's kind of a moral argument there. If you don't buy that, there's also a practical argument, which is that it's just wasteful.
One way to think about it is, as programmers, imagine you have an eight-core processor, and you're writing some code. It's kind of a sequential algorithm, you can't parallelize it. So you're running this code, and it's maxing out one of the cores, and the other seven cores are just idling.
And that hurts, right? As a programmer, you get this feeling in your stomach, like, oh god, this is so inefficient. If only I could parallelize this algorithm, there's so much latent power I can draw on.
And that's exactly the emotion that I get when I'm looking at this tiny-rectangle-based knowledge work that we do. There's all these cores!
We have all these capabilities, and they're just sitting there idle. And if we could only parallelize our representations across all these capabilities, who knows what we'd be capable of thinking?
The good news is that we now have an opportunity to do something about this. We're now at a, I believe, unique moment in history where we're inventing the next medium of thought after the printing press. We're inventing the dynamic medium.
In the same way there are certain thoughts that can be conveyed pretty well in speech, and conveyed in theater or in print, there's, I believe, a very wide range of thoughts that can be conveyed in programs. Once we understand how to do that.
This medium that we're inventing, the dynamic medium, has three very interesting properties.
One is that it's computational, which means that it's capable of simulation. It can simulate processes out in the world.
It's responsive. So, unlike a book, where you poke at the book and the book doesn't do anything, dynamic things can respond to stimuli. When they're simulating stuff in the world, they can respond and show you the results in many different ways.
And it's connected. So, unlike a book here and a book there, which are isolated objects, dynamic material can exchange information and talk to each other.
So, what this means, if you're trying to represent something like a system:
Today, in language, if you're trying to represent a system, you describe it. You talk about it. That's a very poor way of representing it.
In a computational medium, you can model it. You can actively model and explore that system, and then see what it's doing in many different ways. And you can start to fill in this.
We've got a responsive medium. Today, we mostly think of computational responsiveness as a visual thing — you've got a screen and it does stuff. And some amount of aural responsiveness.
I think things are only going to get really interesting once we have tactile responsiveness. I'm not really talking about haptics here. I'm talking about taking computation out of the screen and into the physical world.
Infusing physical matter with computation, and letting dynamic responsiveness be something that exists in the world, something you can touch and feel and interact with with your hands, the way we've been working with tools for millions of years.
Once we have a full physical responsiveness, we can start to fill that in.
And as a connected medium, that means that if you have a space filled with dynamic material, it's not all isolated. They can talk to each other, they can work together to make large-scale representations.
We no longer have to work with little things that we hold in our hands. All of the material can host things that are body scale, or room scale. Start to fill that in.
And we can start to have the potential, at least, to design a medium that is humane and empowering.
These are two sides of the same coin. Humane, meaning: allowing us to draw on our full range of capabilities, getting us out of the cage, allowing us to express our full intellect.
And empowering, meaning: taking advantage of that. Drawing on our full intellect to enable people to think thoughts they couldn't think before.
In the second half of the talk, I want to take you through a little sketch of what it might be like working in a fully dynamic medium. How thoughts would be represented in these different modes.
Person-to-person: one-to-one conversations or one-to-many presentations.
Thing-to-person: what does it mean to read, acquire knowledge from a knowledge object, or to browse many of them?
Person-to-thing: what does it mean to author this sort of material, or to use it as a way of thinking?
This is a forty-years-out "As We May Think" or "Dynabook" kind of thing. [1] [2]
But it's not really intended as a prediction, so much as just a guidepost to steer the ship towards. I've found it very useful as guidance for my own work, and so I hope you'll find it useful as well.
Technology
I'm only going to be talking about the human experience of working and thinking in this medium.
I'm not really going to address the implementation of this stuff. The technology that underlies it isn't really the point here.
But I've been reminded that there are people at this conference who are very interested in technology, and I would be remiss to leave it unaddressed.
So here's the part of the talk about technology.
I'm taking the liberty of inviting in legendary MIT professor Gerry Sussman,
author of Scheme and SICP and SICM and all these wonderful things. [1] [2] [3] [4]
Sussman is going to tell us about the technology of the future. [more]
"But here in the future, it's going to be the case that computers are so cheap, and so easy to make, that you can make them the size of a grain of sand, complete with a megabyte of RAM.
"You're gonna buy them by the bushel, I suppose. You can pour them into your concrete. You buy your concrete by the megaflop, and you have a wall that's smart.
"So long as you can get the power to them, and they can do something, that's going to happen.
"Remember, your cells are pretty smart, and they seem to talk to each other and do useful things. So we have to begin to worry about that kind of a world.
"That'll happen, it's gonna happen. Because there's no other way to make the future happen."
So if Sussman says it's gonna happen, that's good enough for me.
And as he says, we have to begin to worry about that kind of a world.
What he's worrying about, in this case, is the engineering problem of how do we build working, reliable systems out of this distributed computational material?
What I'm worrying about, and I'm hoping to impress on to you as well, is the humanist problem of what should we build? And why are we building it, and what will we do with it, and what is it going to do to us?
Because this technology that he's talking about, 300 computers per square inch or whatever infused into physical matter, that sounds like really far off. That sounds like we've got time.
But technology does the exponential thing, and we're always blindsided by exponentials.
So this technology that he's talking about is going to be here before we know it. And it's up to us to prepare for that.
We have the choice to say, let's start envisioning and sketching and prototyping, designing something that we understand to be humane.
So then when the technology is ready, we're like, okay, we know what we want to build with this. We know it's going to work. We know it's going to be good.
So, you can do that, or you can just do the incremental thing, and jump on whatever technology bandwagon comes along, and let technology follow its own course.
And I can guarantee you that that is not going to be humane. That result is just going to lead you into a tighter and tighter cage.
Because, when technology takes its own path, it finds certain veins of human capabilities, and drills further in those, and leaves the rest of the capability landscape untouched.
So even though it doesn't seem feasible right now, I think we need to start thinking about how to design a humane medium with the technology that we want to have.
Conversing
So, let's start off with the most fundamental act of communication, which is realtime conversation between two people.
This is something that we've been doing for hundreds of thousands of years. And we do it nowadays basically in the same way that we have been doing it for hundreds of thousands of years, by spoken words going back and forth.
That's how we converse. Spoken words going back and forth, wave our hands around, maybe we'll draw some sketches on a whiteboard or a cave wall.
There are some things that you can represent well in that medium. And there are many things, especially modern things that we need to talk about nowadays, which are not well represented in spoken language. And one of those is systems.
We live in an era of systems. There are natural systems like the environment, ecosystems, biological systems, pathological systems.
And there's the stuff that we make. Political, economic, infrastructural systems, things that we make out of concrete and metal and electronics.
The wrong way to understand a system is to talk about it, to describe it. The right way to understand it is to get in there and model it and explore it. And you can't do that in words.
So, what we have is, people are using these very old tools. People are explaining and convincing through reasoning and rhetoric. Instead of the newer tools of evidence and explorable models. We want a medium that supports that.
So, we can start to think about a medium for conversation that is naturally show and tell, where depicting and describing are on an equal footing. It's just as easy to create an image as it is to create a description.
And what you are depicting are not static images as we do today, but actual working models. You're creating simulations. You're creating programs.
For example, if you want to talk about how, in this case, an airplane wing generates lift, if you want to talk about the effect of some policy change, all these things can be modeled.
We want to have a medium where you can model them and explore them, and have that be the content of the conversation.
And you want these models that you're building to be evidence-backed. You want them to draw upon the facts and knowledge that we know about the world, incorporate them into the model, and have everybody be able to see the facts that you're bringing in, the provenance of the facts, how the model's working.
This leads to the first of several big juicy research questions, which is, how do we do this kind of modeling?
Because the way that we create dynamic models today is with this activity that we call "programming", which is going off and staring at a screen, and bashing on a keyboard for a few hours or weeks.
And I'm talking about getting this down to seconds. Improvising, sketching dynamic models in real time, as part of the real-time give and take of a conversation.
We don't know how to do that. We don't know how to create working dynamic material in seconds. And you might think that's impossible. How can you program in seconds?
I would agree that it probably is impossible, if you're thinking about staring at a screen and bashing on a keyboard. I'm talking about something where it's using all the capabilities of the human body.
We're pretty fluent in talking in words. And somehow we got very fluent in writing and drawing. We can just kind of do that without thinking about it. Playing a musical instrument, we can do that without thinking about it, if you've been trained.
And I think it will be the same thing for dynamic modeling. If we find the right medium, then we can do that, in real time, as part of the give and take of a conversation.
Again, this is a big research question. Not a whole lot of people are working on it.
If you've seen some of Ken Perlin's latest work with Chalktalk, he's starting to make the initial baby steps on creating a dynamic language that can be used in real-time conversation. [more]
What I'm talking about here, this is not "computer-supported collaborative work". This is not Google Docs. [more]
Those things are about making the old representations more efficient. Those things are about sticking with words and static pictures, and moving those around faster.
I'm talking about pushing words to the sideline, and having dynamic modeling, creating programs in real-time, be the content of the conversation. You're modeling, exploring, together with the other person.
Presenting
Now we can think about one-to-many kinds of communication.
Obviously, let's get away from clicking through a PowerPoint deck, and something closer to a blackboard, where we're, again, sketching material from scratch, where you can go in any direction.
And again, it should be dynamic material. So what you're sketching, as part of your presentation, is working models, and you're exploring them there with the audience.
And they're evidence-based models. Meaning that, conversations today tend to be a string of anecdotes. "Oh, I heard that." And then you say, "Well, I will trust him because he's an authority figure." Or, "He looks trustworthy." Or, "Someone else said he was trustworthy."
There's this complex notion of trust, which is hopefully going to be obsolete, if I'm up here building my models, and you can see the model I'm building.
You don't have to take my word for it. You can see the model. You can see the facts and knowledge that I'm bringing in to support the model. You can see where those are coming from. You can see the provenance of everything I'm using to support my argument.
And you can critique the model. You can check the facts yourself.
If you make all that visible, it leads to a very different notion of trust and integrity.
This is, I think, a really important part of the "empowering" aspect. Trusting authority is disempowering. Giving people the ability to be independent is empowering.
So, presentations in the form of improvising dynamic models that are explored together with the audience.
But I think the most interesting possibility has to do with using the space.
So right now, I'm on a stage, and the stage is completely empty. There's a podium, and there's a guy, and big pictures over there.
It would be nice if I had props. I could be showing you things. But I would have had to carry that in my suitcase. I don't have a way of downloading things to the stage.
But if we have an environment which is full of dynamic material, then we can start to think about actually using human-scale representations, using body-scale and room-scale representations.
Having a presentation where the presenter is moving around, and interacting with real things in the environment.
We can start to think about things like mapping concept space to physical space. The different topics that the presenter is presenting actually are there. You can see them. They don't go away.
The presenter presents by moving their body from one topic to another, as the argument moves from one topic to another. Having a very strong spatial grounding.
We can start to think about things like having the outline of the talk evident in the space. What I talked about five minutes ago is there, what I'm going to talk about in five minutes is there. It's not just like one thing after another, after another, as it is today.
Again, using the medium to invoke these kinesthetic and spatial forms of understanding that are really important.
So for these real-time forms of communication, I'm talking about improvising dynamic models as the content of the conversation or the presentation, and exploring them with the other person.
Reading
Now, we can think about non-real-time forms of communication.
What is the equivalent of a book or a website?
Books and websites nowadays are basically big piles of words. So they inherit the same problems as speech, in that the authors are explaining and convincing through reasoning and rhetoric, not through evidence and explorable models.
And they have an additional problem, which is that they're one-size-fits-all. They're mass-manufactured.
I read a book, you read a book, we both read exactly the same words, even though we're coming to it with different levels of knowledge, and we want to get different things out of it. In a dynamic medium, that makes no sense.
So we can start to think about, first of all, context-sensitive dynamic material. What I read and what you read are not necessarily the same thing. The material takes into account who we are, and what we want to get out of this.
This is, again, a big juicy research question. How do you create context-sensitive material?
Many years back, I wrote a paper called Magic Ink on this topic of context-sensitive reading material, but there's so much more there that needs to be done. [more]
The big thing is explorable models. What the author is sending out to the reader is not a big pile of words, but a working thing. It's a program that the reader then actively explores.
What this can lead to, one possibility, is the reader getting more out of the material than the author put into it.
In the same way that game designers are always surprised by what players end up doing in their game, and authors of creative tools are always surprised by what people make in their tools, I'd like to see authors being surprised by what readers end up learning from their material.
Because the author's not just sending out something static. They're sending out a program which is capable of emergent behavior. So the reader will be able to try out different things, and discover things that the author hadn't intended.
When you have this kind of active reading, it brings back some of the things that we lost when we went from learning through active dialogues with another person to just reading a single argument. This allows you to have a more interrogative mode, where you can have a kind of dialogue with the material.
But like what I said with presentations, I think the most exciting possibility here has to do with the form factor.
What I've depicted up here is the legacy of the printing press. It's this flat thing you hold in your hands.
I don't really think that's what a book should look like in the dynamic medium. I think a book wants to be a space that you walk around in.
Something that feels a little bit more like a museum gallery than a book today. You "read" this book by walking around in it, engaging visually, spatially, tangibly. Using all of those capabilities that we've evolved for understanding spaces and environments.
If you want to learn linear algebra, for example, you download the linear algebra "textbook", which is this entire space. Maybe each floor there is a particular chapter.
You make your way through the book by making your way through the space, interacting with the things. The concepts are represented in tangible physical form. You can use your spatial forms of perception to understand the gist of the entire material.
I want to clarify that what I'm talking about here is not VR. This is not VR. It's not AR.
It's just R. DR, if you like. "Dynamic reality."
It's real physical matter that you can touch, real physical matter that offers all of the thousands of perceptual cues and physical affordances that we've been honing over the last 100,000 years.
But it's dynamic. In the same way a picture on a computer screen is a dynamic picture, this is kind of dynamic physicality. [1] [2] [3] [4]
Browsing
So if that's what a book looks like, you can start to think about what a library looks like. No surprise, I think that a library should also be a physical space that you walk around in.
Libraries today are physical spaces, but you walk around and you just see a bunch of spines, which aren't very useful.
You can start to think about a spatial representation of the entire breadth of human knowledge, having different areas of the space represent different areas of knowledge.
You can walk into a space, just glance around, and get the gist of some particular branch of knowledge.
If you want to go deeper on something, you just move towards it, and it gets more detailed. You move towards it, it gets more detailed.
You have this continuous transition between browsing and skimming, and engaging deeply, that you can do, again, with your body.
Writing
So far I've been talking about knowledge in this medium being represented, not as big piles of words, but essentially as programs, as dynamic models that the reader is actively engaging with.
So we can ask the question, how do you create that material? How do you create those programs? What does authorship look like in this medium?
The closest thing that we have right now is this activity that we call "programming".
I think that programming, as we understand it today, conflates two very different activities, two very different things that we've both lumped in with programming.
I'm going to call them "engineering" and "authoring". Engineering is thing-to-thing, and authoring is person-to-person.
Engineering is building a working reliable system that meets a measurable spec.
If you're designing an opamp or a hydroelectric dam, you have this material that you're working with, and it's a matter of, what is the configuration of my material that meets the spec? It's between you, and the material, and the forces of nature.
That's thing-to-thing. And then there's authoring, which is person-to-person.
If you think about writing a newspaper article or creating a picture or something, the entire point is to create an impression in another person's mind.
The thing that you're making is just an intermediary. What you really care about is getting your message from the person to the person.
These are, I believe, very different activities. And we've lumped both of those things into "programming". We do them both in the same way, with the same tools, both of them just writing code.
I think that moving forward, we need to disentangle those. Moving forward, engineering might continue using code. But for authoring, I don't think code is an appropriate way of doing that.
Because it's indirect manipulation. If you think about a journalist creating a newspaper article, they're directly manipulating the words in the article. If you think about an artist creating a picture, they're directly manipulating the ink on the paper.
There's something very important about being able to directly manipulate the experience that the recipient is going to get. The author has to be empathizing with the recipient, and has to be able to see exactly what they're going to experience.
And so if you're creating a newspaper article, you're directly manipulating the words. If you're creating dynamic behavior in a dynamic medium, then I think you need to be directly manipulating that dynamic behavior.
Not going through the intermediary of code, but directly manipulating the end result.
This notion of direct manipulation of dynamic behavior is something that I and others have been working on for some time. [1] [2] [3] [4]
There are certain threads in programming language research which go back several decades. [1] [2]
But it's still kind of a niche, obscure thing. I think that's also a big important research question.
For this crowd especially. It's good to work on developing methods for that engineering stuff. But we also need to think about how people are going to be creating dynamic messages for other people.
I think it needs to be more direct manipulation, and needs to be in a more sketchy, improvisational mode. Getting away from the engineer's obsession with precision, and more towards free expression.
Thinking
So lastly, we can think about using dynamic material to think with.
A mathematician eventually prepares their proof for publication. But before they get there, they're doing a whole bunch of stuff on scratch paper.
We can think about that. What is using dynamic material to support your own thought process?
A lot of what I've said already applies, because the representations we use to communicate end up being the same representations that we use in our head a lot. We think in the same language that we speak in.
But we can start to think here about the form factors of dynamic material. What is the "stuff" that we'll use to think with?
I think that there's a duality. There are certain representations that you want to hold in your hand, and inspect from the outside. And there are other representations that you want to be embedded in, and explore from the inside.
So, there's this duality between objects and environments.
Almost all intellectual work that we do nowadays is flat and intangible. It's ink on paper, or it's pixels on a screen.
Physical objects to think with, we used to have them. We used to have things like slide rules, architectural models, molecular models. These things are going extinct because of virtualization.
A molecular model on a computer screen is weaker in some respects, because you can't actually hold it in your hands and think about it with your hands. But it's stronger because it's dynamic, and dynamic trumps everything.
And so we've been led to more and more virtualization. Not because we actually want virtualization. What we want is dynamicness, we want dynamic behavior. And putting it virtually on a screen is currently the only way we know how to get dynamic behavior.
And so that's why it's so important to get things out of the screen, infuse computation into the physical world, create dynamic physical matter, and then we can start to be able to think with our hands again.
I think the most intriguing possibility here is the possibility of abstract, tangible representations.
The literal representations are pretty easy. An architectural model is just a house, scaled down. A molecular model is just a molecule, scaled up. That's pretty easy.
But how do you represent, for example, an algebraic equation, in a form that you feel? What is the representation of "y = x ^ n"?
That concept has always lived in flatland. It's almost unthinkable — how could it be anything other than the flat symbols that we've been using so long to represent that?
Again, I think that this is entirely an accident of history that we've designed a mathematics for flat. We can redesign a mathematics that draws on all of our physical, tactile capabilities.
It's kind of a weird thought — what would an algebraic equation look like in a physical object? But you have to think about the Playfair thing.
What does reading maps have to do with understanding data? They're totally different things.
But Playfair invented a representation, the data graphic, which allowed people to use their map-reading ability to understand data. It was totally transformative.
And I think it's going to be similarly transformative to draw on our incredibly profound capabilities that allow us to tie a shoelace, or play a musical instrument, or do all the things we do with our hands, and bring those capabilities to bear on more abstract thinking.
Lastly, the dual to objects are environments.
Today, everybody works in some sort of environment. We work in some sort of room, that is in a building, that is in a neighborhood. And all that is static.
And because it's static, it can't participate in work that happens at the speed of thought.
Today, all work that happens at the speed of thought has to happen within the reach of your arms, because your hands are your only dynamic instrument.
That's what led to the desk, and working with papers on a desk, or objects on a desk. Everything had to be within arm's reach.
But when you have an environment that's full of dynamic material, that can respond at the speed of thought, then you can get away from that. You can start to think about representations that take up a large amount of space.
Start to think about a form of intellectual work that involves walking around, moving around, looking around, interacting with human-scale spatial representations. Taking advantage of peripheral vision, and spatial scanning, and sense of scale, and all these things that we've been honing for millions of years.
I mean, it's obviously debilitating to sit at a desk all day. We've had to invent this very peculiar concept of artificial exercise to keep our bodies from atrophying.
The solution to that is not Fitbits. It's inventing a new form of knowledge work which naturally incorporates the body, that draws on the strengths of the body, that uses the body in the way that the body was always meant to be used.
Conclusion
So this is my vision of a humane medium of thought.
This is my vision of a medium of thought that treats the human being as sacred, that treats these capabilities that we've been honing for hundreds of thousands of years as sacred, treats the physical world and our interaction with the physical world as sacred, and builds on top of those things to continue this ascent of intellectual progress and enable people to think ever greater thoughts.
You might have a different conception of what a humane medium would look like, and that's totally great. The point that I want to get across here, the point that I want to leave you with, is that "humane" won't just happen.
This is not like Sussman's technology, which is "going to happen" because there are already really powerful forces at play.
Humane is never a default. Humane only comes out of very deliberate and conscientious design work.
If you do the incremental thing, and just ride the current wave of technology, and let technology lead you wherever it leads you, it's going to lead you into a tighter and tighter cage.
It's going to lead to more virtualization, more disembodiment, more dehumanization.
And so if you believe in the possibility of a humane medium, then it's up to us to fight that trend.
It's up to us to make it happen.
Thank you.
