[long] [abstract] [but lots of pictures] [most pictures are links]
In our imagination jams, we've been imagining a platform where people could author dynamic things in-the-world.
At lunch a week ago, there were some questions surrounding what form the implementation of the platform would take, and what the platform's purpose was. I mentioned that the purpose of all this in-the-world stuff is to enable new forms of understanding, and it's a goal for the platform itself to be understandable. That led to comments that "understandability" may not have been made explicit as a value, and questions about what "understandable" means, and "understandable to whom".
This email started out as an attempt to address these questions, and turned into... well, whatever this is.
As a project progresses, it's easy to get lost in the details, and lose sight of the deeper reasons for the work. (At least, it certainly happens to me.) I tried here to review the values and motivations guiding the project and the research group, to "rederive" why we're doing this.
These aren't easy things to talk about, and I doubt I've expressed them well. Words are insufficient, as always, but they can hint and suggest.
Here is the introduction from the red poster last year, which still stands.
Introduction
We are building a computing system to support the creation and exploration of dynamic spatial media. This document is an attempt to clarify and articulate the current thinking behind the project, and figure out what we need to build in order to progress.
In the same way that the Alto was seen by some as an “interim dynabook”, we see our system as “interim dynamic material” — a means of inventing and prototyping dynamic spatial media decades before the proper technology exists.
We intend for the computing system to be “built in itself” to the extent possible. The system will be designed around the same principles of visibility, physicality, and in-the-world-ness as the media that it supports.
The project is exploratory and will happen in stages. We cannot set forth a “grand design” up-front. As we come to understand what spatial computing is, the system will gradually transition from drawing on conventional techniques to being its own thing.
A computing system is a means to an end. The goal is a medium for understanding systems related to mathematics, science, and human affairs. Our hypothesis is that the necessary medium can't fit on a computer screen, so the first step is escaping the tiny rectangle.
But let's go back a ways, and try to work up to that.
Here is a reasonable starting point for why our group exists, if you replace "talk" with "group":
Our goal is to invent a new medium
capable of hosting representations of systems
which enable the people who need to understand these systems
to understand them.
Despite what's implied in that talk, the systems in question, and the people who need to understand them, go beyond what is conventionally labeled "science" and "engineering". In a deeply interconnected world, systems touch on every aspect of human affairs, and I can't see how we can maintain a sane democracy, or livable earth, or empathetic society, without a drastically better way for people to be seeing the systems they are embedded in.
So in many cases, the "people who need to understand these systems" is every functioning member of society, and science, like literacy, needs to become something everyone does:
But I don't know a word for that, so let's just go with "systems" for now.
This was the goal that Alan set in motion. We're very far from widespread literacy as Alan envisioned it, but I think it's undeniable that his foundational research bore fruit, and continues to.
If our new platform is significantly better than this one, and we expect it should be, then it will offer a unique opportunity to "maximize potential understanding" of itself.
Our goal is to invent a new medium
capable of hosting representations of systems
which enable the people who need to understand these systems
to understand them.
What makes us think that a new medium is possible? What technology do we have now that Gutenberg didn't have?
We can now make a dynamic medium that hosts dynamic representations -- not just static marks. A dynamic representation can compute, respond to people, and connect to other things.
Now it's our turn to do foundational research. Can we do better? What technology do we have now that Alan didn't have?
We can now make dynamic representations that are in-the-world -- not just intangible symbols in a tiny rectangle. An in-the-world representation can make use of all the diverse and powerful senses and capabilities that human beings have evolved to understand and manipulate the world around them.
And an in-the-world representation can bring hands and bodies, touch and movement, back into what it means to be alive. Which can, in turn, bring life back into the act of understanding.
Our goal is to invent an in-the-world dynamic medium
capable of hosting in-the-world dynamic representations of systems
which enable the people who need to understand these systems
to understand them.
What is a "representation"? It's an externalized thought.
What is "communication"? It's the externalization of thought.
A medium is, by definition, a means of communication. One way of considering the facets of a new dynamic medium is to look at it from the "angles" of the forms of communication:
Many people, when they combine "communication" and "understanding", get something like "teaching" or "explaining" -- a kind of asymmetric information transfer. But there's a different way of looking at it. We can see a medium for people "discovering and understanding together".
If we think that people being together is important and valuable, we can amend our goal to a medium for mutual understanding, which can host mutual representations -- representations of mutual thought.
Being in-the-world can support mutual understanding, because the world is a mutually-shared environment. The world offers a much wider "horizon of observation" than a screen -- things you see are seen by others.
Our goal is to invent an in-the-world dynamic medium
capable of hosting in-the-world dynamic representations of systems
which enable the people who need to understand these systems
to mutually understand them together.
What does it mean for a representation to be "better" or "more understandable"?
For me, it is difficult to articulate the answer to this question -- you sort of have to feel it. It has something to do with the way in which the representation matches the structure of the system to the senses and capabilities of the human being,
and especially in the ways in which it draws on (or repurposes) powerful latent human capabilities.
It's not quite right to think about a particular representation as inherently "more understandable". (That's a function of familiarity, and of a person's existing state of knowledge.) Rather, we seek representations that "maximize potential understanding" by making stronger human connections available.
Often, that's by enabling people to directly see or manipulate concepts or structures that they would otherwise have to reconstruct in their heads.
On the left are some examples of representations that I think have this sort of advantage over those on the right.
Particularly relevant for us are representations that encourage dynamic understandability -- where a person understands not by passively seeing, but by actively moving. Our group has produced some excellent examples of such representations.
Of the many kinds of systems to be represented and understood, many of us have shown particular interest in representing computational systems, or programs. Perhaps because they are the material out of which we build all of our other representations. (And perhaps because the conventional representations are so clearly lacking.)
It's clear that we care about good representations for the structure and behavior of programs. (Although I think we share a feeling that programming is not an interesting end in itself, but merely a means of "getting to" systems that touch human life.)
Our goal is to invent an in-the-world dynamic medium
capable of hosting in-the-world dynamic representations of systems
which enable the people who need to understand these systems
to mutually understand them together.
In the short term, we are building a platform, "to support the creation and exploration of dynamic spatial media". The platform is itself a system, a program. The "people who need to understand this system" include
- the builders (us)
- potential collaborators (especially students)
- the audience (the people that we are trying to influence with our research, who need to understand what we're doing enough to get "infected" and carry it on)
Given that we care about good representations for the structure and behavior of programs, it makes sense that we would apply this care to representing the structure and behavior of the platform itself. It's not unreasonable to expect our platform to look more like column "this" than column "not this".
The representation of Hypercard in the World as a big live poster has been both a delight for visitors and a boon to me as the developer. It's a reasonable hint of how a dynamic in-the-world representation can usefully reveal a computational system's structure and behavior much better than text files in Github.
If our new platform is significantly better than this one, and we expect it should be, then it will offer a unique opportunity to "maximize potential understanding" of itself.
It can be both a provocative example of a significant non-toy software system "made flesh", as well as physically inviting for people to tinker with and improve. It can entice and guide people into knowing it.
I've mentioned before my aversion to pedagogical "explainers" that talk about a system instead of being the system.
The truest "explorable explanation" is a working system that explains itself.
A creative person chooses their work by following an internal compass, which they must trust to point in a meaningful and valuable direction. For me, this compass is a fragile thing, easily confused by interfering magnetic fields. Any time I'm in a social situation where I'm expected to explain myself -- "no, it's not Internet of Things... no, it's not Augmented Reality... no, it's not Artificial Intelligence... no, well, I don't know..." -- my compass gets twisted around and I lose all grasp on why I am. It's an awful feeling, and can sometimes take days to recover.
This is related to why I instinctually keep some distance from certain things (we've had this come up with ************************, to name a few) that have superficial trappings of what we're trying to do -- physical! tangible! in-the-world! beep-beep-boop! -- but have nothing to do with the values and goals described above in this email. These projects are not for representing unthinkable thoughts, revealing hidden structures, expanding humanity's thinkable territory, enabling deep mutual understanding of the complex systems of the world. They are for making gadgets.
And I know the appeal of gadgets! I majored in gadgets! I gadgetted professionally for years. Twenty years ago, I fell in love with EE because it was bringing magic into the world. It was so much more alive, freeing, thrilling than that "domesticated" magic trapped inside a screen. My heart has been in-the-world ever since, and always will be.
But I failed to find any gadget that could lift humanity in the way humanity needed. I eventually found the path with Licklider/Engelbart/Kay/Tufte, in media for thinking in. And so I reluctantly crawled into the screen again, because that's where the dynamic media was. It was only in the founding of this lab that I saw the chance to close the twenty-year loop, in the form of dynamic media-for-thought in-the-world.
Which is all to say why I'm wary of the seductive distractions of many "physical computing" projects, such as gadget toolkits. I've gone down that road, and couldn't find any way that a more understanding, enlightened civilization could be built on gadgets. And while it may be wise to keep an open mind and mine such projects for usable ideas, from the perspective of "representations of systems", ********************** are Neanderthal, and mostly have the effect of lowering one's artistic standards.
And the DIY/Maker stuff may be a necessary reaction to the present moment in history, but it's a regression, not an ascent. (Everything was already DIY before mass production and hyperconsumerism.) It's in many ways opposite to what this lab is about, which aspires to something fundamental and timeless, like the great representations of thought in our pantheon (alphabetic writing, mathematical notation, data graphics, etc).
Alan: "generate enormous dissatisfaction with one's designs using the entire history of human art as a standard and goal".
It's easy to lose the compass needle, it's easy to forget the deep why. The day-to-day work is many levels removed from its purpose. On a given day, you might be
designing a messaging protocol
to implement a pub/sub system
to make a network of IMPs
to make sensor data easily accessible throughout the room
to implement an object model based on observation
to enable authoring of dynamic media in-the-world
to represent complex systems that used to be unseeable
to understand complex systems that used to be ungraspable
to introduce new forms of human thought and communication
to expand humanity's thinkable territory
to escape tribalism and lift humanity to a sufficient level of enlightenment that technological power can used responsibly, not destructively
to prevent the world from tearing itself apart
The items at the top of the yak stack are easy to talk about, they lend themselves to lunch chat and status reports. The items at the bottom are abstract, hard to articulate, emotionally charged, unfashionable, easy to mock, even embarrassing. At least, I often feel like I can't discuss these things with anyone.
Alan's Early History of Smalltalk is a remarkable document in part because of the way it moves freely through the stack, top to bottom, no hesitation, no apologies. It's incongruous with the other HOPL papers, which stick to the top layers. The thinking behind many technical projects sticks to the top layers, or bottoms out in simplistic or facile justification... or bottoms out in a worthy and important cause, but a cause that is a reaction to the present moment in history.
Technology is transient, and even technologists who are trying to "make the world a better place" can only think about this world. The representations of Playfair and Viete outlived their world and their technology. Our loyalty should be to "medium", not "technology", and we should aspire to enable the next Playfairs and Vietes to invent representations of comparable transcendence.
Our goal is to invent an in-the-world dynamic medium, etc.
The platform we're building this year is not "the dynamic medium". The platform will be "technology", not timeless or transcendent.
But it should make possible
the exploration and perhaps invention of new kinds of representation-for-understanding
which will then make a bit clearer
what the medium for these representations should be
which will then inform
the design of the next platform
ad transcendum.