Date: Tue, 18 Jun 2013 17:28:17 -0700
From: Bret Victor
Subject: directions for lab and work
Hi Alan,

I'm not entirely sure what you mean by "how I would set up the lab" (although I just visited SAP today, and one answer might be "exactly the opposite of ***").  But I can try to address the (perhaps easier!) question of what I want to invent, and its implications for the space and people involved.  This will at least give you a clearer picture of my intentions with this lab.

(I've never written any of this down before!)


All of this work follows from the need to "augment human intellect" for understanding and creating systems.  It focuses on the representations that we think in, and the media that host those representations.

I have a near-term goal (could exist in 3-10 years) and some initial projects to start going there, and then a few 30-years-out visions that I'll be trying to point the boat toward.  Here they are in abbreviated non-convincing probably-crazy-sounding form:


Near-term:  new medium for understanding/building systems

I laid out my near-term motivations in my talk "Media for Thinking the Unthinkable".  http://vimeo.com/67076984  Basically, an environment for working with systems via many powerful representations of the system's structure and behavior.

I feel that the key to inventing this way of working will be a deep understanding of system representations in general.  (Of all kinds -- symbolic, visual, visual-interactive, physical, aural, etc.  This is very general -- I think of a "programming language", for example, as just one representation of a system's structure.  I think of a "tool" as a collection of linked interactive representations.)  There isn't much of a "theory of representations" yet; they are currently very ad-hoc, often invented by people who are immersed in some technical field but know nothing about design.

I have some exercises in mind to explore this broadly and deeply.  For example --

 - breadth:  choose twenty small individual systems or simulations or math problems.  For each system, explore a variety of representations for its structure and behavior, and design a small tool for designing or working with that particular system.
 - depth:  choose one particular system, and design twenty different tools and sets-of-representations for that system

As with any design exploration like this, the goal is to be able to step back afterwards, squint, and notice patterns across the designs -- common ideas that are popping up in different guises in different cases.  You try to abstract out those powerful general ideas, and design around them a more general-purpose environment for representing and working with systems.

Implications for the space:  I would like the space to be a "living gallery" of representations.  I'd like people to feel like they are immersed in a  world of representations, each one being a uniquely powerful perspective into some particular system or situation.  To goal is to inspire people to adopt, remix, and rethink the representations they see around them, as well as give each idea a physical location that we can point to.

Part of this will be collecting and curating representations from all fields, and decorating (covering) the space with them.  But I'd also like the group's work itself to be part of the gallery -- every design or prototype that we make goes on display somewhere; it exists in the physical environment around us; every idea has a physical location; it is always "at hand".  Interactive representations should be interactive.  (At face value, this implies a lot of interactive displays.)

I'd also like to always be questioning and exploring better representations for everything in the space.  For example -- a bookshelf.  Are rows of spines the best way to represent the knowledge embodied in a bookshelf?  Can we redesign the concept of a bookshelf?  Or a book?  What happens when we tear out all the pages of a book and turn them into a mosaic over a wall?  How differently does that bring the knowledge into the environment?  Etc etc.  Always exploring the notion of "representations for thinking in", as opposed to "computing" specifically.  (Because, what is a "computer" anyway?)

As much as possible, I'd like the space to feel like a "playground" for exploring new representations.  I'm not sure what this means exactly.  Maybe a lot of physical prototyping materials (wire, clay, 3-d printer, etc) but also materials for dynamic representations scattered around, whatever that means.



The long-term visions have to do with the theme that we are human beings, and the "representations that we think in" should be designed for the full human being, not just eyes+fingers+brain.


1. Escape the Tiny Rectangle

Almost all intellectual work today is done sitting at a desk, staring and poking at a tiny rectangle.  But we have bodies, and we have hands.

Bodies.  We are bodies, embodied in space, and I think there are powerful ways of thinking that require embodied spatial thinking.  I'm interested in media that are less like dynamic books, and more like dynamic rooms (or even buildings, or even neighborhoods).  What if you could design a system within an environment that resembled a dynamic museum exhibit?  Where you were always moving around and relying on your spatial senses, like a chef in a kitchen?

A lot of aspects of this vision aren't feasible today (and may even be misleading to prototype in today's screen-oriented technology).  But in the near term, we can prepare for it by thinking about representations in space (eg, the "gallery of representations" I mentioned before), and making the space very malleable and encouraging people to explore how different arrangements of representations affect thinking.  To encourage this, I'd like the space to feel very three-dimensional -- convenient ways of placing things on the ceiling or in the air; ways of sculpting the walls; tree-houses, etc.

Hands.  Hands are for feeling and manipulating physical shapes.  I think that, long-term, our powerful dynamic representations will be 3d objects, not flat pictures.  This will require some sort of "dynamic matter".

In the near-term, we can at least think about embodying static representations in physical objects, as well as simulating dynamic representations by playing with certain malleable physical materials (wire, clay, lego).  Again, this isn't an "active project", so much as designing the space to encourage "thinking with the hands" to the extent that we currently can, to prepare for the far-off technology.  (And maybe playing with simulated haptics, although see next thing.)

The "bodies" and "hands" aspects are complementary -- some representations we need to live inside, and other representations we need to hold and inspect from the outside.  I think the design of the space can help build habits that will lead to better thinking about these representations once the technology exists.  We can try to design the space and workflows so our work involves standing, moving our bodies, using our hands, and being embodied in a space -- instead of sitting in front of a tiny rectangle all day.


2. Unvirtual Unreality

Most of our dynamic representations today are virtual.  Computer displays, virtual reality, augmented reality, holography, haptics, etc., are all ways of hacking the human senses to simulate something that doesn't exist in the physical world.

As human animals, we are very very good at dealing with real things in the physical world.  Every virtual representation is missing dozens of the visual cues and physical affordances that we rely on to understand real things, and I think this limits the sorts of thinking that we can do through these representations.

As the world gets more and more virtual, I'd like to push back, and invent a world of dynamic physical things, where everything is dynamic but nothing is virtual -- all representations have shape and weight and texture, reflect light properly, can be physically manipulated, etc.  (Especially abstract representations!  What would a "physical scatterplot" be?)

In the long-term, again, this probably requires "dynamic matter", but in the near term, we might simply encourage thinking about physical representations ("What if this weren't a flat picture?") and exploring the range of senses that virtual representations neglect.


3. Evolution of Language

Currently, we talk and write in words.  Conversations are streams of words, books are piles of words.  These are the "representations we communicate with".  I'd like to evolve language.

The goal is to allow us to share our ideas -- "the pictures in our heads" -- with each other as cleanly as possible, and allow us to work with these pictures.

Writing could be a fine-grained mixture of interactive words and interactive visual representations.  A piece of writing needn't be one-size-fits-all, but can be dynamic for every reader and situation.  Etc.

Long term, this will require a new form of "literacy", a new corpus of "literature", and new authoring tools.  Near term, it can extend from the "active essay" concepts, or my "scientific communication" example, and the exploration of tools for creating such things.  (Although the long-term vision is something so much more fine-grained, with its own patterns of vocabulary and forms of visual abstraction, that it would be unreadable by someone today.)

Talking could likewise be words mixed with visual representations conjured up and manipulated in realtime.  

Today, most talking involves words plus hand-waving, or whiteboard-scribbling.  Ken Perlin has suggested some ideas for conjuring up AR representations while talking, and near term work can extend from that, and exploring tools for creating and manipulating representations in realtime.  (Long term, again, I envision something so much more fine-grained, with entire conversations taking place through manipulating dynamic visual representations intermixed with scattered words and sounds -- a true audiovisual language that would be unintelligible to someone today.)

Again, the design of the space and workflows can encourage thinking about these concepts, without them necessarily being "active projects".  When we have discussions, we can think about what would be a better medium for this discussion?  What concepts are not getting communicated, and what representations would allow them to be communicated, etc?