If humanity is to have a future, we must become able to see, understand, and discuss the vast invisible systems that shape our world. And if we do become able — to see, touch, and say what is now invisible, intangible, and inexpressible — what a future that could be.

We've done this kind of thing before. Written language, mathematical notation, data graphics, interactive computing — each of these inventions enabled humanity to think and communicate in ways that were previously inconceivable. They affected everything, forever.

Now we need to do it again, and fast. We need a new medium that enables us to collectively grasp and grapple with the complex systems surrounding us — technical systems, environmental systems, societal systems — that now feel so out of sight, out of reach, out of control.

This report outlines Dynamicland’s progress towards inventing this new medium, and incubating a culture that is literate in it. While every year has brought major technical advances, they are merely the backdrop for the series of realizations — glimpses of the future — which have driven the progress. In this report, we hope to share a glimpse of our glimpses.

July 2013: YEAR ZERO

FOUNDING A LAB. In 2013, we started a new research organization from scratch. Alan Kay suggested the theme “Engelbart in the 21st century“.

To us, continuing Engelbart’s trajectory meant:

augmentation — inventing new computing systems to support fundamentally new ways of thinking and communicating

bootstrapping — inventing new computing systems and committing to live within them

We articulated a 40-year vision, recruited a world-class team of dynamic media inventors, built out a space, and set the context with a library and representation gallery. By mid-2014, the initial team was hired and ready to go.

July 2014: YEAR ONE

EXPLORING THE DYNAMIC MEDIUM. In our first year, we published screen-based projects demonstrating new forms of “reading” and “writing”, designed around the most powerful dynamic techniques we knew — immersive context, direct manipulation, explorable models...

A book can be a series of playable dynamic models... We can manipulate functions with our hands instead of symbols... Time-based media needn’t be trapped in time... Media can surround us with context and invite active exploration...

We also continued to articulate the long-term vision.

Could we design systems in a physical environment with everything visible? Could a humane dynamic medium be grounded in the physical world?

DISCOVERING SPATIAL MEDIA. We were supposedly doing computing research, but more and more, we couldn’t achieve what we sought within the bounds of a computer screen. To get real immersive context and direct manipulation, we found ourselves designing posters, dioramas, physical models, board games...

We can see context and connections! We can see the shape of the whole! We can think with our hands and bodies!

Could a poster be interactive? The yearning to interact with our posters led directly to our first-generation spatial computing systems, most notably “Hypercard in the World”, in which authors program by pointing directly to physical objects in the real world.

We can explore details and context at the same time! Dynamic media can live in our shared physical reality! Programs can be real things! We can use the whole space to dynamically show context! The building is the computer!

NASA launched the space shuttles from a control room, surrounded by views of everything that was going on. new medium must likewise exist in physical space, where complex concepts can be shown in their entirety.

July 2015: YEAR TWO

EXPLORING THE DYNAMIC MEDIUM. In our second year, we continued to publish new techniques and technologies for dynamic media on current platforms.

We can draw interactive diagrams instead of coding them... We can program while seeing all the data all the time... We can design media with tightly integrated speech and text... There are principles for designing playful interactive environments... The survival of humanity depends on people deeply understanding complex situations. So if dynamic media enables much deeper understandings of complex situations.....

RALLYING AROUND REALTALK. But the potential of spatial media was irresistible. We made dozens of physical-world prototypes and exhibitions, exploring how dynamic media literally in our hands could enable deeper seeing and understanding.

We can combine Sketchpad-like computation with hand-drawing... We can organize and browse dynamic media on physical pages... We can dynamically annotate physical books... Projection can have both wide coverage and high-detail areas... We can improvise dynamic spatial environments for watching and discussing... We can handcraft dynamic mathematical worlds... We can hold hyperlinks in our hands... There is a beer bottle in this “computer”. (Computation can share the real human world.)

These prototypes gave us a glimpse of what lay beyond Hypercard in the World, and we yearned to live there. That’s where the goals we had all along — live direct-manipulation authoring of computational media, spontaneous playful collaboration, ubiquitous visibility, immersive explanations that “show the whole” — could finally be realized.

We committed to design, build, and live in a computing system based in the real world: “Realtalk”.

We studied a wide range of literature, experimented with authoring models, object identification and tracking techniques, projection mapping techniques...

Watson and Crick figured out the structure of DNA by manipulating physical models with their hands. The new medium must likewise exist in physical objects, so people can think with their bodies and hands.

July 2016: YEAR THREE

BUILDING DYNAMICLAND. We realized that we needed to not just build a computing system, but build a place that is the computing system. We planned our dream community space, and found the perfect location in Oakland. It took a year to design and build out.

A medium is more than an authoring tool — it requires a deep literature, powerful ideas embodied by that literature, and a culture that has woven those ideas into daily life. We realized it was our responsibility to incubate that culture. This was a spatial medium; it needed its own space. We decided it should be a community workspace, so a diverse set of people could participate and contribute in a wide variety of contexts.

Andrew Carnegie jump-started mass literacy in the US by seeding thousands of community-supported free libraries. The new medium must likewise be grounded in community-supported civic institutions.

BUILDING REALTALK. Meanwhile, from an interim space across the street, we built our second-generation spatial media systems, in which all of Realtalk’s core concepts were invented.

In the La Tabla system, we prototyped working freely with paper as a computational material. In the Token Table system, we prototyped working with objects which move themselves. Another prototype system introduced claims and wishes. This loosely-coupled, decentralized communication model proved essential to Dynamicland’s unique social dynamics.

In the real world, multiplayer is free... People brilliantly improvise with real objects and materials around them... “Dumber” computation enables MORE improvisation... Code becomes minimal when physics does most of the work... Minimal code + real materials means people can make things for themselves, instead of being beholden to “developers”...

Through hundreds of prototypes, we invented answers to each of the questions: How are objects recognized? How do objects move? How do objects communicate? How do authors program objects? How do authors visualize what objects are doing? How is the hardware infrastructure built into the space?

At the end of the year, we chose a wall where Realtalk’s core OS would live, and the entire team live-coded it together on the wall.

We can build computational systems together, in the real world, with our hands, and it’s fucking amazing.

July 2017: YEAR FOUR

REALTALK IS BORN, EVERYTHING EXPLODES. We moved into the new Dynamicland space and completed our third-generation system — Realtalk. Thousands of people showed up to create dynamic spatial media.

We can use all the space we need to fully explore a concept... Spontaneous evidence-driven discussion requires computation on the table! “Reading” a book can be constructing the book itself... Abstract concepts can be tangible objects... When everyone sees what everyone else is working on, collaboration becomes spontaneous... Presenter and audience can be realtime co-creators! Ideas develop over time by people riffing on stuff lying around... When the operating system lives in the real world, everyone can extend it... Programming becomes communal when spread out on a table... Most projects can be a page or two of visible code, not a big lurking “codebase”... A new computing environment — designed around authoring, not “using” — is actually possible.

We’ve cataloged five hundred projects made at Dynamicland. But enumerating “projects” misses the incredible social dynamics — the new ways in which people work together, or how people effortlessly combine projects, or how Realtalk is used as “scratch paper” to support realtime conversations. These feel like the deepest glimpses of the future.

CULTIVATING THE DYNAMICLAND COMMUNITY. Because our goal is universal literacy in the new medium, we can’t work in isolation. We need a diverse set of people using the medium for their own purposes, shaping it with us, and we need them here in the same place, teaching and learning from each other. We deliberately sought people with a wide range of interests and backgrounds, with a focus on those underserved or alienated by current forms of computing.

We held open houses to introduce Dynamicland to local people, and from them selected our core volunteer staff. Our volunteers helped run our regular weekend community hours, plus tours, workshops, classes, and other events. Dynamicland was used for real work by local educators, students, artists, scientists, activists, and many others.

Every conception of democracy, from the Forum to town halls, is founded on people coming together to discuss. The new medium must likewise bring people together and support them in understanding one another and the world.

FUNDRAISING AND SURVIVING. The year also began with our parent organization abruptly shutting down. We refused to die. We were now on our own, responsible for funding a non-profit with almost no runway. We made a promotional zine and website. We gave tours to hundreds of potential funders: individuals, foundations, corporate executives... We experimented with corporate workshops, fundraising dinners, membership programs, etc etc. By the end of the year, we had lined up enough ongoing support to mostly kind of survive. We were also burnt out. We took some time off.

July 2018: YEAR FIVE

COMMUNICATING IN THE NEW MEDIUM. By this time, we were working entirely in Realtalk instead of conventional computers, and “dynamic paper” was normal instead of a novelty. We undertook projects to realistically use Realtalk as a medium for representing and communicating ideas while “living in the future”.

Art students at the California College of the Arts did thesis projects using Dynamicland. The UCSF Bionano Lab put Realtalk on the lab bench, and developed simulation tools and dynamic explanations. We created a dynamic scrapbook/gallery to exhibit the breadth of projects in Dynamicland.

Designers don’t need apps to make computational media... Simulations can live side-by-side with experiments... A book can contain live software and video... A book can transform the space around it... A book can be a place... Dynamic materials can feel like painting a mural together... Dynamic materials can feel like painting a mural together...

We hosted a series of Radical Learning workshops with teachers and learners over three months, culminating in a community showcase and spin-out projects. These workshops were a model for going beyond one-off community events to sustainable longer-term engagements, in which people deeply interested in a certain topic could use the dynamic medium to develop and discuss their ideas.

There is no aspect of human life untouched by writing and literacy. Writing is everywhere, used for everything. The new medium must likewise lift all people and support the entire range of human endeavors.

CONNECTING TO THE PHYSICAL WORLD. We expanded Realtalk’s physical capabilities. Realtalk isn’t about cameras and projectors — it’s about dynamic media in the real human world. Each new sensor and actuator provides a deeper connection to the world, enabling people to compute with their hands and bodies in even richer ways.

Dragglebot allows physical objects to move themselves. Laser trackers allow people to point to far-away objects. Pi bots bring arbitrary sensors and actuators into Realtalk. Dynabulb puts Realtalk anywhere. Also, experiments with: new fiducials, RFID tracking, 3D tracking, laser cutting, new printer types, programming by hand-drawing...

We can interact with dynamic objects up close in our hands... and surrounding us in the room... Computation can be infrastructural like electric light...

PREPARING TO REINVENT DYNAMICLAND. All this got started in year one with posters and dioramas. In year five, we’ve been studying the design of great museums, libraries, civic institutions... Big things are coming.

OUTREACH AND DISSEMINATION. We’ve presented this work in many venues. Harvard, MIT Media Lab, Caltech, Stanford, UC Berkeley, UC San Diego, Santa Fe Institute, California College of the Arts, National Academy of Sciences, National Academy of Engineering, Computer History Museum, Game Developers Conference, Eyeo Festival, Designer Fund, Foo Camp, TTI/Vanguard, Google, Stripe, SPLASH, UIST.

We’ve brought in visitors from many organizations to learn it firsthand. SFMOMA, Meow Wolf, Code for America, Mozilla, Exploratorium, Long Now Foundation, Internet Archive, Khan Academy, Project Jupyter, Iridescent, Protocol Labs, Rhizomatics, OpenAI, Grey Area Foundation, Institute for the Future, Buckminster Fuller Inst., Oakland Kids First, Oakland Youth Radio, Concord Consortium, Powderhouse Studios, Knight Foundation, Emerson Collective, Kapor Foundation, CK-12 Foundation, CODEd, ScriptED, SIGCHI, MIT Media Lab, Sidewalk Labs, Autodesk, Pixar, Media Molecule, Unity Labs, Apple, Google, Dropbox, Volvo, LEGO, IDEO, many others.

We’ve hosted over a dozen field trips with students of all ages. And multi-week residencies with local educators, students, artists, scientists...

In earlier years, we published our research on the web, because the web is a medium that carries screen-based ideas well. It works less well for Dynamicland. We’ve found that people require significant amounts of hands-on time and coaching to really “get” it. Accordingly, we’ve initially focused on teaching these ideas in person. Work on a comprehensive publication is in progress, but it will take time to figure out how to capture and convey what actually matters.

July 2019: YEAR SIX

REINVENTING REALTALK. In the upcoming year, we will design and build our fourth-generation computing system — Realtalk-2020.

When we started the lab five years ago, all computing was isolating and intangible — individual screens, individual keyboards, individual mouse pointers in tiny artificial worlds. At Dynamicland today, computing is a shared activity, where people work together with their hands in an abundant and infinitely adaptable computational space. There are no screens or mouse pointers, and even keyboards are on their way out.

Now that we and our community have lived this way for two years, we can see the next steps. To realize the full potential of this new form of computing, we will be generalizing Realtalk to work with any kind of physical material and physical space, and maturing Realtalk to prepare for incubation outside the lab.

COMPUTING WITH ANYTHING. In Dynamicland today, dynamic paper is taken for granted, and an incredible range of projects have been built on Realtalk‘s limited object recognition. Realtalk-2020 will be designed for authors to create their own object recognizers, and build dynamic media out of whatever material is best for representing their ideas — clay, coins, cups, pencils, paint...

What if we could explore microworlds that felt like dynamic game boards? What if we could repurpose ordinary objects around us during a conversation to improvise a dynamic simulation?

GOING EVERYWHERE. Dynamicland hardware is built into the ceiling, and Realtalk projects usually spread out over a table or wall. Realtalk-2020 projects will be able to go anywhere and take up all the space they need, including throughout a room, a building, or a city. Our next-generation hardware, the dynalamp, will go anywhere a lamp can go, emitting “computational light”.

What if a textbook or a scientific paper could be a museum-like room-scale exhibit? What if dynamic explanations were built into city infrastructure?

SEEING AND UNDERSTANDING EVERYTHING. By virtue of its physicality, Realtalk is already a uniquely visible form of computing. Realtalk-2020 will take ubiquitous visualization to the next level, using itself to explain itself, and to explain media created in it. We intend Realtalk-2020 to be the most understandable computing system in the world. We eventually intend to spread Realtalk by teaching people how to make it for themselves, from scratch, in such a way that they thoroughly understand it and can change it to be whatever they actually need.

What if we designed systems within environments in which we could see everything they were doing? What if the dynamic medium was not a product to be “installed and used”, but a pattern that people learned to make for themselves?

REINVENTING DYNAMICLAND. We started Dynamicland as a community workspace, because communal computing needed to be born in a dedicated community.

The next stage of Dynamicland will involve incubating the medium within existing communities outside the lab. We will work with organizations and cultural institutions that urgently need a form of computing that brings people together and enables them to see and grasp large-scale problems. Realtalk-2020 will serve as the foundation for this work.

We set out to invent a medium that enables all people to see, touch, and discuss the invisible, intangible, and inexpressible systems that shape our world. We were led to a computing system grounded in physical reality. Why?

By “medium”, we mean universal literacy for all people, not tools for specialists. This requires a form of computing that is as transparent, authorable, and adaptable as ink on paper. The complexity of conventional computing makes this impossible. But by not attempting to simulate virtual objects, and instead letting people compute with real objects, most of the code simply disappears, and computing can be accessible and human-scale.

By “seeing”, we mean seeing big, seeing whole systems, immersing oneself in context. This requires an unbounded canvas, where people can spread out and use all the space around them. Not scrolling through a pinhole.

By “touching”, we mean literally getting one’s hands on concepts, freely manipulating and arranging them, improvising with all the fluencies of the human body and properties of physical material. No computer can simulate this, and any attempt brings in untenable complexity.

By “discussing”, we mean people physically together, face-to-face, in realtime, working with seeable, touchable thoughts backed by evidence and computational models. This requires a form of computing designed from the ground up around people looking each other in the eyes, and thinking together with their hands.

This brief report can’t convey all of the experiences at Dynamicland that have substantiated these convictions, and compelled us to devote our lives to this work. Perhaps the best we can do is trace a single example. Here is a thing that actually happened:

The real estate group that owns the Dynamicland building uses their purchasing power to influence the city to improve infrastructure to underserved areas of Oakland.

The head of this initiative was visiting Dynamicland, and she began describing her work around the lunch table, which happened to have Geokit spread across it. Without breaking the flow of conversation, I dealt the transit card to display Oakland bus routes, and she grabbed the “zoom-and-pan dial” without any instruction to zoom in on a portion of West Oakland — noticing a huge hole in route coverage she’d never seen before where she knew there were tons of working families.

She resolved to bring her discovery to the city council, and we spent the next fifteen minutes exploring as she taught me more about city transit than I ever knew I wanted to know.

Where did that “zoom-and-pan dial” come from?

While exploring Geokit, a visiting designer (who “didn’t program”) saw the generic dial and Geokit's zoom lenses, and immediately thought about how to combine the two into a “zoom-and-pan dial”.

A couple of hours in the afternoon, working at a table together with paper, markers, and a keyboard, we had a working prototype.

Where did the “generic dial” come from?

At our first game jam, someone wrote a card that simply claimed which direction it was facing. Someone else had the idea to tape the card to a turntable. It was now a dial.

Over the next year, many people made many kinds of dials, and used them for everything from controlling music tempo to picking colors to generating random results for a magic 8-ball.

These moments actually happened: genuine data-grounded realizations in conversation, working dynamic media cooperatively authored in realtime, small ideas growing through riffing and physical remixing until they enabled genuine moments of realization.

We’ve seen these moments happen constantly, organically, among all kinds of people. Moments that are unprecedented in computing, and impossible with computers as currently conceived. By now, they’re almost mundane.

We believe there is a future in which these mundane moments happen every day, for every person on earth.

By sponsoring Dynamicland, you make it possible for us to make progress towards this new medium. More funding means quicker progress and more ambitious goals. Dynamicland is organized under a 501(c)(3) nonprofit, and is funded by annually-renewing corporate memberships (90%) and individual donations (10%).

We are not a large group. Everything shown in this report was done with a full-time staff of around five or six. This has included: Bret Victor, Glen Chiacchieri, Toby Schachman, Chaim Gingold, Robert Ochshorn, Michael Nagle, Paula Te, Josh Horowitz, Virginia McArthur, Luke Iannini, Weiwei Hsu, Omar Rizwan