Dynamicland Research Note #3

The Research Gallery

Dynamicland is a non-profit research lab. We are inventing a new computational medium where people work together with real objects in the real world. For the past two years, we have operated as a communal workshop. Using the prototype system installed in our space, a community of volunteers and guests have built hundreds of projects which could not exist anywhere else.

These creations fill shelf after shelf at Dynamicland. If you know where to look, these shelves tell dozens of stories of inspirations and improvizations. But by themselves, the shelves are opaque, and the stories are hidden.

We wondered, could we make these stories visible? Could we build a space that knits projects together into an explorable whole?

Between March and May 2019, our group spent 14 days in "stampedes", building a response to these questions in the form of a Research Gallery.

This note documents the Research Gallery we built. First, it explains how the Gallery works, as a system for browsing, searching, and authoring. With that foundation, the note recounts a few reflections we have on the Gallery – patterns we think are useful for those developing spatial dynamic media.

How it works

With the Research Gallery, you can browse projects that have been made at Dynamicland, learn more about them at a glance on the gallery wall, and dive deep into a project's backstory by flipping to the corresponding project page in the Research Scrapbook.

While the Research Scrapbook and Gallery Wall are spaces and objects to view and browse, the Scrapbooking Table is a space for authoring. At the Scrapbooking Table, you can design a scrapbook page, add details and media scraps to the page, and create its corresponding project card that goes on to the Gallery Wall.

Browsing, viewing, & learning

Here's a closer look at the gallery wall. Cards on the wall represent projects, events, creations in Dynamicland. The project "Utility Kit" is already selected on the wall, noted by the green outline. Up above the cards, the details of "Utility Kit" are displayed.

There are two other cards outlined on the wall, "Dot Counting World" and "My First Realtalk Book". These outlines mark the pages in the research scrapbook that are currently open. These scrapbook pages are the documentation vessels for each of the projects; they contain more descriptions, handwritten details, personal stories, and photos and videos of the project in use. Some scrapbook pages, even contain the program itself. We set the scrapbook on a coffee table in front of a couch so you can comfortably read and flip through the scrapbook.

From the couch, you can also look back up and browse through the gallery wall using a laser pointer. Let's say you want to bring up a bit of information about Music Cards to a friend or guest who's just dropped by. Point the laser at project "Music Cards" and the displays above the cards change to show information about that project. What appears on the wall actually comes from the media and detail scraps on the scrapbook page - all of the content on the wall comes for free when you document a project. You can select different videos with the laser pointer to listen to them/turn on the sound for that video.

The timeline above also highlights when the project was created. You can also browse the different projects by pointing at the different spots on the timeline. When you want to learn more, you can flip to the scrapbook page of that project, with the cards acting as a table of contents so you know where to flip to (both the cards and scrapbook are arranged chronologically).

Searching & seeing connections

All of the details, like the title, date, description, and authors, are searchable on the gallery wall. We have a dedicated keyboard to search through the gallery. If you search for Weiwei, for example, you'll see all the cards that have been created by or are associated with Weiwei. Searching for music highlights projects that are associated with music. This allows you to see connections and browse through related projects.

Authoring

The scrapbook table has everything you need to document a project and design a scrapbook page to showcase that documentation. There isn't any particular order to designing the scrapbook page, but there are a couple of best practices.

A scrapbook page needs, at minimum:

• A cover scrap, which will become the cover image for the research card representing the page on the wall
• A details scrap, containing the project's title, date, authors, and description
• At least one piece of media, a video or a photo of the project

The cover scrap first goes on the top right corner of the page; the white outline that appears around the page marks that everything within this region is a part of this particular scrapbook page.

Next, program pages, photos, and videos of the project are gathered. You can use a Realcam to record photo and video; the Realcam will save the media to the server and print out that media on a page that you can use right away. You can also place previously gathered media into the server and it'll print out a page in the same way.

We used iPads as Realcam prototypes, programming them to record media and send it to the printer. (Will Felker, video credit)

The details scrap is made next by picking up a template and editing it. In addition to the standard details, you can also add metadata to describe which media you'd like to appear on the displays on the gallery wall. You can also add metadata about related pages (which could then be used to see connections between projects).

Using the 'details scrap print' tool, you can get a printout of the edited details page (a computer-readable scrap) and associated title and text scraps to facilitate the scrapbooking process.

After all of the media and detail scraps are gathered, it's time to design and lay out the scrapbook page. The cover page is filled out; you can add other handwritten notes, text descriptions, or drawings and embellishments to the scrapbook page.

The questions and steps on the wall are meant to guide you in documenting your project.

Now you're ready to place the scrapbook page into the scrapbook! You print a research card with the 'print research card' button or toolpage. A card comes out containing information about all the details and media that have been pasted into the scrapbook. The date-order toolpage helps to place the card and scrapbook page in their correct locations. It's important to note that while usually this happens when the scrapbook page is "finished," it can happen at any time, for a work-in-progress, or a documentation-in-progress.

Reflections

Now that you've seen what we built, we'd like to share some aspects of the Research Gallery which we find interesting. Some of these were discoveries we made in the course of this project. Others were familiar themes, expressed in new ways by the Research Gallery.

Our five reflections, in sentences:

1. The gallery is a place.
2. The scrapbook is the physical source of truth.
3. The gallery helps us see the whole.
4. Computers can guide physical actions.
5. We build the gallery improvisationally.

The gallery is a place

In the end, the software we wrote for the gallery played only a small role in the project. The research gallery is a place. Most of our effort went into thinking about where that place was, what purpose it served, what was in it, and how it should make people feel.

We considered the roles the Research Gallery had to play. It is a library. It is a theater. It is a gathering place. It is a book. It is an exhibit. It is a studio.

We needed many other kinds of technology than software to support these roles: couches, tables, shelves, lighting, signs, crafting supplies, recycling bins, plants. We needed a space to put it all that would make us feel at home. We wanted it to attract people, to invite them to wander through the projects, explore the book together and add to it.

Consulting our concept art, we scouted locations in the Dynamicland space that would best fit the research gallery as we imagined it. This included considerations like: where it fit in the flow of daily life in the lab, what the lighting was like, how cozy it was, and lines of sight created between the gallery and other important locations in the space.

We imagined a cycle of use. A person might find a project while wandering the gallery, become inspired, and print it out to create a remix. Excited by their new creation, they would want to document their new work for all to see. They would come to the scrapbooking table and use its tools to create a dazzling page. They would add their page to the scrapbook, add its cover page to the wall gallery, and sit with satisfaction on the couch, exploring related projects on the theater wall. Someone might join them, and begin paging through the book as the cycle began anew.

The space is set up to encourage this. The couch sits with a coffee table in front of it holding the book, supporting a leaning-forward perusal of its pages. The table is also outfitted with a laser pointer holster, supporting a leaned-back browsing of the gallery, with videos, descriptions, titles and timeline comfortably in your field of view. The scrapbooking table is just to the right, always beckoning you and reminding you that this is a living gallery to be added to. Between the furniture we left wide walkways, allowing many people to comfortably inhabit the space simultaneously without getting in each other's way. The space is surrounded by four windows and a skylight, which (with the help of well-placed window shades) provide warm daylight without overwhelming the light projections.

Modalities of the Gallery as Place

The scrapbook is the physical source of truth

In the course of this project, we found a new pattern for managing information in the physical world: a scrapbook serving as a physical source of truth for a larger system. Here's what that idea means, and how we came to it.

A card in the research gallery represents a project, like Music Cards. To be useful as a part of the gallery, this card should store associations with many bits of information: metadata (like a creation date), pictures & videos, and links to other cards. Making a gallery card involves bringing these pieces of information together… somehow. How should this process work? How should this information be managed?

At first, we imagined that this web of associations would be stored in a database, and we would build physical workflows for editing it. Perhaps you would bring a video card up to a gallery card, wave a magic wand, and the video would now be connected with the card forever.

But this approach comes with a series of thorny problems. How would you break one of these connections, if you made a mistake? Or how could you survey the space of existing connections? By default, you can’t. An “association” magic wand doesn’t come with a “dissociation” magic wand or a “see associations” magic wand. Each of these tasks requires a custom-built interface which people would have to learn and grapple with.

These limitations are rooted in the fact that a database of associations exists only as virtual state. Fortunately we have discovered that they mostly disappear when you replace virtual state with physical state. Find some way to represent your information in a physical form, and it automatically becomes visible and modifiable, because the physical world is visible and modifiable.

So we didn’t spend too long worrying about databases. When we thought through (and acted out) the process of making a gallery card, we were naturally attracted to a different approach. Maybe all the bits of information associated with a gallery card could live together somewhere in the real world, represented as physical objects. Perhaps they live as scraps, pasted onto a common page. Making and breaking connections then becomes an inversion of the edit-menu metaphors. Want to add a new video? Paste it in, literally. Don’t like it anymore? Cut it out, literally.

Using the physical world as a canvas opens up new possibilities for expression. inspired by the traditional craft of scrapbooking. If you are pasting bits of information onto a page, why not spend some time thinking about how they are laid out? Why not write captions? And draw in the margins, and cut out colored backing paper, and put in pop-up tabs? Why not put running programs right on the page? We began to see the scrapbook not just as a physical database, but as a space for open-ended, playful documentation.

To summarize our approach: The scrapbook page serves as a physical source of truth. It is the canvas that the author modifies to define all the digital information associated with a project. For the time being, we have adopted this property as an experimental discipline. We want to see where it takes us, even when the easy route would be to diverge from it.

In general, this approach has worked well for the Research Gallery. It keeps the implementation of our system simple and extensible. And, as mentioned above, it makes the authoring process natural and easy to understand.

The main weakness of this approach is the significant friction it introduces to the editing process. Making a change to a project's associations requires going back to the scrapbook page and making physical alterations. In practice, this means that pages don't get edited after they are made.

There may be ways to compromise here, between physicality and virtuality. As one model, the "patching" mechanism in Realtalk allows you to edit a page's code virtually and experiment with changes quickly, all without printing a new copy. Once the new code settles, it can be printed, restoring the benefits of solid physical existence. Perhaps there is an analogous way to add temporary virtuality to the Research Gallery. And, more exciting than compromise, we suspect that there are yet more ingenious patterns left to discover which keep the speed and agility of the digital world while gaining the stability, openness, and understandability of the physical world.

The gallery helps us see the whole

To understand something, you need to see both the parts and the whole. In the modern world, we have innumerable tiny rectangles for seeing parts. We flit back and forth between these screens, tabs, and windows at increasingly rapid speed, trying to keep sight of how everything fits together, but this awareness floats away the second we focus on a piece of the puzzle.

A theme at Dynamicland is using our spatial environment to see the whole. In the case of the Research Gallery, the "whole" is the broad arc of projects built here, the timeline they sit in, and the relationships they share to one another. By laying projects out in a large space, we build a stable map of this whole which we can use to orient ourselves within larger patterns and stories.

• The gallery is sorted by date, so motion across the wall corresponds to movement from simple projects to more complex ones. New capabilities in our system stand out as landmarks, creating ripples of experiments ahead of them.

  

• At the top of the wall, we have a traditional year-by-year timeline. Here, both clusters of projects and the gaps between clusters tell stories.

  

• Searching with the keyboard doesn't just create a list of search results. It creates a pattern of highlights on the wall. Search for a topic. The highlighted cards might suggest the emergence and decay of a trend. Search for a person's name. The cards might show their arrival and departure from the lab.

  

With the gallery, we can skim across months and years of work. Once we choose a project to dive into, the scrapbook gives a source of rich details. But the wall is still there to show where the project resides in the scope of Dynamicland's story. We can see the parts – but we always keep the whole in our peripheral vision.

The pattern of a linked book and gallery feels good to us. We wonder if it could find use beyond the Research Gallery.

For instance: The Realtalk OS has always lived as a set of whiteboards covered in pages of code. This is simple and transparent (and it helps keep us honest about our goal to maintain the OS's small size!). But we've found it lacking in deep understandability. We covered the whiteboards completely with code, and there's no room left to document the code's functionality, or to visualize in detail how the pieces work together.

What if the Realtalk OS adopted the Research Gallery's combination of a large space for "seeing the whole", and a book for "seeing the parts"?

The pages on the whiteboard would be sorted into an OS Book, which would be the complete implementation of our operating system. We could use all the familiar and powerful conventions of a book for this. Chapters would organize the pages into the OS's major components (translator, evaluator, renderer, etc). With the aid of annotations and diagrams, reading the book would tell the story of how the system comes to life, sensing input from the world and producing actuation (sound, light, motion). But since our system is dynamic, each page would also be a dynamic visualization of the process it covers. And not just a visualization, but a playground — try inputting new data and see how the the system responds!

Meanwhile, freed from storing the detailed "parts", the OS Wall would become a legible map of the whole. Each page of the book would be represented in miniature, and could point you to its location in the book. But most of the space would be used by large-scale visualizations, showing how data and activity flows through the system. And just like with the Scrapbook: as you page through the OS Book, the OS Wall would highlight your location and keep you oriented to the big picture.

As we build the next iteration of Dynamicland, we intend to embed this principle of "seeing the whole" in everything we do. We are excited to explore how this new book/gallery pattern can help.

Computers can guide physical actions

In the long run, our vision of spatial dynamic media goes far beyond projectors. If we are going to store and manipulate information using physical objects in space, we need to give our computing system the ability to move these physical objects just like people do. We imagine robots rearranging objects on tabletops and walls, and further down the line, objects rearranging themselves. But for now, aside from a few experiments, the Realtalk computing system has been confined to the ceiling. With a projector, it can throw a picture onto an object, but it can't throw an object.

However, several times in the course of this project we found that the right computer/human collaboration could mitigate this limitation. Instead of directing an as-yet-nonexistent robot to do something, a program can give a person the information they need to perform an action confidently. In short, people can play the role of actuators, with computer guidance.

Take “turning pages”, for instance. When you laser a project’s gallery card, a few bits of information are brought up right on the wall. If you’re interested in knowing more, you want to go to the project’s scrapbook page. This page is sitting somewhere unknown in a very large book. Ideally, the book would be robotic, and would offer you the option of flipping right to the page you want. But until we have robotic books, we'll be the ones turning the pages.

Our initial experience with this was frustrating. We found ourselves leafing through the enormous stack of pages without direction, searching up and down for the pages we wanted. As a result, there was an enormous chasm of thought and effort between the gallery wall and the scrapbook. The two were, for all intents and purposes, disconnected. Fluid movement between the whole and its parts was impossible.

To our surprise, this problem was extremely simple to fix. Our solution:

• Keep the cards on the wall sorted by project date.
• Keep the scrapbook sorted in the same order.
• Run a four-line program which says: "When the book is open to a project, highlight the project on the gallery wall".

Once we did this, the wall showed everything you needed to know to navigate the book. A person looking at the wall can see "if I flip forward 5 pages, I will find what I am looking for”. And as they flip, the wall will show their progress, live. This simple bit of feedback makes movement, if a bit mechanically involved, cognitively automatic.

For this to work, the gallery and the scrapbook must be kept in the same order. Keeping things in order is another mechanical process where the computer can help. We’ve made tools which overlay numbers and arrows onto cards on the wall, showing how they should be ordered. When new cards are printed out, these projections guide the humans in the sorting process.

"Finding a scrapbook page" and "keeping cards in order" are both examples of physical tasks where people do the moving, but computers guide their actions. These sort of hybrid tasks excite us for a few reasons. They offer a stop-gap until we have more pervasive robotics, making dynamic spatial media more useable in the meantime. But they also suggest long-term alternatives to robotics which put people in the loop, creating opportunities for closer awareness, and serendipity.

We will continue looking out for more collaborations like this, building a toolkit of “computer-advised actuation” design patterns.

We build the gallery improvisationally

In many creative pursuits, we are used to the idea of a permanent transformation. Wet paint dries on canvas. Clay is fired into pottery. There is no way to get back the original ingredients, or fix a mistake made long ago without starting from scratch.

The digital world is no different — MIDI scores and virtual synthesizers are rendered into MP3s. Vector files are rasterized into PNGs. Source code is compiled into binary executables, or minified into incomprehensibility.

But in other parts of everyday life, we are used to a breezy reconfigurability. We hang a birthday card on the refrigerator. We repaint the living room, move the futon into the den, and hang a calendar in the office near photos of our loved ones. We continuously adapt our environments to changing needs, casually collaborating with our cohabitants.

Realtalk brings this improvisational approach to software, and this has profound effects on how we collectively tackle a project like the research gallery. Software grows like a garden, never fixed or finished. And software grows in concert with the environment itself.

The first thing we did was find a nice couch and coffee table and drag them into location. This became our home for the project. During development, it was surrounded by sketches of our concepts and plans. We kept the preceding research gallery nearby to remind us of what we loved about it. We then built the gallery by pretending to build the gallery. We built the gallery from inside the gallery. We built the gallery the way we built the Realtalk Operating System — continuously, casually, capriciously and in good company. There were no commits, no builds, no deployment, no merge conflicts. There was pushing and pulling of tables and chairs.

At one point we wanted to reorganize the title cards on the walls chronologically, to match the way we'd organized the scrapbook itself. A few lines of code later and the cards were lit with a tangled web of arrows, each card pointing to another. Untangling this bramble would give us the time-ordering we wanted — it became a game. We were compelled to clean up this mess, and ended with a neat zig-zag of perfectly ordered cards. This spontaneously whipped-up sorting game is now a "feature" of the gallery — any time we want to add a new entry to the gallery we flip it and it tells us just where to place it. There is no distinction between this improvised scrap of paper and the "core software" that runs the gallery.

Searching the gallery, showing a timeline of events, lasering a video to turn on sound — all were added in the same way.

Going back to a traditional software development experience after spending time together at Dynamicland can be jarring. It feels like building a ship in a bottle. It's lonely. It's cramped. You can't see anything.

If you've ever watched an 80s movie-montage of everyone banding together to save the rec center with paint rollers and spandex, you've seen what collaboration at Dynamicland can feel like.

Conclusion

Building the research gallery was joyful and unique. To gather together with friends and make a space habitable and knowable feels warm and human, and is so far from what usually passes for collaboration in the world of computing that it feels silly to even compare the two. When we imagined experiences we wanted to replicate, spending time with family browsing old photo albums came up again and again. To our delight, our research scrapbook let us sit down with friends and visitors and share stories of the past while holding live, dynamic artifacts in our hands. "Programming" the research gallery was almost an afterthought relative to all the other types of activity that produced it — a few lines to highlight pages, a few lines to implement search, but mostly it was hand-drawings, interior design, craft scissors and Elmer's glue. This is what we believe computing can feel like, and we hope it is only a glimpse of what's possible in the future as we continue our research and follow these threads.

A special thanks to our wonderful Dynamicland community for helping to fill our scrapbook with delightful pages and memories.

Appendix

• For more on our stampede process.