Alex and I had a good discussion the other day about layout analysis. That is, given that you want to do something virtual that is based on the layout of objects, how do you "parse" or recognize the structure of objects so as to most naturally program the behavior.

Holders from eToys came up. Holders are an object that you can put other objects in. The holder gives the objects an ordering and the holder has a "cursor" which always points at one of the objects. They are kind of like arrays in traditional programming. (Correct me if I'm wrong about that.)

The "cone of vision" sensor is kind of like a holder, especially if you give it a wide angle and talk about the set of objects in its vision:

In A Small Matter of Programming (in our library, see page 95), Nardi talks about Visual Formalisms. "Visual formalisms are diagrammatic displays with well-defined semantics for expressing relations." Examples include tables, graphs, lists, maps, outlines, plots. They are "visual, because they are to be generated, comprehended, and communicated by humans; and formal, because they are to be manipulated, maintained, and analyzed by computers." She argues that the spreadsheet's use of the visual formalism of the gridded table is a key to its success as an approachable programmatic medium.

Likewise, I think visual formalisms will play a big role in Realtalk. We want visual formalisms (like tables, grids, graphs, plots, lists, outlines) to be primitives which we can grab and put on the table. These formalisms might be embodied by large "holders" or enclosures, like a game board, or they might be tokens or cards that we place on or near a generic supporter (a table, a piece of paper) that tell the supporter to interpret the arrangement of objects on the supporter using the visual formalism. For example, the instrument tokens tell the nearest piece of paper to become a gridded sequencer in La Tabla's music activity.

What kinds of visual formalisms would provide the most generative potential in Realtalk? Are there "primitive" visual formalisms that you can use to make richer ones, the way you can use arrays and maps to make richer structures?

I mocked up a way to do the animation layout analysis using a "row" constructor. I put it together using the "magnet poetry" tokens that we've been playing with on the small table.

The idea is that you'd build up a query, which would do the layout analysis, piece by piece. And as you built it up, you'd always see live what objects you were selecting, either projected into a query inspector or projected onto the objects themselves, like when you hover an element in the Chrome debugger.

First you put down a "paper" query, which selects all objects that match "_ is a paper".

Then you augment this with "row of paper" which clusters paper objects into rows,

"Row" is a clusterer, a holder, a visual formalism. It's compositional. It modifies a query so you can use it to find a row of any kind of object. The row's orientation would be determined by context. Just as we had a "you" or "this" (the "self" part of context in object oriented programming), we might have a "here" part of the context which establishes the local frame and orientation (in this case, the table in a given orientation).

The query also highlights the subqueries (so far "paper" is the only sub query).

Next we modify "row of paper" to find only those rows "spaced apart less than 2 inch",

Then finally we select the paper above the row (the screen),

Maybe you would name these separate subqueries so that you could refer to them later, like Inform 7's "(called X)" construction,

A nice bootstrapping feature with this scheme is you could use the visual formalism to parse the layout of the word cutouts themselves,