[Stream-of-thought, trying to figure out something out at the end of a long tiring day. It's not figured out yet, but there are some footholds. I can't guarantee this will make sense.]
[written last night]
I was thinking about ************************ on a long drive today, because it's superficially similar to what we're setting up in the lab now -- projectors and cameras onto surfaces, dynamic interaction on physical objects in the space. But I think it's philosophically very different, and it felt useful to unpack that -- the contrast with other projects can give clarity to the meaning and purpose of one's own project. And lord knows I could use some of that.
This isn't intended as a disparagement of ******** in any way -- it's just an exercise to try to understand my own motivations more deeply, by exploring how a similar-looking project feels at odds with them.
********. ******** is "Ad Hoc Interactive Applications on Everyday Surfaces". It places UI controls and dynamic information on any surface. It turns every surface into a screen. It makes an arbitrary physical object capable of hosting arbitrary virtual objects.
There's a snowclone I think, of the form, "if everything is x, then nothing is." where x is "important" or "urgent" or "beautiful" or something? There's something here like this, I can't quite figure out how it fits, but it's something like, "If every physical object is capable of meaning anything, then no physical object is inherently meaningful." ... Or maybe, "if every physical place is capable of being any virtual space, then physical place becomes meaningless." ? something like.. if you can have the same UI no matter where you are -- on the couch, at a desk, in any room -- then it doesn't matter where you actually are, and the places themselves lose meaning. Because why go to the couch, or the desk, or this room or that room, if they are all just hosts for the same virtual world? Why move at all? And why pick up this object or that object, if their physical form has been cast to the Object base class, and they are simply shells for some virtual representation? Their physical identity, and physical characteristics, become irrelevant.
In a way, it almost feels like a form of virtualization even more insidious than VR, in that it appears to embrace the physical world, but instead devours it by overlaying a virtual world on top.
There's something about physical identity that seems so important to me [which I don't understand yet -- this is what I'm trying to figure out here; I don't have it yet] A book (for instance) is *this* chunk of atoms, that lives in *this* shelf location, and can be read on *this* couch. Your spatial mind can *locate* it. It has a home in your head. Maybe that's not a great example, I'm groping for something here, maybe better is something like -- in a kitchen, the knives are on *this* counter, and the spices are in *this* drawer, etc, and in this way, for the chef, the kitchen can become an extension of his body (extension of his mind? synonymous?). And the knife counter and spice drawer are given meaning because they appear in the chef's mental map. And the knives are given meaning because each one is an individual, *this* knife is for paring, *this* knife is for chopping... There's a *reason* to grab one or the other.
The situated cognition thing. That thinking is interacting-with-the-world, or at least a model of the world that you've built up by interacting with the world. I think that's the thing about ******** -- it's "anti-situated". You can situate yourself in arbitrary physical space, and interact with arbitrary physical objects. The physicality is meaningless, the only meaningful mental maps are of the virtual world that the physical stuff is proxying.
I've thought before of the escaping-the-rectangle stuff as maybe a kind of "situated computing", analogous to situated cognition. Where meaning lies in the physical situation -- where you are and what's around you -- instead of physicality only being used for affordances to virtual representations. (The mouse, for example, is a physical object... but its only function is to point into virtual world. It doesn't have physical *meaning*.)
A poster of a video grid is a thing-in-the-world. It is either here or there, you can hold it and walk around it and point to it with people and discuss it with people and *know* that it's there. I want to use the projector-camera-stuff to bring just a little bit of magic into the poster, to bring it (to the extent we can) into the dynamic medium. But not to rob it of its physical identity! To me, it feels totally totally different than *displaying* a video grid on a screen or blank poster or wall, which can then poof away and be replaced with something else. The "displayed" grid is virtual, it doesn't exist, you can't think about it spatially or locate it in your mental map (maybe you can while it is being displayed, but then *poof*). The poster grid exists!, undeniably and irrevocably.
And then, outward, toward constructing a workspace around yourself, a "kitchen", with video grids or whatever meaningful objects whose existence you can locate and trust. Where every clump of atoms around you is an individual, something meaningful and relevant to what you're working on. Where it truly matters that you are in *this* location, because what you're working on and what you need to work on it is *here* and can't be *there* -- that's what makes *here* meaningful, having a *reason* to be here and not there.
******** (and every other system of virtual representations, even computer screens) are coercing the world into lying to you. Coercing objects into not being true to themselves, into faking it. Is this computer screen a calendar, or is it a list of mail messages, or is it a newspaper, or is it a game? It shows different faces at different times, how can you *trust* that? How can you be-in-the-world, how can you situated-think, when the world is an ever-changing illusion? When you can't think about *this* object as being a calendar, and *that* object being a mail message, but there's just this one shapeshifting object (the screen) which is "everything and nothing"?
wikipedia Situated cognition is a theory that posits that knowing is inseparable from doing[1] by arguing that all knowledge is situated in activity bound to social, cultural and physical contexts.[2] ... situativity theorists suggest a model of knowledge and learning that requires thinking on the fly rather than the storage and retrieval of conceptual knowledge. In essence, cognition cannot be separated from the context. Instead knowing exists, in situ, inseparable from context, activity, people, culture, and language
"Cognition cannot be separated from the context." The reason I want to cobble together projectors and cameras, I think, is to create a dynamic but *knowable* physical context. I want to give dynamic magic to meaningful individual physical objects -- to make this label a magic label, and this poster a magic poster, even this wall a magic wall -- magic-enhanced objects that *exist* and have been placed somewhere for a reason.
[next evening]
Adjectives vs nouns. I think what feels right is to use the projector etc. to reveal an object's *state*, not to generate its *identity*. For example, a frame in the video grid might be highlighted to represent that it's currently playing. This is different than projecting (an unstable illusion of) the frame itself onto a blank surface. Our projector provides adjectives, not nouns. This allows the nouns to have physical identity, to have a knowable place and form, to exist-in-the-world and be true to themselves. To be real, not virtual.
I think this is related to why I'm uncomfortable with designing telepresence systems. Telepresence virtualizes people. When you are teleconferencing, you are not interacting with a person, you are interacting with a picture of a person. (And if virtualizing a thing feels wrong, virtualizing a *human being* feels like an even more violent violation.) There's a lot more to say about telepresence (and I've sort of said some of it to Aran, who responded with "okay, that's a bit... extreme") but I might be able to understand it better with this identity/existence viewpoint.
Might also be related also to why I prefer models to description? A description is the most extreme form of virtual representation; it doesn't even offer an illusory physical existence. This morning, some people were talking about whether a ribosome is an organelle, without a clear picture of what either a ribosome or an organelle is, so the discussion was mostly on the level of definitions (what is the definition of an organelle, does a ribosome fit that definition), which to me, is at the same level as "is Pluto a planet" discussions -- it's about an arbitrary label, rather than about meaning -- what a ribosome does and how it does it. That is, it was a discussion that could only take place in description, because if it took place by seeing and manipulating models, arbitrary labels and definitions wouldn't even enter into it. You'd be working with meanings, not virtual representations made of words.
I think all this is closely related to what Craig Mod was describing
here as the tension between the physical and the digital, e.g.:
There’s a feeling of thinness that I believe many of us grapple with working digitally. It's a product of the ethereality inherent to computer work. The more the entirety of the creation process lives in bits, the less solid the things we’re creating feel in our minds.[3] Put in more concrete terms: a folder with one item looks just like a folder with a billion items. Feels just like a folder with a billion items. And even then, when open, with most of our current interfaces, we see at best only a screenful of information, a handful of items at a time. ...
When all the correspondence, designing, thinking, sketching — the entirety of the creative process — happens in bits, we lose a connection. It's as if all that process is conceptually reduced to a single point — something weightless and unbounded. Compounded over time, the understanding of where one is as a creative in a digital landscape collapses to the just-a-little-while-ago, the now, and maybe the tomorrow
except that, for me, the dichotomy is not physical vs digital, but physical vs virtual. Maybe it's just semantics, or maybe it's just that Craig lives in the today-world, where almost everything digital is indeed virtual, so the distinction is invisible. I see nothing wrong with digital (computation, bits) itself. The problems he's describing -- the intangibility, ethereality, boundlessness, formlessness -- are those of virtual representations. Infuse computation into physical objects, let objects keep their physical identity -- (
intangibility -> tangibility,
ethereality -> physical location,
boundlessness -> edges,
formlessness -> physical form
) -- but let them use computation to express themselves dynamically. Is that it?
(And obviously obviously I'm not talking about internet-of-things SmartToasters, right? I'm talking about physical objects used as representations of thought and tools for thought, in which the purpose of the dynamicness is to allow them to be thought with more fluidly, or to represent dynamic informative attributes (such as the currently-playing video frame).
[okay enough]