[ramble] [some thoughts prompted by Paula and Toby's recent mockup]
Bring together a piece of paper, a pen, a pair of scissors. It doesn't matter who made them or where they're from, they'll probably work together. They might work together in different ways -- different types of paper respond differently to different pens -- that's one of the pleasures of learning the materials -- thick paper might be tough to cut with normal scissors, so you get the shears. But it's generally a very "open" system of objects, defined by their physical properties rather than their origins, interacting in physically apparent ways.
In Photoshop, you can only use the Photoshop paper, the Photoshop pens, and the Photoshop scissors. You can't use ArtRage's pens on FiftyThree's Paper, that doesn't even make sense. ArtRage's pens don't exist outside of ArtRage, in the same way that Mr. Darcy doesn't exist outside of Pride and Prejudice, they're fictional characters within their own fictional universe.
The application silo is a tragedy for any number of reasons -- vendor lock-in, users' pet-like dependence on a vendor to provide what they need, the inability for users to solve their own problems by making their own tools, the inability to share tools, monoculturalism, etc etc. Even a "plug-in architecture" presupposes a given application that's being plugged into; it's as if you could bring in your own paper, pen, and scissors... as long as they were specifically designed for your brand of desk.
---
The dream of virtual objects that can be used in any context, regardless of their origins, extends back to Alan's original vision of objects ("we made objects too small"), and can be seen in Raskin's "commands and transformers", maybe OpenDoc and OLE, and so on, etc.
How do these diverse components communicate? One approach is standardized protocols. Another approach is "communicating with aliens", discovered or negotiated protocols, "call by meaning".
---
In our room, you could say that we're trying to escape the tiny rectangle by doing knowledge work with gently magicalized physical objects; seeing and thinking and building out in the physical world.
The current implementation is a kind of "operating system", which makes it very much a silo. An object's magic exists as an entry in a particular database, interacting with other objects through this database. The frog looks like a physical object, but its soul is bound to this room; you can't carry the frog out the door any more than you can carry Mr. Darcy out of the book. Picture Hobbes when Calvin isn't looking.
Obviously, we don't want this, we want physical objects that can be used in any context, regardless of their origins. Objects that will "do their thing" wherever they are and on whatever they are being used on. We want our magic paper, pen, and scissors to be as reliable, universal, and portable as the mundane ones.
---
We can talk about physical objects that communicate via protocols, either standardized or discovered. We can go ubicomp or IoT, objects
exchanging data with objects. We'll have to, in the short term, because we don't know any better.
But it's not right. It's data-centric thinking. It's "computers exchanging data". We (as a field, or maybe a species), are stuck thinking of computers as "processing data" and "communicating data".
But the way that scissors cut a piece of paper isn't "data". Neither the physical interaction between the blade and the paper, nor the human's tactile experience. It could conceivably be made into "data", using "sensors", but somehow this feels "old world" to me, "epicycles", an unnatural framework imposed because we don't have the better way of thinking.
---
Instead of data-centrism, reducing the world to data so our computers can chew on it, we want... what? Physicality-centrism? Things happen, the computation proceeds, because physical things push on other physical things, things influence each other in the world, directly, in a way that we can see and feel and control and understand with our eyes and hands?
Of course, in the computers-exchange-data model, it must also be the case that physical things influence other physical things, otherwise nothing would actually happen. But it's mind-bogglingly indirect. A sensor on a stylus converts pressure into a number in digital memory, which is converted into radio waves, which are converted into a number in digital memory, which continues in its unfathomable contortions until some pixels on a screen change color.
Thinking of the pressure of a pen against paper as "data" gives enormous leverage, it's the CS cliche's "additional level of indirection", allowing this pressure to "mean" just about anything imaginable. But at what cost. The entire interplay between brain, eye, hand, pen, paper, desk, gravity -- the hundreds of degrees of freedom, the conversation with the material, the texture, the craft -- modeled as a single number.
"We don't know how to measure what we care about, so we care about what we measure."
That measurement is then used to simulate -- to generate a crude illusion -- of what would have happened if it had been a physical pen on physical paper.
---
In the short term, we'll build OSs and define protocols, we'll sense data and compute with it and actuate from it -- A-to-D and D-to-A.
But long-term... the computation needs to be analog. The computation needs to be analog! The computation needs to operate in the physical domain, instead of translating back and forth from some hyper-simplified numerical model.
What does that mean? I'm not sure exactly, but here are a few "hints" it makes me think of.
---
It used to be the case that pushing your foot on the car's accelerator pedal would directly control the gas valve. Now, there's some complex and hidden software-based control system between your foot and the engine. But the output of this computation is not a simulation, not an illusion. The car exists, the wheels exist, the road exists. You are feeling the real road under your real wheels, and your performance on the gas pedal is a direct response to what the real car is doing, not to the computer's model of what the car is doing.
The purpose of the computation is to modulate one stage of a physically-coupled feedback loop. The wheels, road, and driver are not "exchanging data".
---
This is a form of mechanical computation:
https://www.youtube.com/watch?v=s1i-dnAH9Y4&t=455 The components of these systems are physically coupled, they influence each other by directly touching each other, not by "exchanging data". It's hard to argue that these antiques have advantages over the sublime flexibility of software, and yet... They still exist when the power goes out. They even still work when the power goes out. You can grab the mechanism and drive it yourself with your hands, put your eyes and body in the feedback loop, feel the response, learn to finesse it.
What if these mechanisms weren't fixed, but dynamically and instantly reconfigurable -- like motorized audio faders, perhaps, but much finer-grained -- so you could get the flexibility of software in something that "computes in the world, on the world"?
---
Carver Mead's work with Analog VLSI -- building vision and hearing and touch systems, artificial retinas and cochleas. Commercially, a lot of it ended up generating "data" for "computers" -- the Synaptics touchpad, the Foveon camera -- but... could it just "be itself"? Could these inherently parallel, inherently analog, coupled-to-the-physical-world computational systems be "the material"? Or be incorporated into actuating material?
Perhaps this material wouldn't be "programmed" so much as have interesting responsive properties that people could learn to work with.