Date: Mon, 21 Aug 2023 13:37:45 -0400
From: Luke Iannini
Subject: Re: CCC talk
Yeah, I love all these and I'd be super excited to think about a Setup Kit — I agree that the nonlinear optimizer is far too black-boxy, and camera/projector calibration seems like it should be amenable to a simple visible geometric implementation. I can also imagine taking advantage of the ability to recognize dot patterns in projected images to implement straightforward auto-calibration without Gray codes, and it looks like connected laser range-finders exist which I think would let us calculate real space from a projected image given known projector intrinsics. (Lens distortion should also be something we can calculate from projected dots)

We should also be able to calibrate a machine from another machine.

I'm imagining Setup Kit could be a modular collection where each component knows how to instantiate itself in a keyboard/touch-controllable Box for before calibration (or on a tablet screen, especially useful for calibrating other machines now that we can stream video!), or on the page itself for after-calibration tweaking with physical controls ("knobs" etc) in a book or poster, where there will be ample room for visible CV pipelines, etc!

There's probably some Raspberry Pi project to turn it into a remote-controllable router (or a hacked router like the old Linksys WRT54G projects) for Realtalk-configurable networking.

Big agreement to lamps being actual lamps with Realtalk-controllable diffuse lighting!

If we wanted to get fancy we could couple an IR emitter directly to the projector's IR receiver and configure our projectors from Realtalk too.

On Aug 14, 2023, at 12:10 PM, Bret Victor <****************> wrote:

A few ideas for making presentations in Realtalk go more smoothly:


Setup Jam.  Realtalk has come a long way since the Realtalk-2020 calibration mode was designed.  We can now make use of lasers, Dynapads, Realtalk-connected web pages on our phones or laptops, plug-and-play displays...  Meanwhile, setup has also become more complex:  we have many cameras by default, we want to travel around, we want to light up walls, shelves, giant DNA tubes...

I can imagine a "Setup Kit", which subsumes the duties of the current calibration mode, but is also designed around getting network connectivity, positioning the projector, positioning and focusing the cameras, adjusting lighting and exposure, basically everything you need to do to get Realtalk up and running in a new place.  It would also be designed for lighting up any kind of space or object (such as a giant DNA tube), without assuming that a console can be projected on it.  

Ideally, calibration could also be more visible and directly controllable.  Currently, the nonlinear optimizer is a bit too "magical" -- a slight nudge of a calibration point can give drastic shifts, and it's not clear what's going on.  You should be able to stitch supporters by manually steering them around until they look right, etc.  Other things to think about:  autocalibration, realspace calibration in places that can't be physically reached, lens distortion correction...

It would be fun to have a Setup Jam where we talk through ideas for next-generation Realtalk setup!


Networking.  It was great to have a local router for our Realtalk machines!  We just need a better way to connect the router to the internet, ideally one that can be controlled from Realtalk itself.

For switching between wifi networks, I can imagine having a playing card for each known network (home, travel router, etc.), plus a card that acts like wifi buddy, but then generates a new card for the chosen network.


Lighting.  Like at the Stripe conference, we should assume that we can't rely on ambient lighting, and diffused mounted lights should be part of the kit.  Dynalamps should probably return to being actual lamps.  Even better if the light can be controlled from Realtalk.

I can imagine a next-generation mounting plate which incorporates lighting (as well as mirrors for more easily aiming the projection and cameras...) (as well as cable organization and strain relief...) (as well as a mirror around the projector's IR receiver so the remote control works reliably...) (as well as a water-cooled projector heatsink, sigh)


Laser dots.  Projector interference is apparently a problem in certain environments.  I'm sure we can address this with some simple processing of the camera image (maybe even just a threshold).  But I wonder if it can be done in a more Realtalky way than just a numerical parameter in a setup screen.  E.g., a page with a printed signal-flow diagram, where you adjust the parameters in context and see the intermediate images.  (And likewise for the entire dot-processing system, etc...)