Way cool! I think this neatly wraps up a bunch of our dreams of a true leap in recognition into a cohesive vision!
Some gameplans...
For recognizing cards and matched tags I'm imagining a second pass at the upgraded contour detection I was playing with before I left, with the most important addition being recognizers with a visible pipeline where each point along the way has some tunable parameters — basically OpenCV made into a nice improvisational kit, and the ability to collapse a pipeline down tunable-snapshot-style once you've got it as you want it. I think I've got a good sense of how to do this — I'll get the framework in place and then we can all play with it and pluck useful functions out of the OpenCV menagerie as needed.
For manipulating 3D stuff, Apriltags give us depth, position and orientation so we can use that to make the spaceball-tennis-ball work as a 6-DOF controller with e.g. 6 Apriltags on the poles of each axis such that at least one is always visible to the camera, obviating the secondary flagpole or depth cameras and then you can just have a spaceball per molecule with varying lengths of paper roll for height. I think I got the kinks all worked out in the "3D-printed heart" research prototype from a few months ago so it's just a matter of splatting it into the system.
For recognizing handwriting I'm (confidently) hopeful my hard-won hand-drawn shape recognizer is going to make all our dreams come true. Hand-drawn edits should be possible with a parallel recognizer piggybacking on the same fine-detail pipeline that will be delivering shapes to the shape recognizer.
If Apriltags and shape-recognition work as well as I hope then those will also be an option for e.g. the selection pointers and the matched tags (e.g. draw a unique glyph twice to create a pair of matched tags).
For recognizing printed documents, yes I will get us some SIFT to try out as part of the aforementioned OpenCV Kit and I've also got some image hashing algorithms in my queue I've been meaning to try out for that purpose (the only thing I want to prove out is how many objects we can register with a SIFTy system before it starts to slow down, but I think we'll be using a small enough number to be fine no matter what and also we can get around latency with persisting SIFT recognition via inter-frame shape coherence rather than doing the full SIFT N*N match every frame)
3D scenes with manipulation is well underway!
Party on,
~Luke
On May 12, 2022, at 7:27 PM, Bret Victor wrote:
Here's a video mockup of what Protein Kit could look like with improved Realtalk capabilities.
A list of the main points:
- recognize cards without dotframes
- search for proteins by handwriting search term
- matched tags for "putting that there"
- recognize printed documents
- linked 3D structure and 1D sequence
- selection highlighting
- shared annotations
- make 2D maps/schematics
- by referencing both 1D and 3D
- annotated (e.g. "small helix", "turn")
- 3D scenes
- physical 3D manipulators
- hand-drawn edits
- structure prediction for edited proteins
- visualize the experiment
- gels
- plates, EM images, plasmid maps, tube racks...
- plan the paper
- snapshot images for paper or group meeting
- pose your molecules as you want them in real life
- then add the peptides or origami to pin them there