I have a projector, an RGB camera, and an IR (for retroreflectors) camera all mounted to each other, mounted on the ceiling over the in-progress token table.

I mark the borders of each pseudoscreen with retroreflective tape. Here's a sample of what the IR camera sees:

To find the quads, I take the convex hull of each contour and then approximate to a quad. Taking the convex hull makes it robust to edge occlusions, though it currently will fail if you cover up a corner. I'm sure this can be improved.

The RGB camera is not used for this prototype (except during the calibration procedure), but I would eventually want to use it for visually ID'ing the pseudoscreens and other tokens.

First the RGB camera and IR camera are calibrated to each other using the green calibration thingy.

RGB:

IR:

Then the RGB camera is calibrated to the projector by projecting the calibration pattern onto a large piece of foamcore.

Projection:

RGB: