Watch the video.

Here's a gist of the rectangle recognition algorithm:

I get the camera image and make it grayscale.

Now I need to find the sharpie markings. Originally I was doing this with an adaptive threshold which asks "is a pixel significantly darker than the 9x9 neighborhood around it?"

This worked well, but not great. The problem is sometimes parts of the marking are not dark enough. This is exacerbated by projecting the cats, which lightens the markings.

So I elaborate the adaptive threshold technique by taking two thresholdings: to find strong marks and weak marks.

You can see the strong marks are high-confidence dark areas, but that there are holes, places where the sharpie doesn't get detected. The weak marks capture most of the sharpie marks but it also picks up a lot of noise (white speckles).

I combine these two by doing some morphological operations (dilations followed by erosions) and then taking their intersection.

Aside: I would like to explore lighting the scene with IR and using an IR camera. (Luke is also exploring this.) The advantage of IR is that lighting can be controlled. I can ensure the camera has plenty of light regardless of the human lighting in the room. Further, there is no interference from the projector. A lot of the adaptive thresholding tuning was to cope with the cat projections making the sharpie markings look fainter. And a lot of the jittering of the rectangle position is due to feedback effects from the projection. Perhaps things could be simplified and the above steps could be replaced with just a simple adaptive threshold given a good IR image.

Now that I've found the sharpie marks, I find contours,

then I filter these contours to find the rectangles,

To do this, I take the area and perimeter of the contour. Consider the ratio of the perimeter squared divided by the area.

A square has ratio 16. A 2x1 rectangle has ratio 18. A 4x1 rectangle has ratio 25. As the contour becomes more slender and less rectangle-like, this ratio increases. By only taking contours whose ratio is < 35, I get rid of the snake-like contours I don't want.

We then approximate the remaining contours to quads and draw cats. Here is the code.