Date: Sat, 12 Jul 2014 16:08:21 -0700
From: Dave Cerf
Subject: database evolution
Hi guys,

At some point last night, I decided I wanted to see how this movie-making FileMaker database has evolved over the last two projects I've worked on with Walter. The hardest part was figuring out how to automate exporting the schema for each database backup.

[All_Database_Evolution.m4v]

Evolving Code
This got me thinking about how interesting it would be to visualize the evolution of software structures (code, relationships in the code) over time. The database schema is already visual, so it's fairly easy. But code seems harder.

Evolving UI
Then I started thinking that it would be interesting to look at how the UI evolved over time. What's interesting about FileMaker is that it is a software development toolkit—it has data structures, UI elements, and scripting. It has always made me wonder why FileMaker Pro is popular among filmmakers. Partly, it's because they are keeping track of lots of data. But NLE software like FCP and Avid also keep track of lots of data. The reason must be that the tool is more flexible. You can make your own layout that suits your needs. And when that layout is no longer sufficient, it takes 30 seconds to adjust it and you're back to whatever task you were doing.

Flexible Tools
Now, FileMaker Pro databases tend to look terrible because they often aren't designed with any forethought. And their backend data structures can be equally terrible for the same reason. It is easy to chastise a novice FileMaker user for all the things they've done wrong, but at the same time, look how far they were able to get with a tool of their own making. Working on FCP X, I became exhausted by the corporate obsession to nail down the user workflow and design just for those cases. Obviously, you have to start somewhere or the software will attempt to be all things to all people, which is never good, but the goal of Simple Design did not allow space for Flexible Tools. The most obvious example is that FCP X has a single window design, and there's nothing that can be done about it.

Evolving Data
There are so many things I want to know about the movie I am working on. How long is it? How long did it used to be? How long might it become? How many shots are used? These are all relatively static questions that could be addressed by a scripting API (which none of the major NLE supports, maddeningly). But let's assume that I could write a script like "tell Avid to set clipCount to (count clips of sequence "Final")." I could run that script whenever I want and know how many shots are in the movie. But what if I want to know about how the number of shots has changed over time? In that case, I would need to anticipate needing this temporal data at the beginning of the project, or rely on some kind of automated retrieval of project backups to retroactively gather that data. What if it was assumed by the software developers that people using their tools would like to know about how their data has evolved over time? How might the software be developed differently to account for that? Such questions can obviously get bogged down in technical details, so it seems the important thing is to prototype examples of what project evolution might look like when visualized. 

Evolving Data for Making Movies
I'll start with what I know best: our movie is a constantly reordering data set, and I'd like to see how that reordering has happened over time. I am not sure the best way to visualize this but I do sense that visualizing it may reveal patterns that static numbers may not show. Even though movie and sound editing software deals with a moving medium, the movie structure itself—as stored by the software—is static at any given moment. A project's dynamism is limited to undo states whose navigation is almost as barbaric as fast-forwarding or rewinding an audio cassette. To easily see what has been done and to observe the transitions over the course of the project is almost impossible (today).

The idea of traversing states of software history in a visual manner is very compelling. I don't know yet what patterns it would reveal. Because video and audio are heavy weight media, creating enough historical slices (screenshots of timelines, thumbnails of entire sequences, waveforms) is time consuming just to get to the point of traversing them. And what if you capture the wrong data during your slice retrieval?

I realize that Robert has been working on a lot of the heavy lifting for precisely the reasons I describe (and more). This is all very exciting…

More to come,
Dave