Diffing two filesystems

Is there any sensible way to diff two filesystems?

On occassion, you have a box that is working, and, say, perform an update and it no longer works. You have dozens of other machines that are (theoretically) identical, save for user data [and information tied to the specific hardware, and, say, ssh keys and so on]. Is there any way to determine what is different between the failed machine and one of the good machines?

I've tried mounting volumes and running ls -l or md5deep and recording the results and diffing that. Heck, I've even tried mounting two filesystems and just plain diffing them (with a graphical tool, mind you). The results have not been as useful as I'd expect.

Part of me wonders about having a database of files on each machine and what their md5sums are, with a cron job to update it on occasion, but I'm not even sure that would be as useful in practice as it sounds in theory.

So, is there any sensible way to diff two filesystems (or even, diff one filesystem against itself at a different time)?

From serverfault Clinton Blackmore

• Use rsync with --dry-run option.

  For monitoring local changes, use tripwire.

  Clinton Blackmore : Now that's brilliant. Thanks.

  Zoredache : Also check out AIDE as the OSS solution (http://www.cs.tut.fi/~rammer/aide.html)

  Ian Kelling : For those of us who use this regularly, -n = --dry-run

  Ian Kelling : If you want to be thorough, use the checksum option of rsync.

  hendry : Rsync does not give any intelligble output. hendry@x61 tmp$ rsync -cPnra foo bar sending incremental file list foo sent 64 bytes received 15 bytes 158.00 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) Is bar different to foo, vice-versa or the same ?
  From vartec

• There is a tool called fsdiff [I could not find the homepage] that will compare a filesystem to its transcript. A free, multi-platform tool called radmind includes it and claims,

  At its core, radmind operates as a tripwire. It is able to detect changes to any managed filesystem object, e.g. files, directories, links, etc. However, radmind goes further than just integrity checking: once a change is detected, radmind can optionally reverse the change.

  From Clinton Blackmore