How do I audit changes made to our servers, routers, etc.?

We have a lot of servers (running Windows and Ubuntu) along with a mix of Cisco and Juniper routers with a side of HP Procurve switches. We have a few sysadmins who like to make changes to configs without telling anyone or documenting it anywhere. You can imagine how this can blow up in your face.

Is there any software out there that can log and audit configuration changes in Windows, Linux, and Cisco devices? Or if there's no software, maybe some policies that can effectively stop this sort of behavior?

From serverfault zippy

• http://serverfault.com/questions/140347/record-everything-on-command-line-centos-fedora-ubuntu

  ---

  How to keep a detailed audit trail of what’s being done on your Linux systems

  zippy : I'll have to check this one out.
  From Felipe Cardoso Martins

• Let me side-step some gratuitous comments about the apparent lack of control in your environment. My apologies for the situation; try to reign those cowboys in :)

  Definitely look at Rancid for your networking needs. You can monitor changes to configurations. Additional integration will let you automate backups upon detection of configuration changes based on Syslog messages or SNMP trap notifications.

  For Linux, consider forcing admins to access hosts through a logging portal (like an SSH jump with a ForceCommand that wraps script(1) before connecting to a destination host). Venerable tools like Tripwire can log inappropriate changes made to system files.

  For Windows, check out the pretty software from ObserveIT, which can do host-based monitoring of interactive sessions.

  Given that you seem to have already had some face-blowing-up going on, I strongly encourage you to foster a culture of responsibility about this (a "soft" control / policies). Some admins do behave like cowboys, but surely most understand that undocumented. unannounced changes lead to problems. Establish work windows, production blackouts, change notifications, etc.

  This are simply smart practices, which they and customers will come to appreciate; the admins because they'll be able to find out when they shoot themselves in the foot more easily and customers because they'll feel more aware of what's going on.

  Andrew : ... and I nearly posted "you need puppet for switches"!

  zippy : I'll take a look at ObserveIT...looks pretty good.
  From medina

• Manage your users sysadmins: As Jeff said, try an illustrative story:

  At my last job we had X thousand users on Y hundred devices and Z dozen sites. Some bright spark decided to change the configs on all Y hundred devices and reboot them remotely to apply the configs.

  1 day later, the config started failing. He ended up having to drive to every site to reset the devices using a serial cable and a laptop. He learnt his lesson and never did it again. Now he's a manager and makes sure his sysadmins don't repeat his mistakes.

  Or whatever suits.

  (The above is an exaggerated version of my own experience with a remote firmware update on a switch; we had to bring the switch back from the site and re-flash it using a serial cable.)

  From Andrew