Some you noticed that our website was offline last night, it’s back up with a whole new look and feel. Check that out at http://www.watchmanmonitoring.com/ and let us know what you think!
We also ended up replacing the database servers last night, in what should have been a one-hour procedure.
Sadly the rightscale based images have only 10GB boot volumes and the restoration process we’d initially planned failed to complete due to lack of /tmp space, then again when the logs filled up the boot volume.
The database is 1/2 done importing now, and the server should be back in action shortly.
We’ve got some great support from Rightscale through the process, and they’ve made a number of notes for improvements which could speed up similar situations in the future.
Allen Hancock - Watchman Monitoring
Proactive Support starts at http://www.watchmanmonitoring.com/case-studies