Hi all
Some you noticed that our website was offline last night, it’s back up with a whole new look and feel. Check that out at http://www.watchmanmonitoring.com/ and let us know what you think!
We also ended up replacing the database servers last night, in what should have been a one-hour procedure.
Sadly the rightscale based images have only 10GB boot volumes and the restoration process we’d initially planned failed to complete due to lack of /tmp space, then again when the logs filled up the boot volume.
The database is 1/2 done importing now, and the server should be back in action shortly.
We’ve got some great support from Rightscale through the process, and they’ve made a number of notes for improvements which could speed up similar situations in the future.
More soon
–
Allen Hancock - Watchman Monitoring
+1-408-352-6145
Proactive Support starts at http://www.watchmanmonitoring.com/case-studies
Well, that was exciting.
Way back when, our servers were built in MySQL 5.5.20 on CentOS. They’ve been wearing under the load and the tweaks carried out two weeks ago didn’t have the intended effects. It was time to get the boxes more up to date, and we had a plan which would allow us to safely take down both servers at the same time, allowing us to change the EBS volume size as well.
The plan would have worked to, but for a lack of space on the boot volumes (despite two 40GB SSDs being called upon to do the work.
That, and the size of the database turned into a number of false starts, with restoration failures only becoming evident after 20+ minutes of waiting.
In the end, the database dump was restored, and no data was lost. We expect that any further database work to be completed using our traditional, seamless, database failover plans.
During the time your monitored computers were unable to report their monitoring payload, they held on to the report, and did not check for new errors. Now that reporting is available again, the agent wills scan logs from the point they left.
If you have any further questions, or are just curious, please don’t hesitate to ask.
Be well
-Allen Hancock