[RESOLVED] Performance issues on August 27 2015 - Delays

All - we are looking into an issue where incoming job queue is not being managed in a timely fashion.

We have put in a measure to prevent incoming jobs, which will resolve this sooner.

The issues at hand include:

  • Reports from newly installed agents will not show as expected.
  • Reports from existing computer records will have a delay before they are shown in the dashboard
  • Notices of reported issues will be delayed.
  • Group-specific installer builds will be delayed

We will edit this post with details as we know them, and update it (send another email) when the status changes.

The issue has been tied to how we were sending requests to Zendesk while they were experiencing issues.

Better countermeasures will be implemented in the coming days. Full reporting should resume in the next few hours.

The dashboard remains available, and we are processing existing reports. Full operation is expected to resume within the next few hours.

We’re nearly caught up from the outage. As we complete this, we’ll be taking the Dashboard down for a brief (15-20 minute) window while we move to a database server which will allow us to handle more connections, allowing for better formance when dealing with external ticketing systems. :wink:

Last update on the matter here:

We’re back in business, running at full speed, and ready to get back to designing the Next Big Thing for Watchman Monitoring.

All our best to you!