[Resolved] Sept-Oct 15 - Many restart warnings on Crashplan (Green/Blue)

We are investigating reports of out of memory warnings from Crashplan green, and will update this thread with more information.

(If emails from this thread is not useful to you, you can opt out of just this topic using the links below)

1 Like

Datapoint: We’ve received 4 warnings this morning, 3 of which are on always-connected-to-the-Internet Macs.

Logged onto one these Macs. Could not access the Crashplan menulet at first—other menulets were fine, so I launched Crashplan itself. Notably, it said, “Analyzing: Library > Mac-MSP > Gruntwork > chiefact.txt”

I just received a notice from Crashplan this morning that this Mac was 100% backed up, for what that’s worth.

As I’ve typed this, Crashplan has moved on passed the above Gruntwork area of the library. Not sure what might have stalled it out, but seems to be going now.

Hope that the above is helpful.

1 Like

This appears to be a case where Code42 instructed a series of restarts.

The warning we sent is one-time and will clear itself on most systems. If the plugin state returns to normal/green on the next hour, no action is required on your behalf.

Notices of restarts are useful in the case where the Crashplan client is out of memory, or crashing for others reasons. In these cases, the local client cannot reliably know if it has completed backing up, and will not report properly to its Crashplan server (or to Central, in the case of Blue or Green)

We have found as many as 27,000 restart logs on computers, and in those cases almost no data was protected. To that end, we will continue to look for this case, and of course will work to keep false positives at bay.

We’ve had this problem on two different computers where Crashplan was repeatedly crashing partway through the backup and yet Crashplan itself both on the computer and on the server was reporting completed backups.

Crashplan backups up files in alphabetical order and would consistently crash about the time it got to folders starting with “G”. Everything before was backed up properly. Everything from “G” on had not been backed up for several months since the crashes began. We only figured out the problem when we attempted to restore some files for the client. Fortunately, the files were located in a section of the file share early in the alphabet, but it was then when we noticed that folders recently added to the root level of the share that came alphabetically after G were missing completely.

A very scary moment for us when we realized this issue. A few false positives for me is a much better issue than customer data loss.

Hi All

This past week Crashplan released a number of new versions of the Green & Blue versions.

In some cases, versions went from 4.3.4 -> 4.4.0 -> 4.4.1, rather than right to 4.4.1. Each resulted in its own restart.log

Additionally, the Crashplan upgrade scripts moved restart logs into /Library/Logs/Crashplan, artificially increasing our restart count. The result was many sporadic warnings that Crashplan had restarted.

We’ve pushed a small fix for this in 6.3.3 build 337, and a future version will both report the Crashplan Client version, as well as skip any restart notices if the version number incremented in the past hour.

We want to thank all of you who reached out for support in the face of unclear notifications. Your actions allowed us to create a better product.

Sincerely,
The Team at Watchman Monitoring