RECAP - For customers who have purchased BinaryLane-managed backups, we have restored the most recent backup and your service should be online.
For customers who are not using BinaryLane backups, we have re-installed your operating system and sent a new login password via email. You should now proceed to install the software you require (HTTP daemon, etc) and restore your data from third-party backup solution/re-upload from your local PC as appropriate.
We have not re-installed the server if you logged in to mPanel since this event occurred - in this case, you will see mPanel display "Running: No". To proceed with reinstallation, go to https://www.binarylane.com.au/mpanel/reinstall and set Deployment Script to "None"
Posted Nov 06, 2016 - 20:09 AEDT
Update
We have begun reinstalling servers for all customers without BinaryLane-managed backups who have NOT logged into mPanel since 18:30AEDT; a new root password will be emailed to you.
Posted Nov 06, 2016 - 19:10 AEDT
Identified
We have begun to perform restores on servers with BinaryLane-managed backups.
Posted Nov 06, 2016 - 19:05 AEDT
Update
We have completed hardware replacement and are currently re-imaging the host.
Posted Nov 06, 2016 - 18:25 AEDT
Investigating
bnecompute07 array has suffered a similar failure to earlier this morning. We will soon be onsite to replace the controller and disks.
Posted Nov 06, 2016 - 16:43 AEDT
Update
RECAP: Host node bnecompute07 suffered a complete loss of all VPS data at approximately 9:30 AEDT. If https://www.binarylane.com.au/mpanel/manage shows "Host Node: bnecompute07" your server is affected.
For customers who have purchased BinaryLane-managed backups, we have restored the most recent backup and your service should be online.
For customers who are not using BinaryLane backups, we have re-installed your operating system and sent a new login password via email. You should now proceed to install the software you require (HTTP daemon, etc) and restore your data from third-party backup solution/re-upload from your local PC as appropriate.
We have not re-installed the server if you logged in to mPanel since this event occurred - in this case, you will see mPanel display "Running: No". To proceed with reinstallation, go to https://www.binarylane.com.au/mpanel/reinstall and set Deployment Script to "None".
Posted Nov 06, 2016 - 14:50 AEDT
Monitoring
[Customers with BinaryLane backups]
We have restored all affected servers from your latest backup.
[Customers with third-party backups]
We have begun reinstalling servers for all customers who have NOT logged into mPanel since the data loss, and new root password will be emailed to you.
If you HAVE logged into mPanel, we will not do the reinstall action and you should carry this out yourself (or clone from another BinaryLane server, etc). To reinstall, go to https://www.binarylane.com.au/mpanel/reinstall and set Deployment Script to "None".
Posted Nov 06, 2016 - 13:38 AEDT
Update
If you do not have a BinaryLane-managed backup, you may reinstall via mPanel now.
Posted Nov 06, 2016 - 11:53 AEDT
Update
Two drives on host bnecompute7 in a mirrored pair have failed in such a way that VPS disks are not recoverable - that is, this is a data loss event.
Where customers have purchased backups with BinaryLane , we will begin restoring the most recent backup shortly. For customers who are storing their backups with a third party solution, we will reinstall the server from its base image shortly if you have not already done so yourself.
Posted Nov 06, 2016 - 11:23 AEDT
Identified
We have brought the host back online but are currently unable to access VPS disks. We are currently working to determine whether the problem can be corrected.
Posted Nov 06, 2016 - 11:09 AEDT
Update
The host's root filesystem has errors preventing the server from correctly rebooting. We are working to correct the issue.
Posted Nov 06, 2016 - 09:56 AEDT
Investigating
Host node bnecompute07 has experienced a kernel fault requiring a restart to resolve. We are currently shutting down all VPS on that host.