Zebis’ current top priority is assisting those impacted by the Rackspace outage. This page recaps the issue, answers command questions, and provides the status.
December 21, 2022
Status:
We are continuing to address refinements, connect smartphones and devices, and rebuild automatic mail handling rules that were in place with the old provider.
The new provider has implemented a characteristic which may result in legitimate e-mails being marked as junk. We are aware of this characteristic and working to address this.
Rackspace has stated they are planning to restore access to prior data. We are closely monitoring this and, if data does become available, we will migrate this to your new accounts.
Within the upcoming weeks, we will be making a new backup option available to safeguard your e-mail, calendar, contacts, and tasks should a similar attack compromise the new provider, Microsoft 365.
At this phase, if you do have an urgent request, please reach out to us and we will aim to address as quickly as possible.
December 15, 2022
Status:
We are continuing to provide assistance to access calendar, contacts, tasks, and e-mail history that was hosted by Rackspace. In many instances, there may be cached data residing on a computer from the last time that computer able to successfully connect.
The new e-mail backup system is estimated to be ready within 2-3 weeks to reduce the impact of a future cloud attack.
Rackspace status is continuing to be closely monitored regarding news of data availability. When / If Rackspace restores access to past calendar, contacts, tasks, and e-mails, we will start migration efforts to help those who’s computers did not contain cached information.
December 9, 2022
Status:
With most Zebis clients now able to receive and send e-mail, Zebis is progressing with next phases including assistance with smartphone and tablet connections, applying help with settings, workflow characteristics, and other refinements to help return to work as usual.
When will access to my my e-mail history, calendar, contacts, and tasks be restored?
E-mail history, contacts, calendar, and tasks are held on the Rackspace servers. Rackspace has not yet provided an estimated time by which access to your information will be restored.
Rackspace has status states they have restored access to tens of thousands of accounts. Do these accounts have access to their e-mail history, calendar, contacts and tasks?
No, Rackspace is using the term “restore” to refer to assisting subscribers establish e-mail receiving and sending ability with a service provider other than Rackspace resold through Rackspace. Rackspace has not stated they have restored any of their customers’ access to their mailboxes. This was confirmed with a phone conversation with a Rackspace representative.
Without Rackspace restoring access to mailboxes, will there be any way I can access my e-mail history, contacts, calendar, and tasks?
There is reason to have hope. In many instances, applications on your computer will cache content from the last time it was able to successfully connect. In that instance, it may be possible to extract that cache and apply it to your new account. This is referred to as an “Offline Migration”. We are in the process of addressing this for each Zebis client impacted by the Rackspace outage.
An Offline Migration has been applied to my account. Why do my contacts, calendar, tasks, and e-mail appear incomplete?
Offline migrations use information cached to your computer from the last time a successful connection was made. This information may be incomplete. Offline migrations also, in many instances, require a lengthly synchronization from your computer. It may be this process has not yet completed.
There may be other factors causing this as well so please let us know and we will investigate.
What can we learn from this incident and what protection is available?
Cloud providers may offer incredible workflow assistance and convenience. This incident stresses two key points.
- In spite of cloud providers stating they implement strong disaster recovery (DR) plans, maintain extensive backups, follow best practices and all due diligence regarding security, and follow best practices and all due diligence regarding privacy, it is near impossible for a 3rd party to confirm these claims prior to an incident revealing a disparity between these claims and the implemented systems.
- Even if a cloud provider follows all best practices, a motivated attacker (which may include an inside staff member) may be able to discover a method of penetrating the cloud provider placing your information and workflow in jeopardy.
Protecting yourself from the effects the above two points along with other characteristics that may effect access to your critical information and resources is accomplished by keeping your cloud information backed up to an alternate resources. In the upcoming months, Zebis will be developing such a system which will be able to backup your e-mail, calendar, tasks, and contacts to a storage facility not associated with the your e-mail cloud provider. This system will protect you from service outages or even situations such as accidental deletions.
We will be aiming to reach out to Zebis clients when this becomes available. If this backup system is a high priority for you, please let us know.
December 7, 2022
What is the current status?
We have successfully created new e-mail accounts for most of our clients at an alternate e-mail hosting provider restoring the ability to receive and send e-mail. If you are currently not able to receive or send e-mail, please let us know and we will help you as soon as possible.
What Occurred?
The service provider Rackspace suffered a catastrophic failure on Friday, December 2, before 3:00AM EST effecting customers internationally.
This failure disabled access to Rackspace’s Microsoft Exchange hosting which provides business class groupware (E-mail, Calendar, Contacts, Tasks) hosting service. Users were unable to access e-mail, calendar, contacts, and tasks.
E-mails sent to Rackspace hosted Exchange e-mail addresses were bounced to the original sender.
By Saturday, December 3, Rackspace had not provided an estimated time of resolution so Zebis activated a protocol to establish e-mail receiving and sending capability by establishing new e-mail accounts at an alternate service provider for all of our impacted clients. Those of you following the Rackspace status updates may have noticed Rackspace is also suggesting to establish new accounts at an alternate service provider: Microsoft 365. Their suggestion is to obtain the alternate service through Rackspace, however, due to this outage, we suggest obtaining alternate service without the involvement of Rackspace and we have now established accounts for most of our clients at the new provider.
FAQ
What happened to e-mails sent to me between the start of the outage and the establishment of my new e-mail account?
These e-mails were bounced (rejected) and returned to sender.
The Rackspace status page states they have restored service for thousands of subscribers. Is this true?
No. Rackspcae has not “restored” service. Instead, Rackspace is suggesting customers establish service at a provider external to Rackspace: Microsoft 365. Rackspace is offering assistance to establish these new accounts, however, they are not providing access to prior calendar, contacts, tasks, and e-mail. The new accounts are new and empty of previous data.
Will Rackspace be able to to provide access to contacts, calendars, tasks, and e-mails from before incident?
Rackspace has not provided details as the status of their backups or the availability of custemer data.
Is Zebis Satisfied with Rackspace’s Response?
No for the following reasons:
Unmanaged expectations with minimal updates: Rackspace has not yet provided details and has not met self stated expectations of updates every 12 hours. Rackspace has not provided details regarding the status of customer data.
Rackspace status updates state they offer an alternate solution to restore e-mail service by establishing an Office 365 account. Is this worth considering?
Zebis does not suggest their offer for multiple reasons: 1.) Their offer does not “restore” service, but creates a new empty account and 2.) their offer establishes service at a provider external to Rackspace, but resold through Rackspace with a margin to them after the grace period expires.
Zebis does suggest establishing new service at an external provider without Rackspace’s involvement.
Is there a benefit to continuing to check status updates posted by Rackspace?
Though we at Zebis did suggest this early in the outage, at this phase, it is no longer useful as most accounts have been established at the new provider. Zebis is closely watching the status for updates regarding customer data.
When Rackspace restores service, would Zebis suggest returning to them?
Due the extend of this outage, their limited status updates, and even Rackspace’s suggestion to establish new accounts at an alternate provider, we suggest permanently moving to an alternate host without Rackspace’s involvement.
To what e-mail host has Zebis established my new account(s)?
We have established all new accounts directly with Microsoft. New accounts are business class Microsoft 365 tightly integrated with Zebis management systems providing enhanced reliability, troubleshooting, strong support, etc.