“What we have here is a failure to communicate.” -Captain
Rackspace played an important role in part of the tech stack I implemented for many of my IT customers for nearly ten years. We started implementing Rackspace’s Hosted Exchange solution back before Microsoft Office 365 hit its stride, and their service offering was truly first-rate at the time. Unfortunately, that time is gone, punctuated by Friday’s dismal service breakdown and Rackspace’s complete failure to communicate with their customers in real-time as things unfolded.
If I am managing the Exchange server for a single company, never mind thousands of companies – which is likely what Rackspace is doing – and that server is not working, I have one responsibility that is just as important as getting the server back online. I must communicate with managers to give them information about what is going on to create reasonable expectations for when and how the issue will be resolved and facilitate their ability to mitigate risk. In a normal situation, doing so makes perfect sense.
There is no good reason that wouldn’t be done. The fact that this wasn’t done throughout the day on 12/2 can only mean a few things: absolute chaos, inadequate staffing, lack of information or perhaps some of each of those things. Almost anyone managing IT and Exchange knows this. I realize that Rackspace was likely determining the scope and severity of the issue, but in not communicating anything meaningful for the entire business day, Rackspace failed its customers. They put the IT workers who support their solution in the unenviable position of only being able to communicate to their managers and customers that Rackspace wasn’t communicating with them.
To those who called Rackspace multiple times, listened to incessant jazzy hold music, and kept a vigilant eye on their status page most of the day, it no doubt became clear that this issue wasn’t something they could count on Rackspace to resolve in the short-term. We will eventually know more about what happened, but the real story so far is Rackspace’s poor communication about what was going on in the moment.
For those still monitoring the status at status.apps.rackspace.com on 12/3, there was an update at 1:57am. Any lingering hope of Rackspace resolving the issue sometime soon died with this update: “security incident … do not have an ETA for resolution … may take several days” So too would any other plans that IT workers utilizing Rackspace as part of their tech stack to provide Hosted Exchange had for their weekends.
The full message as provided from Rackspace at 1:57am on 12/3 follows.
What happened?
On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.
The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.
Has my account been affected?
We are working through the environment with our security teams and partners to determine the full scope and impact. We will keep customers updated as more information becomes available.
Has there been an impact to the Rackspace Email platform?
We have not experienced an impact to our Rackspace Email product line and platform. At this time, Hosted Exchange accounts are impacted, and not Rackspace Email.
When will I be able to access my Hosted Exchange account?
We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. We will be providing information on this page as it becomes available, with updates at least every 12 hours.
As a result, we are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
Is there an alternative solution?
At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.
To activate, please use the below link for instructions on how to set up your account and users.
https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel
Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.
IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.
I don’t know how to setup Microsoft 365. How can I get help?
Please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743).
Can I access my Hosted Exchange inbox from before the service was brought offline?
If you access your Hosted Exchange inbox via a local client application on your laptop or phone (like Outlook or Mail), your local device is likely configured to store your messages. However, while the Hosted Exchange environment is down, you will be unable to connect to the Hosted Exchange service to sync new mail or send mail using Hosted Exchange.
If you regularly access your inbox via Outlook Web Access (OWA), you will not have access to Hosted Exchange via OWA while the platform is offline.
As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
Will I receive mail in Hosted Exchange sent to me during the time the service has been shut down?
Possibly. We intend to update further as we get more information.
As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
IT workers likely spent much of Saturday and Sunday migrating email to another provider, such as Microsoft, and some may still not be done today. Depending on the readiness of contingency plans in place at various firms and/or the extent of local OST caching some firms may now be depending on Rackspace to recover their email records. It is a little late to look at the SLA, but it is probably worth another glance now.
Though nearly all investment professionals utilize email journaling due to compliance requirements, I am not sure that everyone doing so has a complete backup of their current active email accounts. They may have the ability to query their email records for compliance analysis using the journal but recovering all of the records that were stored at Rackspace as they were on 12/1 may be more complicated and drawn out.
Based on what customers currently know, it is possible that some users may not be able to recover some emails. Remember that users are waiting for Rackspace to resolve a security issue. Security is as much about protecting data from being lost as it is about it being compromised. So there may be an issue with data loss rather than potential hacking that could have exposed passwords or data. Rackspace hasn’t divulged the exact nature of the security incident.
One obvious takeaway from this issue is that you should be locally caching all Exchange data for your account in your local environment if you can. To check your settings in Outlook, you can navigate to the screen shown below in Outlook by doing the following:
- Click on File, Account Settings, Account Settings (again).
- Select the email account you want to verify and click on the Change button.
- The default for downloading email for the past is typically “1 year.” If yours is set to “1 year”, you probably want to drag the control to the right to until it says “All” as shown below; however, I would defer to your IT people on this, because if they aren’t downloading all of your data, they could have a good reason.
- Once you have updated the setting, click the next button and then done button to commit the changes.
Migration, Initial Recovery and Complete Recovery
For the companies faced with this issue, restoring complete functionality of email and supporting applications will take time. If they haven’t already, they need to initiate migration by redirecting their DNS records so that email flows to another service provider and perform an initial recovery to get email running on computer/phones. They may also need to do a more complete recovery that includes all of the records that were stored in the users’ email and any specific email profile configuration settings that might have been lost.
Assuming the migration process goes smoothly, my estimation of the time required is roughly 2+ hours to update the DNS records necessary to point your email to a new service provider, wait for that info to propagate, and make sure all users are set up in the new service provider’s environment and everything is working properly. Let’s be pessimistic and say this takes four hours. Beyond that, you would still need to do the following items for each individual user:
- Have a backup of the PST on hand and ready to import, or create one from existing cached copies.
- Create new mail profiles to replace individual accounts within the current email profile. (My recommendation would be new profiles because I would want to maintain the old ones with their email records.)
- Depending on how things are configured, that might be a process that you would have to do once per user, or multiple times if they have notebooks and desktops with separate email profiles.
- Additionally, any mail accounts on Apple iOS and Android devices would need to be deleted and recreated.
Expecting to spend less than an hour per user on average to do this would be overly optimistic, but two is probably a reasonable guesstimate and some of the processing could likely be accomplished for various users simultaneously. But things like this almost never go smoothly. These times could potentially be reduced through the use of third-party tools and automation, but let’s assume you don’t have access to those. A relatively small ten-person office that was using Rackspace could require 24 hours of IT work done over the weekend to bring them back online with most of their email on a new service.
What happened with Rackspace should also be a wake-up call to firms utilizing any cloud services and depending on them for real-time business continuity without necessarily having a full understanding what will happen in certain contingency scenarios. Any service, whether it is cloud-based or on-premise, is only as good as the people managing it and your SLA.
Thankfully, the number of customers I service with a dependency on Rackspace has shrunk to almost none. Most have moved to Office 365. Given this latest issue, it appears to me that Rackspace has been treading water with their Hosted Exchange service for the past year or so. During that time using Multi-Factor Authentication (MFA) with email has become a critical business requirement and Rackspace hasn’t answered that call on their Hosted Exchange platform. Their recommended solution for Hosted Exchange customers has been to buy Office 365 via Rackspace to get that MFA functionality from Microsoft.
To Rackspace’s credit, they did eventually start to give more useful information and constructive advice regarding the situation at 8:19 pm EST on Friday, but they went a whole day without providing anything of note. I don’t think I have ever seen a critical IT issue handled quite this way. If you are dealing with a Rackspace employee today, or with someone at your office who has been impacted by this event, try to be patient and kind. Doing anything else is pointless and counterproductive. These people are in an unpleasant and untenable situation today.
About the Author: Kevin Shea is the Founder and Principal Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment advisors nationwide.
For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.
The moment I almost forget what a pain Windows 10 is, this message pops up on my PC. Why did you have to ask me this question again, Microsoft? Why must you remind me of my suffering? All the details of what I have experienced are too much to cover in a single blog, so I will do my best to focus on the big issues. As such, I won’t be whining about Windows 10 not consistently recognizing my finger, but that is a common theme here.
