Tag Archive: contingency planning


The Rackspace Blues:  A Cautionary Tale

Filed under: Best Practices, Business Continuity, Security — Leave a comment
December 5, 2022

“What we have here is a failure to communicate.” -Captain

Rackspace played an important role in part of the tech stack I implemented for many of my IT customers for nearly ten years. We started implementing Rackspace’s Hosted Exchange solution back before Microsoft Office 365 hit its stride, and their service offering was truly first-rate at the time.  Unfortunately, that time is gone, punctuated by Friday’s dismal service breakdown and Rackspace’s complete failure to communicate with their customers in real-time as things unfolded.

If I am managing the Exchange server for a single company, never mind thousands of companies – which is likely what Rackspace is doing – and that server is not working, I have one responsibility that is just as important as getting the server back online. I must communicate with managers to give them information about what is going on to create reasonable expectations for when and how the issue will be resolved and facilitate their ability to mitigate risk.  In a normal situation, doing so makes perfect sense.

There is no good reason that wouldn’t be done.  The fact that this wasn’t done throughout the day on 12/2 can only mean a few things: absolute chaos, inadequate staffing, lack of information or perhaps some of each of those things.  Almost anyone managing IT and Exchange knows this.  I realize that Rackspace was likely determining the scope and severity of the issue, but in not communicating anything meaningful for the entire business day, Rackspace failed its customers.  They put the IT workers who support their solution in the unenviable position of only being able to communicate to their managers and customers that Rackspace wasn’t communicating with them.

To those who called Rackspace multiple times, listened to incessant jazzy hold music, and kept a vigilant eye on their status page most of the day, it no doubt became clear that this issue wasn’t something they could count on Rackspace to resolve in the short-term.  We will eventually know more about what happened, but the real story so far is Rackspace’s poor communication about what was going on in the moment.

For those still monitoring the status at status.apps.rackspace.com on 12/3, there was an update at 1:57am.  Any lingering hope of Rackspace resolving the issue sometime soon died with this update: “security incident … do not have an ETA for resolution … may take several days” So too would any other plans that IT workers utilizing Rackspace as part of their tech stack to provide Hosted Exchange had for their weekends.

The full message as provided from Rackspace at 1:57am on 12/3 follows.

What happened?

On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.

The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.

Has my account been affected?

We are working through the environment with our security teams and partners to determine the full scope and impact. We will keep customers updated as more information becomes available.

Has there been an impact to the Rackspace Email platform?

We have not experienced an impact to our Rackspace Email product line and platform. At this time, Hosted Exchange accounts are impacted, and not Rackspace Email.

When will I be able to access my Hosted Exchange account?

We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. We will be providing information on this page as it becomes available, with updates at least every 12 hours.

As a result, we are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

Is there an alternative solution?

At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.

To activate, please use the below link for instructions on how to set up your account and users.

https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel

Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.

IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.

I don’t know how to setup Microsoft 365. How can I get help?

Please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743).

Can I access my Hosted Exchange inbox from before the service was brought offline?

If you access your Hosted Exchange inbox via a local client application on your laptop or phone (like Outlook or Mail), your local device is likely configured to store your messages. However, while the Hosted Exchange environment is down, you will be unable to connect to the Hosted Exchange service to sync new mail or send mail using Hosted Exchange.

If you regularly access your inbox via Outlook Web Access (OWA), you will not have access to Hosted Exchange via OWA while the platform is offline.

As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

Will I receive mail in Hosted Exchange sent to me during the time the service has been shut down?

Possibly. We intend to update further as we get more information.

As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

IT workers likely spent much of Saturday and Sunday migrating email to another provider, such as Microsoft, and some may still not be done today.  Depending on the readiness of contingency plans in place at various firms and/or the extent of local OST caching some firms may now be depending on Rackspace to recover their email records.  It is a little late to look at the SLA, but it is probably worth another glance now.

Though nearly all investment professionals utilize email journaling due to compliance requirements, I am not sure that everyone doing so has a complete backup of their current active email accounts.  They may have the ability to query their email records for compliance analysis using the journal but recovering all of the records that were stored at Rackspace as they were on 12/1 may be more complicated and drawn out.

Based on what customers currently know, it is possible that some users may not be able to recover some emails.  Remember that users are waiting for Rackspace to resolve a security issue.  Security is as much about protecting data from being lost as it is about it being compromised.  So there may be an issue with data loss rather than potential hacking that could have exposed passwords or data.  Rackspace hasn’t divulged the exact nature of the security incident.

One obvious takeaway from this issue is that you should be locally caching all Exchange data for your account in your local environment if you can.  To check your settings in Outlook, you can navigate to the screen shown below in Outlook by doing the following:

  1. Click on File, Account Settings, Account Settings (again).
  2. Select the email account you want to verify and click on the Change button.
  3. The default for downloading email for the past is typically “1 year.” If yours is set to “1 year”, you probably want to drag the control to the right to until it says “All” as shown below; however, I would defer to your IT people on this, because if they aren’t downloading all of your data, they could have a good reason.
  4. Once you have updated the setting, click the next button and then done button to commit the changes.

Migration, Initial Recovery and Complete Recovery

For the companies faced with this issue, restoring complete functionality of email and supporting applications will take time. If they haven’t already, they need to initiate migration by redirecting their DNS records so that email flows to another service provider and perform an initial recovery to get email running on computer/phones. They may also need to do a more complete recovery that includes all of the records that were stored in the users’ email and any specific email profile configuration settings that might have been lost.

Assuming the migration process goes smoothly, my estimation of the time required is roughly 2+ hours to update the DNS records necessary to point your email to a new service provider, wait for that info to propagate, and make sure all users are set up in the new service provider’s environment and everything is working properly.  Let’s be pessimistic and say this takes four hours.  Beyond that, you would still need to do the following items for each individual user:

  1. Have a backup of the PST on hand and ready to import, or create one from existing cached copies.
  2. Create new mail profiles to replace individual accounts within the current email profile. (My recommendation would be new profiles because I would want to maintain the old ones with their email records.)
  3. Depending on how things are configured, that might be a process that you would have to do once per user, or multiple times if they have notebooks and desktops with separate email profiles.
  4. Additionally, any mail accounts on Apple iOS and Android devices would need to be deleted and recreated.

Expecting to spend less than an hour per user on average to do this would be overly optimistic, but two is probably a reasonable guesstimate and some of the processing could likely be accomplished for various users simultaneously. But things like this almost never go smoothly.  These times could potentially be reduced through the use of third-party tools and automation, but let’s assume you don’t have access to those. A relatively small ten-person office that was using Rackspace could require 24 hours of IT work done over the weekend to bring them back online with most of their email on a new service.

What happened with Rackspace should also be a wake-up call to firms utilizing any cloud services and depending on them for real-time business continuity without necessarily having a full understanding what will happen in certain contingency scenarios.  Any service, whether it is cloud-based or on-premise, is only as good as the people managing it and your SLA.

Thankfully, the number of customers I service with a dependency on Rackspace has shrunk to almost none. Most have moved to Office 365.  Given this latest issue, it appears to me that Rackspace has been treading water with their Hosted Exchange service for the past year or so.  During that time using Multi-Factor Authentication (MFA) with email has become a critical business requirement and Rackspace hasn’t answered that call on their Hosted Exchange platform.  Their recommended solution for Hosted Exchange customers has been to buy Office 365 via Rackspace to get that MFA functionality from Microsoft.

To Rackspace’s credit, they did eventually start to give more useful information and constructive advice regarding the situation at 8:19 pm EST on Friday, but they went a whole day without providing anything of note. I don’t think I have ever seen a critical IT issue handled quite this way. If you are dealing with a Rackspace employee today, or with someone at your office who has been impacted by this event, try to be patient and kind. Doing anything else is pointless and counterproductive. These people are in an unpleasant and untenable situation today.

Kevin Shea Impact 2010

About the Author: Kevin Shea is the Founder and Principal Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment advisors nationwide.

For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.

Tags: , , ,
Comment

Planning for a Heart Attack

Filed under: Best Practices, Business Continuity, Portfolio Management Systems, Training — 1 Comment
January 31, 2013

Dad_Vero_ClippedQuarter end reporting for Q4 2011 was a grueling task – one that still wasn’t completed for some firms in February of 2012. I am in touch with the operations staff of representative investment firms of varied disciplines and sizes. That quarter I heard it from just about everybody. Firms were transferring accounts to different custodians, embracing new trading platforms, contemplating portfolio management system changes and enhancements to client-facing reporting as well as more robust outsourcing options … but the show must go on.

The operations staff rarely gets a break. They must deal with new initiatives as they come and still manage to get their regular work done. The act of balancing the two responsibilities can be difficult and exhausting for those faced with these demands. To quote one of my clients, “Every time you turn around its quarter end.”

As a consultant to many of these firms, I have a unique view of the daily grind of typical investment professionals. In the past, they could take reasonable breaks for lunch, etc., but more and more of the people I work with are going without the niceties of lunches out and a little down-time during the day. An illness in the family can be particularly troublesome for those working in investment operations to deal with. Illnesses are seldom convenient or negotiable.

In addition to the normal challenges that I face in any given quarter, that quarter I had to deal with one that I had no control over – the death of a loved one. My father, who, at age 73, seemed to be in outstanding health had a sudden and unexpected heart attack on the evening of January 23rd, 2012. There is never a good time for something like that to happen, but the 23rd was better than the 8th would have been for me. I was on the phone with him when it happened, and heard him utter his last words, “I feel so dizzy.” Then he collapsed.

It was so quick. I could hardly comprehend it. Over the course of the next few hours my brother and sister traveled home to New York to be with my mother as quickly as possible, while I tried to think about what I needed to do to wrap up any loose ends related to my clients’ quarter end reporting, and get home as soon as possible. After a day and half of chipping away at various tasks and setting things in motion, I needed to go home and grieve our loss with my family.

Dennis_shaking_hands_DR003My father had a successful career as a financial officer in the retail and financial service industries acting as consultant to many banks and fortune 500 companies. He kept his financial dealings fairly close to the vest, and though he had shared some of what he had done with me, I was not familiar with everything. His record-keeping was meticulous and nearly perfect in all respects but one – planning for his unexpected death. He had a joint will with my mother, but we couldn’t find it. He had assets and regular income, but getting my mother access to all of their assets would take time.

Over the next week, I went over his records, which included 22 years of tax returns and similar detailed records of my parent’s finances to lend what help I could in assessing the situation for my mother. I am no stranger to investment reports, so I winced when I saw my father’s Morgan Stanley Smith Barney statements. The production of statements as meaningless as these should be a criminal offense. I first saw statements like these back in 1987. In twenty-five years very little has changed. The statements are as dull and drab as possible. The only color afforded is the dark blue line that runs along the top. These statements are difficult for people who know what they are looking for to read – never mind those less familiar. At best, these statements are an inventory of holdings.

After looking through more of the investment statements, I eventually found summary statements that get sent out about a month after each quarter end. These statements were better, but not as good as most of what we create for our clients. Given their ability to produce better statements, firms like Morgan Stanley Smith Barney should be held to a higher reporting standard.

Over the course of nearly two weeks, my brother and I stoically exchanged quips like “Good will hunting” and “Where there is a will there’s a way” in humor that my father would have appreciated. Eventually, we found the will. It was perhaps the only thing improperly filed in his office. Throughout the ordeal, I couldn’t help thinking about how my father could have made things much easier for us by leaving us a list of the top ten things to do if he died. It might have taken him twenty minutes to put together if he had ever thought about it. I half expected to find such a list, but it was nowhere to be found – apparently, my Dad wasn’t planning to die. Here is what the list might have said:

Sorry to leave you all so suddenly, but here is what you need to do:

1. Call my attorney _______________ at _______________ , and have him execute my will. For some reason, I have the original copy of my will in the file marked _________________ at our home in ________________.

2. I have three whole life insurance policies that should provide a total non-taxable death benefit of approximately ___________________ .

They are:
a. _______________________________
b. _______________________________
c. _______________________________

Call my good friend _______________________ in the insurance business at ______________ and have him help you get the forms and file them. The proceeds from my life insurance should help with the transition period.

3. In the event of my passing, Mom should have access to the following regular income sources totaling about ________ per month:
a. ________ Pension.
b. My social security not hers.
c. The annuity.

4. All of the bank accounts are jointly held and your mother is listed as the primary beneficiary.

5. My retirement account will need to be transferred to her and she is listed as the primary beneficiary.

6. There is no money hidden anywhere. It was already raining.

7. Put the funeral on the Amex and the meal afterwards on the Underhill tab.

8. Yes. I want the cheapest casket.

9. Psalm 23.

10. Swing easy.

Love,
Dad

Using my personal experience with my father as an example, we can hopefully learn something.  First and foremost, we should make sure to acknowledge our own mortality and take the steps necessary to make our passing easier on those we would leave behind.  And, of course, since this blog is about investment operations and technology, ensure that your firm has the contingency planning, documentation, staff redundancy and training necessary to survive the loss of key personnel, whether that loss is through a sudden career change, a long-term illness, or an unexpected tragedy.

The impetus for some of the best client relationships I have ever had has been the vacuum created by the loss of personnel.  I have helped firms that experienced 100% turnover in their investment operations department rebuild, assisted those trying to make sense of cryptic documentation left behind by co-workers who left abruptly, and managed to get things running again when the person who “did everything” was severely injured in an automobile accident.

Understanding your firm’s dependancy on key personnel is very important.  Even when systems are documented, that documentation’s usefulness may be questionable.  Documentation that hasn’t been reviewed and tested might be meaningful to those who created it, but not to those trying to use it to complete a process in the author’s absence.

Some of my clients have actually drafted letters to be delivered to their investors in the event of their deaths. Those letters are in their contingency plan – what’s in yours?

–

About the Author: Kevin Shea is President of InfoSystems Integrated, Inc. (ISI); ISI provides a wide variety of outsourced IT solutions to investment advisors nationwide.

For details, please visit isitc.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@isitc.com.

Tags: ,
Comment