Tag Archive: IT


“What we have here is a failure to communicate.” -Captain

Rackspace played an important role in part of the tech stack I implemented for many of my IT customers for nearly ten years. We started implementing Rackspace’s Hosted Exchange solution back before Microsoft Office 365 hit its stride, and their service offering was truly first-rate at the time.  Unfortunately, that time is gone, punctuated by Friday’s dismal service breakdown and Rackspace’s complete failure to communicate with their customers in real-time as things unfolded.

If I am managing the Exchange server for a single company, never mind thousands of companies – which is likely what Rackspace is doing – and that server is not working, I have one responsibility that is just as important as getting the server back online. I must communicate with managers to give them information about what is going on to create reasonable expectations for when and how the issue will be resolved and facilitate their ability to mitigate risk.  In a normal situation, doing so makes perfect sense.

There is no good reason that wouldn’t be done.  The fact that this wasn’t done throughout the day on 12/2 can only mean a few things: absolute chaos, inadequate staffing, lack of information or perhaps some of each of those things.  Almost anyone managing IT and Exchange knows this.  I realize that Rackspace was likely determining the scope and severity of the issue, but in not communicating anything meaningful for the entire business day, Rackspace failed its customers.  They put the IT workers who support their solution in the unenviable position of only being able to communicate to their managers and customers that Rackspace wasn’t communicating with them.

To those who called Rackspace multiple times, listened to incessant jazzy hold music, and kept a vigilant eye on their status page most of the day, it no doubt became clear that this issue wasn’t something they could count on Rackspace to resolve in the short-term.  We will eventually know more about what happened, but the real story so far is Rackspace’s poor communication about what was going on in the moment.

For those still monitoring the status at status.apps.rackspace.com on 12/3, there was an update at 1:57am.  Any lingering hope of Rackspace resolving the issue sometime soon died with this update: “security incident … do not have an ETA for resolution … may take several days” So too would any other plans that IT workers utilizing Rackspace as part of their tech stack to provide Hosted Exchange had for their weekends.

The full message as provided from Rackspace at 1:57am on 12/3 follows.

What happened?

On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.

The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.

Has my account been affected?

We are working through the environment with our security teams and partners to determine the full scope and impact. We will keep customers updated as more information becomes available.

Has there been an impact to the Rackspace Email platform?

We have not experienced an impact to our Rackspace Email product line and platform. At this time, Hosted Exchange accounts are impacted, and not Rackspace Email.

When will I be able to access my Hosted Exchange account?

We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. We will be providing information on this page as it becomes available, with updates at least every 12 hours.

As a result, we are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

Is there an alternative solution?

At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.

To activate, please use the below link for instructions on how to set up your account and users.

https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel

Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.

IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.

I don’t know how to setup Microsoft 365. How can I get help?

Please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743).

Can I access my Hosted Exchange inbox from before the service was brought offline?

If you access your Hosted Exchange inbox via a local client application on your laptop or phone (like Outlook or Mail), your local device is likely configured to store your messages. However, while the Hosted Exchange environment is down, you will be unable to connect to the Hosted Exchange service to sync new mail or send mail using Hosted Exchange.

If you regularly access your inbox via Outlook Web Access (OWA), you will not have access to Hosted Exchange via OWA while the platform is offline.

As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

Will I receive mail in Hosted Exchange sent to me during the time the service has been shut down?

Possibly. We intend to update further as we get more information.

As a result, we are encouraging admins to configure and set up their user’s accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.

IT workers likely spent much of Saturday and Sunday migrating email to another provider, such as Microsoft, and some may still not be done today.  Depending on the readiness of contingency plans in place at various firms and/or the extent of local OST caching some firms may now be depending on Rackspace to recover their email records.  It is a little late to look at the SLA, but it is probably worth another glance now.

Though nearly all investment professionals utilize email journaling due to compliance requirements, I am not sure that everyone doing so has a complete backup of their current active email accounts.  They may have the ability to query their email records for compliance analysis using the journal but recovering all of the records that were stored at Rackspace as they were on 12/1 may be more complicated and drawn out.

Based on what customers currently know, it is possible that some users may not be able to recover some emails.  Remember that users are waiting for Rackspace to resolve a security issue.  Security is as much about protecting data from being lost as it is about it being compromised.  So there may be an issue with data loss rather than potential hacking that could have exposed passwords or data.  Rackspace hasn’t divulged the exact nature of the security incident.

One obvious takeaway from this issue is that you should be locally caching all Exchange data for your account in your local environment if you can.  To check your settings in Outlook, you can navigate to the screen shown below in Outlook by doing the following:

  1. Click on File, Account Settings, Account Settings (again).
  2. Select the email account you want to verify and click on the Change button.
  3. The default for downloading email for the past is typically “1 year.” If yours is set to “1 year”, you probably want to drag the control to the right to until it says “All” as shown below; however, I would defer to your IT people on this, because if they aren’t downloading all of your data, they could have a good reason.
  4. Once you have updated the setting, click the next button and then done button to commit the changes.

Migration, Initial Recovery and Complete Recovery

For the companies faced with this issue, restoring complete functionality of email and supporting applications will take time. If they haven’t already, they need to initiate migration by redirecting their DNS records so that email flows to another service provider and perform an initial recovery to get email running on computer/phones. They may also need to do a more complete recovery that includes all of the records that were stored in the users’ email and any specific email profile configuration settings that might have been lost.

Assuming the migration process goes smoothly, my estimation of the time required is roughly 2+ hours to update the DNS records necessary to point your email to a new service provider, wait for that info to propagate, and make sure all users are set up in the new service provider’s environment and everything is working properly.  Let’s be pessimistic and say this takes four hours.  Beyond that, you would still need to do the following items for each individual user:

  1. Have a backup of the PST on hand and ready to import, or create one from existing cached copies.
  2. Create new mail profiles to replace individual accounts within the current email profile. (My recommendation would be new profiles because I would want to maintain the old ones with their email records.)
  3. Depending on how things are configured, that might be a process that you would have to do once per user, or multiple times if they have notebooks and desktops with separate email profiles.
  4. Additionally, any mail accounts on Apple iOS and Android devices would need to be deleted and recreated.

Expecting to spend less than an hour per user on average to do this would be overly optimistic, but two is probably a reasonable guesstimate and some of the processing could likely be accomplished for various users simultaneously. But things like this almost never go smoothly.  These times could potentially be reduced through the use of third-party tools and automation, but let’s assume you don’t have access to those. A relatively small ten-person office that was using Rackspace could require 24 hours of IT work done over the weekend to bring them back online with most of their email on a new service.

What happened with Rackspace should also be a wake-up call to firms utilizing any cloud services and depending on them for real-time business continuity without necessarily having a full understanding what will happen in certain contingency scenarios.  Any service, whether it is cloud-based or on-premise, is only as good as the people managing it and your SLA.

Thankfully, the number of customers I service with a dependency on Rackspace has shrunk to almost none. Most have moved to Office 365.  Given this latest issue, it appears to me that Rackspace has been treading water with their Hosted Exchange service for the past year or so.  During that time using Multi-Factor Authentication (MFA) with email has become a critical business requirement and Rackspace hasn’t answered that call on their Hosted Exchange platform.  Their recommended solution for Hosted Exchange customers has been to buy Office 365 via Rackspace to get that MFA functionality from Microsoft.

To Rackspace’s credit, they did eventually start to give more useful information and constructive advice regarding the situation at 8:19 pm EST on Friday, but they went a whole day without providing anything of note. I don’t think I have ever seen a critical IT issue handled quite this way. If you are dealing with a Rackspace employee today, or with someone at your office who has been impacted by this event, try to be patient and kind. Doing anything else is pointless and counterproductive. These people are in an unpleasant and untenable situation today.


Kevin Shea Impact 2010

About the Author: Kevin Shea is the Founder and Principal Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment advisors nationwide.

For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.

With my long-standing history as a seasoned and impartial technology consultant catering to the wide-ranging needs of Advent users, it should come as no surprise that companies that have moved away from Advent call me to assist them if they have Advent specific needs after their agreements with Advent have lapsed.  In those specific cases, I suspect my independence from Advent is one of the most appealing features of my service, but many Advent users that have ongoing agreements with Advent also retain me to provide a level of service that Advent seems unwilling or unable to provide.

One of the things I get regular calls about is getting Axys running again.  These calls occur either when firms upgrade their servers or when firms that have moved on to competing Portfolio Management Systems dust off their old Axys files with hopes of tapping into Axys again.  My experience consulting to financial services firms using Advent Software for thirty-plus years facilitates my ability to resolve issues like these easily. 

Many of those calls I get start with the caller telling me, “We reinstalled Axys on the server and it isn’t working.”  And inevitably, this tells me more about the underlying issue than the caller ever could.  You certainly can reinstall Axys, but you probably don’t need to because Axys on the server is just a bunch of files that you access from another PC.  The most important thing to keep Axys working properly aside from the proper installation being done (at some point in the past) is making sure that users have all necessary rights to the shared folders.

This article is focused on explaining what the requirements are to empower you or your firm to resurrect Axys.  As usual, I’ll be providing a level of information in this piece that may be more than you need to solve any immediate problem with the hope that info is useful to you in the future.

Axys Versions

There are two fundamental versions of Axys: the multi-user version and single-user version.  To add a little confusion, the multi-user version is frequently referred to as the network version, but both fundamental versions are regularly installed on networks.  So, the network version is a bit of a misnomer.  Among these two fundamental versions, there is also the version of the software, which is at this point typically version 3.8, 3.8.5, 3.8.6 or 3.8.7.  In addition to these, there are also Monocurrency, Multicurrency and Variable Rate versions, to name a few.  Suffice to say, there are a lot of different versions.

Axys Licensing Model

The concurrent licensing model that Axys implements applies to both single-user and multi-user versions.  In both instances, the number of real Axys users typically exceeds the total licensed users, but having a multi-user version allows more than one user to use Axys simultaneously and adds certain multi-user features, such as user-specific settings and separate blotters, et cetera.

Understanding How Axys is Installed

Initially, the single-user version is simpler to install because the primary program (Axys) and supporting programs (Dataport, Data Exchange, Report Writer, et al.) hypothetically only need to be installed once.  That would be true if there literally was only one user using the software on one PC.  In actuality, the single-user version of Axys and supporting programs get installed multiple times in a network environment. They need to be installed once for every user, albeit to the same destination for each user (e.g., F:\Axys3).

During the Axys install process, certain required files are copied to the user’s PC and/or profile and Axys creates registry keys in HKEY_CURRENT_USER\SOFTWARE\Advent.  The most critical Axys registry keys are stored in HKEY_CURRENT_USER\SOFTWARE\Advent\Axys\3.  Although there are several important Axys files, the firmwide.inf is perhaps the most crucial file.  In a single-user installation, this text file, which can be found in the root folder of Axys (e.g., F:\Axys3), details certain settings in use and where all of the other Axys files can be found.

The multi-user version must also be installed multiple times for users, but the initial Axys install varies.  You install it once to the network/primary destination folder (e.g., F:\Axys3) and then install it again for the rest of the users (e.g., F:\Axys3\users\kevin where a firmwide.inf file will be created).  Similar to the single-user version, the supporting programs such as Dataport, Data Exchange and Report Writer would also need to be installed if the user needs those, or if you are trying to make sure all of the users have access to all of the supporting apps. The same registry keys are used for the multi-user install as the single-user version, but the multi-user (a.k.a. network) version adds an additional critical file: the netwide.inf file.

Netwide.inf versus Firmwide.inf

These two files are closely related.  The netwide.inf file should only be found in the root Axys folder of a network install, but firmwide.inf files exist in both single-user and multi-user environments.  The multi-user version is designed to use the settings in the netwide.inf as the system default and have any settings in the firmwide.inf supersede the settings in the netwide.inf.  As a rule, you should never see a firmwide.inf in the root Axys folder of a network install.  You should also almost never see a netwide.inf file in the root of a single-user Axys installation.


A Recurring Axys Installation Bug

With regard to installing Axys, there is a rather annoying issue that has been going on for several years.  It seems that the Axys install will not recognize certain network locations and/or mapped drives.  The fix requires the following registry settings:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System]

“EnableLUA”=dword:00000001

“EnableLinkedConnections”=dword:00000001

Once those settings have been applied, the Axys install program will be able to find the mapped drives.  It seems to me that this is an issue Advent should have addressed a long, long time ago.

Understanding Those Axys Shortcuts and Corresponding Registry Entries

The working folder of the Axys shortcut needs to point to the appropriate folder for the firmwide.inf file.  That means that an Axys shortcut for a single-user version of Axys should have a “Start in” folder like F:\Axys3, whereas the multi-user version would have “Start in” folder like F:\Axys3\users\kevin.  Assuming the same install folder was used, the target for these shortcuts would be the same: F:\Axys3\Axys32.exe.  Likewise, the registry entries associated with Axys should match these settings.  When I am looking at a system, I can usually determine if Axys has been installed properly by looking for consistency between the shortcuts and the following registry entries: ExePath, NetPath and UserPath.

In summary, your Axys install is dependent on a few things: the files themselves, access to the location where they are stored and proper mapping to the location of those files in the registry, firmwide.inf and netwide.inf if applicable.  Hopefully, you can get things back online on your own, but if you need assistance with your Advent installation, reach out to me and I’ll do my best to assist you.


Kevin Shea Impact 2010

About the Author: Kevin Shea is the Founder and Principal Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment advisors nationwide.

For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.

WhyMicrosoftThe moment I almost forget what a pain Windows 10 is, this message pops up on my PC.  Why did you have to ask me this question again, Microsoft?  Why must you remind me of my suffering?  All the details of what I have experienced are too much to cover in a single blog, so I will do my best to focus on the big issues.  As such, I won’t be whining about Windows 10 not consistently recognizing my finger, but that is a common theme here.

Nor will I spend the time rehashing various feature disruptions associated with forced updates to the degree that they deserve.  Most notably, Bitlocker comes to mind, but I cannot bring myself to go there in any significant detail. Suffice it to say that when I lost access to my encrypted Bitlocker drive due to an update, the documented fix required reinstalling an older version of Windows 10 to recover my data.  I chose to buy another hard drive since it was less complicated and time-consuming.

At one point in January of this year, I estimated that the combined dysfunction of Windows 10 and Office 365 had cost me at least two full days of productivity for my own system, never mind other people that I provide support to.  In that month alone, I personally spent over an hour a day on average dealing with issues that you would never see on a Windows 7 PC running a non-365 version of Office.

As an IT professional with thirty years of experience, I can honestly say that the Windows 10 operating system (OS) may be the most intrusive and unreliable OS ever created by Microsoft.  Computers and operating systems are intended to make our work lives more efficient and less challenging, not less efficient and more challenging.  On a regular basis, Windows 10 and its cohort, Office 365, thwart productivity through seemingly incessant and meaningless updates performed in the almighty name of compliance and security.

artificial-intelligence-155161_640

Even the most basic functionality of turning off your computer is challenged by the HAL-like behavior of this OS.  On my way out for a recent Thanksgiving road trip, I attempted to shut down my PC (four times).  Each time, my PC appeared to shutdown it came back on again.  It was clearly going to do this ad infinitum, which led to a few expletive laden Google searches like, “Windows 10 will not $&%#ing shut down!”

This is not the first time I have seen this particular issue in Windows 10 or similar quirky bugs like the black screen issue, so my patience was tested.  Eventually, I rediscovered and used the “hold the left-shift key and shutdown” method to wrestle my insubordinate PC into submission, then for good measure I actually unplugged it too.  Let’s see you restart now, Windows 10!  Thankfully, it didn’t.

Sure, this OS looks good on the surface, and in some ways it is better than its predecessor, but there are some major drawbacks.  For example, trying to use an app arbitrarily deemed as “not stable” or “incompatible” results in Windows 10 uninstalling that app without users’ permission.  Windows 10 won’t necessarily remove the app as soon as you install it, but when Windows applies updates again, it will remove the offending app and does not notify users.

Want to postpone an update or set the time updates are supposed to occur? … Go ahead.  There are settings for that, but whether you go through the exercise of configuring those settings or not, Windows 10 pretty much seems to do whatever it wants to do when it wants to.  I feel like I have lost control of my computers that run Windows 10.  Microsoft is in charge of them now and decides when and how I can use them.

If you have a critical online meeting, work that needs to be done right now, or a plane to catch, you can almost count on Windows 10 attempting to update or do some other thing that doesn’t need to be done at that exact time.  I don’t know how it does this, but it does.  It could just be that it is always doing an update.  In a nutshell, if you are familiar with the printer in the movie Office Space, Windows 10 is that printer.

Given my experiences, recommending this OS to anyone before they felt that they truly needed to move to it would be willfully irresponsible.  That said, I suspect there is a small contingent of users that Windows 10 helps stay out of trouble.  I know some of those people, but the masses should not have Windows 10 on their computers when there are other more reliable – as defined by computers that do what you want them to do when you want them to do it – alternatives.

Many of my financial services customers have likely moved to Windows 10 or plan to move to Windows 10 in the future.  For those businesses where compliance and security are paramount, staying the course on an aging OS like Windows 7 will become more difficult, given that Windows 10 is widely perceived as being more secure.

Understandably, for corporate use Windows 10 may just be a desktop environment that is used to gain access to a more secure and redundant cloud environment.  As such, the pain points I describe related to Windows 10 could be less of an issue for these users.  However, consigning users to Microsoft’s decisions about how they can use their PCs at any given time is scary.

Ultimately, the path Microsoft is on with Windows 10 is either headed toward total authoritarian rule over personal computer systems, or toward the eventual demise of Microsoft’s stranglehold on the PC OS market in favor of a more agreeable and obedient operating system.

lord-of-the-rings-the-one-ring_800

By way of disclaimer, I am using Windows 10 Professional, but know that Windows 10 Enterprise LTSB, which will soon be renamed to LTSC in 2019, follows the more traditional release policy and is not updated with the frequency of Microsoft’s other versions of Windows 10.  Based on my experience to date with Windows 10 Professional, the Enterprise LTSB product would probably be a much better user experience.  Also, related to Windows updates, my advanced options are set to Semi-Annual Channel (Targeted) with the option to defer feature updates by up to 180 days and security updates by up to 30 days.  I realize that I could gain a greater level of reliability and reduce the problems I experience by changing to the straight Semi-Annual Channel, which would delay feature updates by an additional 4 months.  My opinions are the result of using Windows 10 as both my primary desktop and notebook OS for the past two years.


About the Author: Kevin Shea is the Founder and Principal Kevin Shea Impact 2010Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment advisors nationwide.

For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.

sticky-notes-to-do-listAround this same time last year, many of us said our final goodbyes to Windows XP and Exchange 2003.  This year, Microsoft’s latest End-Of-Life (EOL) event – along with good sense – will force most of the firms that are still using Windows Server 2003  to replace it with a newer version of the Windows Server operating system (OS).  July 14th, 2015 marks the end of extended support for the 2003 product line – after that date, there won’t be any more security updates.

For those unfamiliar with the issue this raises, compliance regulation and standards related to private information and security dictate that firms must keep up-to-date with regular patches to the software and hardware that powers their businesses.  Your firm’s Written Information Security Program (WISP) should detail a policy of adherence to these standards, among many others, and in there somewhere you have almost certainly indicated that you are keeping your systems updated with respect to security.

Like Windows XP, Windows Server 2003 has been around long enough and really should be replaced, so there is not much point in delaying the switch.  Most firms have likely changed over to Windows Server 2008 or 2012, but those that haven’t made the change yet should be planning on upgrading their server(s) in Q2 of 2015.

 

rackAlternatives to Windows 2003?

Assuming your firm is committed to Microsoft Server products, you have two choices:

1. Windows Server 2008 r2 (2008)

2008 is a mature operating system, which is still in use at a large number of firms today. However, mainstream support for 2008 ended earlier this year (1/13/2015), and though extended support is available until 1/14/2020, it probably doesn’t make sense to move from 2003 to 2008 in 2015. Firms that have existing 2008 software licenses may not want to incur the additional expense of 2012 licenses, and those with significant compatibility concerns may opt to install Windows 2008 on new server hardware.

2. Windows Server 2012 r2 (2012)

2012 is the latest and greatest from Microsoft. It has a shiny new interface and a bevy of neat features like deduplication. My experience with 2012 has been overwhelmingly positive. Though worries about 2012 compatibility with legacy applications may delay widespread acceptance of this operating system, many firms will ultimately choose to make the switch to 2012.

What happens if we stay on Windows 2003?

Your server will still work, but you will not get any more security updates from Microsoft, and your firm will technically be out of compliance.

What else could happen?

Software companies and other parties your firm interfaces with will assume that you are making these updates.  Your firm’s failure to upgrade to a later version of Windows Server could cause problems that you and your staff may not be able to anticipate.

As an example of this, one of my clients that was slow to upgrade all of their Windows XP systems last year found that the latest version of Orion’s desktop software, which was automatically updated sometime in Q1 of 2014, was incompatible with Windows XP.  Unfortunately for the client, there wasn’t a way to reverse the update or use an older version.

At the time, I was surprised, especially because the customer wasn’t given any notice of the “feature enhancement.”  It didn’t make sense that a software company would launch an update incompatible with existing customer desktops that were still supported by Microsoft.  Thankfully, Orion addressed the issue quickly by providing the users affected with remote desktop (RDP) connections to Orion servers for an interim period.

About the Author: Kevin Shea is the Founder and Principal Kevin Shea Impact 2010Consultant of Quartare; Quartare provides a wide variety of technology solutions to investment management and financial services firms nationwide.

For details, please visit Quartare.com, contact Kevin Shea via phone at 617-720-3400 x202 or e-mail at kshea@quartare.com.