Last Thursday evening, January 23, a computer malfunction effectively shut down all Metro-North trains leaving and coming in to NYC. The problem caused all trains to stop moving and head to the nearest station to “park” until the problem was resolved. Eventually the action of a wrong power supply being pulled was ultimately resolved about 2 hours later, much to the chagrin of the riders who arrived late to their destinations.
When a Root Cause Analysis was conducted immediately following the episode, it was traced to a Technician pulling a power supply to the main Relay System as part of a replacement task of a larger Electrical Project. What the Technician failed to realize was that a wire on the main supply unit was already disconnected thus bringing the entire system down.
What was missing overall in this project was a lack of performing Risk Analysis and Scheduling functions. From the Risk side, the scenarios of something going wrong and having an adequate risk response was not executed. Had they executed this important part of any project they may have foreseen the possible ramifications of a system outage and prepared a feasible response solution. Had they properly scheduled the event, they would have done so to have the least impact to riders, possibly doing it on an early Sunday morning for example. By executing a risky task during one of the busiest time periods of ridership underscores that scheduling risk.
This is not an isolated instance, its been played out over and over in many business projects traversing the spectrum of industries. Often times when a project goes awry you can trace it to lack of adequate Risk Identification and Mitigation and/or Scheduling. As important as any executable task is for a project, these two items are key to success (or failure) and should always be part of any endeavor. The process of identifying risks, whether small or large, give you protection or insurance against anything that can go wrong and enable projects to continue. Scheduling allows you to execute it at the optimum time to avoid customer impact and have room to remediate issues if necessary.
Metro-North indicated that they will hire an independent counsel to look into the event and recommend changes; they may be better served to ensure that projects follow basic project management methodology, whatever shape or size of the execution.
Nit's Wit 
So true! Unfortunately, many disregard risk information they are given. I bet someone did say “Don’t we want to schedule this off hours?” It might have been from someone who did not have the clout to get his/her message pushed up the organization structure to a decision maker. It may have not have been scheduled on the weekend due to overtime considerations. Cost usually trumps risk. Somehow the calculation of there is 1% chance that 20,000 customers will be impacted if this goes wrong does not feel real until the angry calls start and hits the press.
Discussing risk can also be culturally unacceptable. People who say “Yes, we can.” even when they don’t end up meeting the original deadline or other performance metrics tend to be rewarded with leadership positions. Those who want to provide thoughtful analysis before committing/acting tend to get placed in the “worker bee” category.
Yours truly,
A former IT compliance professional turned designer.
LikeLike
Good thoughts Michel (and thanks for reading my blog!). I daresay that if the Project Manager knows his/her stuff then they would have ensured risks are identified and mitigated. Not knowing how MN handles their projects and thus their PMO lends an air of uncertainty on if this was just SOP or a missed opportunity.
LikeLike