Photo by Sid Ramirez on Unsplash
IT leads prepare for ‘business unusual’ in the event of grid failure
CIOs and IT leaders are under pressure to prepare their organisations for the possibility of extended load shedding or a complete grid failure, however few would be able to sustain business as usual through extended power outages.
This emerged during a high-level round table for CIOs and IT leaders hosted by the Institute of Information Technology Professionals South Africa (IITPSA) to discuss how to manage the risks posed by an unstable grid.
Moderated by Carolynn Chalmers (Pr.CIO), an IT Governance advisor and seasoned CIO/CTO, the event saw IITPSA accredited Professional CIOs (Pr.CIOs®) presenting their viewpoints, and attendees responding to polls. Most said increased power outages had been somewhat or very disruptive for their operations, with only 21% saying the disruption had been minimal.
Worst case scenarios
Only 34% of the CIOs and IT leaders attending said they had the necessary business continuity crisis plan, redundant connectivity and backup power reserves to manage a complete grid failure or outage of a week or longer. 38% were equipped to function through outages only up to 6 hours, 7% said they could function through outages of up to 24 hours, and 21% said they had no backup power or crisis plan.
While most have some backup power capabilities, higher stages of load shedding are already taking their toll and causing breakdowns of UPSs and generators. On the backup and alternative power sources they use to support their IT operations, 34% of participants said they use diesel generators, 24% depend on UPS and batteries/inverters, 10% have solar systems only, and 31% use a combination of these power sources. While 17% already have solar systems in place and 31% plan to install solar systems, 52% said they do not have the resources or space to generate the power they need using solar. Stage 8 load shedding or grid failure for weeks would be ‘catastrophic’ for many.
Ari Levien, a Pr.CIO, consultant and former CIO of a listed financial services company, said businesses were being forced to go back to first principles – what they do for their customers – in their planning for a possible grid collapse. “In some planning exercises, leaders are saying ‘in that scenario there’s unlikely to be a market for us, so we won’t even try to function’. Those who operate in socially important services such as emergency services and healthcare are saying ‘let’s talk to customers or competitors and work with them to make a plan’. People who are willing to say it’s not business as usual will come out with the ability to innovate,” he said.
Andrew Roberts, CIO at Clinix Health Group, said: “Realistically, organisations in South Africa aren’t equipped to function in Stage 8 or grid out scenarios. In the health and medical sector, where lives are at stake and load shedding affects areas like authorisations, admissions, pathology, and highly sensitive robotic equipment, one no longer has the benefit of throwing money at it to make the problem go away. Sustained Stage 8 or worse would mean severe disruption to networks, fibre backbones and all points of connectivity. We need to think smarter about how we operate in constrained environments – we proved we can do it during the Covid-19 pandemic. We’ve looked at working around loadshedding by changing hours and improvising, to do things better and smarter. We should also be thinking about how we collaborate across the sector to combat the basic challenges to create a more stable operating environment. It’s not about being competitive, it’s about survival.”
In addition to basic power concerns, CIOs acknowledged that extended outages would impact connectivity and could cause social unrest, putting staff at risk.
Louise van der Bank, CIO of AfriSam, noted: “At our administrative offices, many of the continuity requirements are catered for. However, the big challenges are at the operations, where a whole range of issues need to be managed – the fact that we can’t always produce what we planned, customers can’t take what they requested, and the impact on our staff coming to work. Moving shifts to align with available power could pose a risk to staff, who might have to travel at night. In addition, we have seen an increase in crime during load shedding periods.”
She said: “The Covid-19 pandemic was good preparation for this. We enabled remote and hybrid work where possible and started collaboration forums with stakeholders to work together more effectively.”
Adrian van Eeden, CIO at GIBS, added: “There are a lot of unanticipated risks. For example, our mitigation strategy was challenged when our generation capacity failed and we couldn’t find spare generators. There are risks around staff and students to consider – if they themselves don’t have connectivity, they can’t participate and there is a risk associated with them travelling to the campus through load shedding. We have to focus on people – the staff, colleagues and customers.”
Load shedding changes cloud strategies
While CIOS brace for the possibility of extended outages or grid failure, the current load shedding cycle is taking its toll – driving up costs and changing IT operations.
One area impacted by load shedding is cloud migration plans, with some IT leaders speeding up their move to the cloud, and others shelving planned moves to the cloud. 21% said they were moving more workloads to the cloud than they had planned to, and 7% were moving all of their systems to the cloud because of load shedding. 36% were already in the cloud and 25% had not changed their cloud strategy because of load shedding. 11% said they were not moving to the cloud.
Louise van der Bank said: “We decided to delay moving to cloud in some areas, because of service interruptions. It’s a huge issue for us if somewhere down the line, a portion of our business doesn’t have access to systems because of load shedding. When you are dependent on internet connectivity there are so many other role players and service providers also being impacted negatively by power outages.”
Adrian van Eeden said: “We’ve gone all cloud because this spreads the risk geographically. However, there’s a point where your response plan must acknowledge that if the hyperscalers go down, it’s a bigger problem than just us.”
One participant noted that having workloads in the cloud helped organisations avoid downtime during outages, but that where cloud was not an option due to compliance, the cost of backup power was significant.
Many CIOs and IT leaders have also had to learn new skills to build resilience into their operations and manage costs better, with 5% saying they had to develop new electrical knowledge to manage backup power, 21% having to improve their cloud skills to make the right cloud decisions and manage costs, 5% having to improve their procurement and negotiation skills to manage costs better, and 37% having to multi-task across several new areas.
Despite the negativity surrounding loadshedding, the result of the discussion was optimistic. CIOs and IT leads agreed that Covid-19 had prepared them to deal with business unusual, and that they believed they could work collaboratively with their peers and stakeholders to address new crises as they emerged.
