Improving employee productivity through rapid recovery after a cyber incident

Outcome

The Challenge

Resume productivity, improve security and build for the future.

In early 2019, G’s Fresh was the victim of an aggressive malware attack that encrypted 50 per cent of their applications and infected over 80 per cent of their desktops. To try and secure their business, the team at G’s decided to rebuild and replace the entire IT infrastructure, including all desktops and networks.  

As an organisation that relies heavily on extensive tracking of produce, large numbers of seasonal workers and international shipment data, being without their systems was hugely detrimental to their business.  

The business was really suffering when we were brought in two months after the initial attack to support on the infrastructure rebuild. 

Impact on their people.  

With the usual reporting systems not working, everything was being done manually in ExCel. The broadband capacity was so long that data updates were taking hours rather than minutes. 

The HR and payroll teams were working around the clock to manually process the payments of over 2,000 seasonal workers to ensure everyone got paid on time. The IT team were trying to keep everything running while also fixing and responding to endless service desk tickets from across the business. The IT Director resigned amid the crisis and several other employees also decided to leave. Workforce morale was at an all-time low. 

Impact on their customers and processes.

G’s scanning system, which picked and dispatched produce and then invoiced customers accordingly, was regularly failing due to the capacity of their disaster recovery infrastructure. This meant that produce trucks often left without scanning the goods, there were incorrect inventory sheets and continuous manual reconciliations were needed with customers to make sure invoicing was correct. 

The underlying operating systems were old and not secure (they were still using Windows Server 2007 / SQL 2005). New PCs had been issued, but they had no way of ensuring regular patches and their entire estate was running on old and unsupported servers that hadn’t been successfully backed up since the cyber-attack. The risk of another security breach was high. 

Impact on their bottom line  

The cyber attack proved to be hugely costly for G’s. The IT team needed more support, so a range of consultants were brought in to help. But without a clear strategy, overall leadership and accountability or an architecture plan, money was being lost and progress wasn’t being made.  

There was also the cost in terms of people and resource involved in the slow, manual processes required to keep everything running. 

G’s needed a new approach to fixing the problem, before it put their entire business at risk.  

The Solution

TTG provided leadership, on-the-ground support and fast-tracked rebuilding G’s IT

Providing leadership and on-the-ground support 

From our meetings and initial assessment, it ws ckear that strategic IT leadership was required to coordinate efforts and rebuild the team, alongside immediate technical support to implement projects at pace. 

We proposed an interim TTG CIO, leading more than 20 interim consultants and 15 permanent IT staff. As it was clear the current approach wasn’t working we brought in senior experienced network, infrastructure, data recovery, application and cyber experts. 

Building a robust and secure network.

Our first priority was putting a secure network in place to enable greater productivity across the business. Farming and production sites that weren’t connected to the multi-protocol label swtiching (MLPS) were using 4G dongles to connect to the corporate network. This was a slow and unreliable solution and regular outages led the business to purchase CISCO Firepower firewalls. 

Within two weeks of joining the project, despite facing an undocumented network, we created an emergency design and led the implementation of these firewalls to the main sites. 

Our next priority was the internet circuit, which was costing £10,000 per year and was far too small for the organisation. We knew we could drastically improve on the quotes of £100,000 for 1GB that were coming in and we increased the internet capability ten-fold with a dual-resilience circuit at a cost of only £12,000 per year.

Production sites could now scan produce correctly, stop using dongles (resulting in a saving of £50,000 per year) and install basic telephony which greatly improved productivity and connectivity across the sites.  

G’s original intention was to use the new CISCO firewalls for their smaller farms that harvest crops, but their remote locations and poor bandwidth posed a challenge. After fully assessing the market, we implemented a 24/7, fully-managed SD-WAN solution which gave great connectivity to the smaller sites and meant an existing MPLS could be decommissioned, saving G’s almost £200,000 over three years. 

We proposed and created the business case for a strategic Wi-Fi solution, to enable secure, seamless access and roaming in the warehouses. We helped with the design and coverage and implemented the core Wi-Fi access controllers. The results exceeded expectations with a significant reduction in lost productivity calculated at over £8,000 per month. We then deployed this solution to the remainder of G’s sites, including their HQ.  

Delivering on their desktops 

In a rush to deploy and issue secure desktops, the active directory was compromised and syncing to Azure AD wasn’t working correctly. We created new AD policies that restricted access, so that changes wouldn’t compromise the directory in the future. 

We recommended using Microsoft’s Enterprise Mobility and Security Suite to integrate all infrastructure and desktops into Azure. This significantly strengthened security as it meant that if a successful phishing attack occurred, the machine would automatically disconnect and self-isolate as well as alerting IT. 

Creating a sustainable infrastructure 

Following the restoration of various business applications, there wasn’t enough memory and storage for HRIS, ERP and other business-critical applications to run. To provide stability and resume service, we purchased cost-effective storage and reused some of G’s older equipment to keep costs low.

With stability resumed for critical applications, we assessed the most cost-effective and sustainable future solution. We made a significant change to the underlying network and infrastructure by putting front- and back-end firewalls in place and implementing hyper-converged infrastructure which included Azure Site Recovery. 

This meant that in the event of a disaster, the entire G’s business could operate in the cloud. They could also decide to keep it located there and use their on-premise solution for disaster recovery. Moving to this new, secure and monitored infrastructure, considerable increased efficiency and productivity.  

Re-engaging employees and enhancing processes.

Following the cyber incident, it was essential that all employees were trained in cyber-security essentials to leverage G’s insurance policy. We implemented online courses across the company to improve security awareness and set up direct communication with the service desk, allowing employees to submit any emails they were unsure of for assessment. 

The cyber incident had a terrible impact on the IT team, with over 50 per cent of team members leaving the business. The IT team was left depleted, disengaged and without the full array of skills and expertise needed to manage and maintain their new infrastructure and applications. We created a new global target operating model, containing the following areas: architecture and governance; development operations; business intelligence and service delivery; and infrastructure teams. We created a cost-effective and sustainable co-sourcing model, to leverage partnerships and support the infrastructure and service delivery function.

We also implemented a fully managed service desk with a 24/7 telephony solution to support the operation. Fresh Service enabled G’s to better manage tickets, inventory and change requests. Over six months, this new service desk significantly reduced open tickets and helped build a better relationship between IT and the rest of the business. The number of escalations to senior management fell from 12 per month on average to 0. 

In addition to enhancing the operating model and capability within the IT team, we also wanted to improve the way G’s engaged their other employees across the business. In addition to their existing email messaging, we developed an intranet that allowed employees to access key information and corporate messages more securely. Over 70% of the organisation use the tool regularly and this has had a really positive impact on their culture, especially during COVID-19. 

The Success

New operating model on a modern desktop with cloud infrastructure, new IT processes and a new intranet.

With a thorough understanding of the challenges posed and with a more secure and sustainable infrastructure and operating model in place, we defined a three-year IT technology roadmap. An application modernisation and BI modernisation programme were both approved following our presentation to the board. 

We have now supported G’s to hire a permanent IT Director. We still provide strategic leadership and are delivering on some ongoing projects, but we know we are leaving them in a much stronger position for growth in the future.

Over the many months we have been working with G’s we have built trusted relationships with their leadership team, improved the reputation of their IT department, brought innovation and ease to their everyday processes and saved them money in both the short- and long-term.