Orange County Grand Jury 2005-2006 Business Continuity: Can Orange County Stay Open for Business After a Disaster?

Summary Orange County government agencies have comprehensive and tested plans for responding to emergencies and for providing emergency assistance to residents. Less apparent is how prepared the agencies are to continue internal Orange County agency operations after their initial response to a disaster. In the aftermath of Y2K, 9/11, and Hurricane Katrina, planning for the resumption of business after a disaster (Business Continuity planning) has become a significant issue for many businesses and the public sector. Business Continuity Plans focus on restoring Orange County agencies’ internal operations rather than the larger external issues involved with coordinated emergency responses by state, county, cities, special districts and local agencies. For example, internal operations include payment for county obligations such as goods and services, management of child abuse cases, and maintaining jail facilities. The 2005-2006 Orange County Grand Jury conducted a study to gauge the Orange County agencies’ current Business Continuity preparedness status and found that: 1.1 The agencies’ Business Continuity Plans are incomplete. Serious omissions in most plans are: a) business recovery centers for Orange County employees displaced by a disaster, and b) backup computer sites for use if primary sites are no longer available. Incomplete plans will inevitably lengthen the recovery time period. 1.2 Current Business Continuity Plans have either not been tested or not been completely tested. An untested plan may not work in a real emergency. 1.3 While agency-specific plans exist and while Orange County’s emergency response and recovery plans provide a framework for County agencies’ individual efforts, some integrative elements are missing. For instance, there is no prioritization of critical business functions across all agencies to ensure that the functions most critical for Orange County residents are recovered first. In addition, the use of common support elements such as business recovery centers for people and backup computer sites is not always considered or coordinated. . Introduction and Purpose of the Study In the aftermath of Y2K, 9/11, and Hurricane Katrina, planning for the resumption of business after a disaster has become a significant issue for many businesses and public sector organizations. Plans devised in the days before computers became an integral part of day-to-day operations are no longer adequate. These plans consisted largely of creating periodic back-ups of computer software and data and then restoring the software and data on computers after a disaster had occurred. In today’s economy, computers and work locations for people often must be available every day (and in some cases, every minute) and plans for recovery after a disaster must be revised accordingly. Orange County has comprehensive and tested plans for responding to emergencies and for providing emergency assistance to residents of the County. The purpose of this study, however, is to determine how well prepared Orange County agencies are to continue internal operations after the initial response to a disaster.

Method of Study The study methodology includes: • Researching the impact of Business Continuity Plans on business recovery after a disaster • Reviewing another county’s approach to business continuity • Identifying best practice guidelines • Determining how well current Orange County plans conform to these guidelines through plan reviews and interviews with personnel of fourteen agencies • Reviewing the County Executive Office’s (CEO’s) assessment of eight agency Business Continuity Plans • Developing observations, findings and recommendations based on analysis of the collected information 4. Background 4.1 Business Continuity Defined 4.1.1 Business Continuity Management Business Continuity Management is a management process that identifies potential impacts and risks that threaten an organization and provides a framework for building organization-wide resilience, enabling the organization to survive the loss of part or all of its operational capability. Business Continuity 4.1.2 Business Continuity Planning Business Continuity Planning is the process of developing and testing the internal operational arrangements and procedures needed to recover from a disaster. When executed, the Business Continuity Plans allow organizations to continue serving customers by: • Ensuring that people in critical jobs are available and have the processes, equipment, and facilities they need to continue providing services • Bringing networks and computer systems back into service to support these key people Business Continuity Plans can be thought of as a part of Orange County’s Emergency Response and Recovery Plans (ERRP). They focus on restoring the County agencies’ internal operations rather than the larger external issues involved with coordinated emergency responses by state, counties, cities, special districts, and local agencies. 4.2 Lessons Learned from Previous Business Disruptions Since computers are now an integral part of day-to-day operations, their loss can have significant impacts on the survival of organizations. For example: • 93% of companies that lose their data centers for ten days or more file for bankruptcy within one year of the disaster. (Stevens Institute of Technology Research Paper) • 75% of organizations studied reached critical or total loss of functioning within two weeks after losing their computer centers. (University of Texas study) • 40% of companies go out of business within five years after a disaster. (Gartner Group research report) Not having Business Continuity Plans markedly reduces the odds that a business will survive a disaster. Support for this assertion is found in the following reports: • 43% of companies without Business Continuity Plans never reopen and 90% are out of business after two years. (University of Texas study) • Eighteen of twenty-one companies in the World Trade Center complex on 9/11/2001 with Business Continuity Plans returned to pre-9/11 levels while only two of four without plans did the same. (Penn State study) • An effective business continuity and disaster recovery plan can reduce losses by 90%. (Info Security News Magazine article in 2000) /11 identified “plans not thoroughly exercised or maintained at appropriate levels” and needs for “desks, chairs, a dial tone, voicemail, a computing device, printers and faxes” as significant lessons learned. (IBM’s report in a Disaster Recovery Journal) • Business recovery centers for people are as important as comparable centers for computers. After the Hurricane Katrina disaster, the computer operations of New Orleans’ accounting functions were not affected because the computers are located in Orange County’s data center. However, payments to city workers were delayed and were based on estimates because the accounting people had no place to go to do their work. (Interviews with Orange County system management) 4.3 Los Angeles County’s Business Continuity Program In 2002, the Los Angeles County Board of Supervisors recognized the need to upgrade their Business Continuity Program and directed their staff to prepare scope, time, and cost estimates for development of a countywide Business Continuity program. Based on recommendations from staff, the Board in early 2003 approved development of such a program and directed all departments to participate. Notable aspects of the program include: • Authorization, direction, and oversight from the Board of Supervisors • Management by a Business Continuity Steering Committee chaired by the Office of Emergency Management with support from the Chief Information Officer, Director of Internal Services, and the Auditor-Controller • An integrated, countywide plan incorporating, prioritizing, and budgeting for the requirements of all county agencies • Plan process in phases: assessment, development, and implementation • Standards and guidelines for department-level Business Continuity Plans reinforced through extensive training, and use of business continuity planning software • Inclusion of business impact and risk analysis in the assessment phase to assure that Business Continuity requirements are well understood before the plan to meet the requirements is developed • Focus on critical business processes rather than on all processes or only on computer applications • Annual Business Continuity exercises and tests as an extension of emergency response exercises and tests Business Continuity Los Angeles County has completed the assessment phase of its plan development and is well along with the development and implementation phases of the department-level Business Continuity Plans. 4.4 Best Practice Guidelines Best practice guidelines are issued by a number of industry groups to help members and other practitioners produce business continuity plans. These guidelines are also useful standards against which to compare current practices. The Business Continuity Institute’s Good Practice Guidelines, the National Fire Protection Association’s NFPA 1600 standard, and the Federal Financial Institutions Business Continuity handbook were reviewed as potential standards for comparison. The Business Continuity Institute’s guidelines were selected for this study because they captured the essential elements and were presented as a comprehensive management process. (Business Continuity Institute is an international organization of more than 2,000 Business Continuity professionals from over 50 countries.) As discussed more fully in Section 5.1 below, the elements selected by the Orange County Grand Jury are Business Continuity Management policy and on-going management, business impact analysis, risk assessment, business resumption strategies, plan reviews and tests, and audits.

Observations and Discussion 5.1 Comparison of County Agency Plans with Best Practice Guidelines The Orange County Grand Jury reviewed fourteen Orange County agency Business Continuity Plans and interviewed agency representatives using a questionnaire that was based on the Business Continuity Institute’s Best Practice Guidelines. (See Appendix 9.1 for a list of the fourteen agencies involved in the comparison.) The plans were largely based on a Business Continuity Plan model provided by the CEO in 2002. All plans conformed to some elements of the best practice guidelines. One plan contained almost all elements. The results of the reviews and interviews for each major section of the questionnaire are presented below. 5.1.1 Policy As shown in Table 1, some agencies believed that Business Continuity Management policy and guidelines were provided by the CEO as part of the emergency response and recovery plan model published in 2002; others did not. Most agencies did not treat the development, test, and maintenance of their Business Continuity Plans as formal projects and consequently did not document project structures, organizations, and implementation plans. : Policy Did the Business Continuity Management policy documentation--- Yes Partially No Include the agency’s definition of Business Continuity Management? 6 5 3 Define Business Continuity Management principles, guidelines 4 2 8 and minimum standards? Describe the structure and organization for managing the Business 3 3 8 Continuity Management project? Provide the implementation plan for the Business Continuity Management 0 3 11 project? 5.1.2 Management The elements in Table 2 are necessary parts of a formal project. Generally they were not documented by the agencies and were not a part of the process used to create the Business Continuity Plans. In most cases, agency executives did authorize business continuity work but did not do so in writing. Perhaps the most serious deficiency is the lack of Business Continuity Plan tests. Though Orange County is a leader in conducting table-top exercises, its agencies have not conducted formal tests of their plans for resuming internal operations after a disaster. Some plans were partially tested in response to local disasters and some recovery alternatives such as emergency power generators and secondary networks are routinely tested. Table 2: Management Did the Business Continuity Management documentation--- Yes Partially No Include authorization by the agency's executive/senior management? 2 3 9 Require management reports at a predetermined frequency? 1 2 11 Include a line item budget for the project? 0 0 14 Require a periodic project review? 1 4 9 Describe successful tests of the Business Continuity Plans? 0 4 10 5.1.3 Business Impact Analysis Business impact analyses enable organizations to recover their functions in priority order based upon criticality to the organizations’ missions. Typically, a business impact analysis includes the organization’s business functions, the impact of the non-availability of each function, the maximum allowable downtime before the impact is felt, and the resource requirements for resuming operation of the function. As shown in Table 3, current plans do include some business impact analysis components but no plan was complete. Without complete plans, business functions must be prioritized in real time after a disaster occurs. All the resources needed to resume operations of the critical functions may not be available at that time. Business Continuity Table 3: Business Impact Analysis Did the business impact analysis documentation--- Yes Partially No Prioritize business functions based on criticality? 6 3 4 Specify maximum allowable downtime for each function? 2 4 7 Define staff numbers and skills for each critical function? 2 7 4 Specify vital records and data for each critical function? 5 6 2 Specify cabling and network links for each critical function? 2 5 6 Identify requirements for alternative locations for each critical function? 4 4 5 Identify suppliers for each critical function? 5 5 3 Note: The number of agencies responding to this section of the survey is one less than for the other sections since the Business impact analysis does not apply to service organizations such as the Orange County Data Center. 5.1.4 Risk Assessment Like business impact analysis, risk assessments are important prerequisites to the creation of Business Continuity Plans. Prioritizing risks based on their impact and probability of occurrence assures that mitigation and recovery efforts are focused on the most serious risks. Orange County agencies have not completely documented their risk assessments and many have not conducted complete risk assessments as indicated in Table 4. Table 4: Risk Assessment Did the risk assessment documentation--- Yes Partially No Identify single points of failure? 4 5 5 Prioritize the threats to the organization or critical function? 2 4 8 Define action plans for the risks to be addressed? 2 6 6 Identify risks that are not to be addressed? 0 0 14 5.1.5 Business Continuity Strategies Organizations use a number of different strategies to recover their internal operations after a disaster. Table 5 lists some popular strategies and how frequently their use has been documented by Orange County agencies. Although an 18 person business recovery center has been established by the CEO in its exceptionally well-designed data center, most agencies do not yet have plans for business recovery centers to which agency employees can go to resume their critical business functions. Backup facilities for computer systems are also not yet included in many plans. Reliance on acquiring computers and computer rooms after a disaster occurs is problematic at best. If a disaster is widespread, many organizations will be looking for computers and suitable facilities and supplies may be limited. Also, most experts agree that rebuilding systems from backup media will take from three to seven days and that may be too long for critical business functions. : Strategies Did the business continuity strategies documentation--- Yes Partially No Describe risk mitigation plans for the critical risks? 1 7 6 Include use of existing in-company facilities as recovery locations? 9 0 5 Include displacing staff performing less critical functions by staff performing 4 1 9 more critical functions? Include "working from home" or other non-corporate locations? 5 2 7 Include reciprocal agreements with other government agencies? 4 3 7 Include third party alternative site arrangements from a service company? 1 0 13 Describe "ship-in" contract arrangements with suppliers? 2 1 11 Identify dual site operations and/or continuous availability solutions? 2 5 7 5.1.6 Review Since change is continuous for most organizations, Business Continuity Plans must be reviewed periodically and/or when some significant event occurs to assure that they are current. Some Orange County agencies have conducted reviews after significant events but, except for reviews and updates of contact names, there have not been any regularly scheduled periodic reviews as indicated in Table 6. Table 6: Reviews Were Business Continuity Plans--- Yes Partially No Reviewed monthly or quarterly for contact details? 4 3 7 Reviewed/tested annually? 0 5 9 Reviewed when there is a significant change in technology? 5 1 8 Reviewed when there is a major business process change? 5 1 8 Reviewed when there is a significant change in staff? 5 2 7 Documented for each business unit? 4 2 8 5.1.7 Audit Auditing is a process that independently ensures that an organization has an effective Business Continuity process and plan. No such audits have been conducted of any of the agencies’ processes and plans. Table 7: Audit Was there a documented Business Continuity Management audit Yes Partially No Validation of compliance with Business Continuity Management 0 0 14 policies and standards? Review of Business Continuity Management solutions? 0 0 14 Validation of the Business Continuity Plans? 0 0 14 Verification that appropriate tests/exercises are taking place? 0 0 14 Highlighting deficiencies and issues and ensuring their resolution? 0 0 14 5.2 CEO Assessment of Agency Business Continuity Plans The CEO is performing its own assessments of agency Business Continuity Plans. Competency Level assessments have been completed for eight agencies. The assessments measure agency competency over 19 factors. The chart below shows an average score for Business Continuity each agency across all 19 factors. (See Appendix 9.2 for the eight agencies included in this assessment.) The score descriptions are as follows: Score Description 1 Challenge - Item is missing from plan or needs considerable work 2 Sufficient - Competency meets the minimum criteria 3 Satisfactory - Fair competency with significant shortcomings 4 Good - Generally sound competency with notable errors 5 Very Good - Above average competency with some errors 6 Excellent - Outstanding competency with minor errors Only one agency’s average score was above the “Competency meets the minimum criteria” level. All others were below that level. This assessment is consistent with the Orange County Grand Jury’s comparison of Orange County Business Continuity Plans with best practice guidelines. Both suggest that additional work needs to be done on these plans. Business Continuity Competency Levels (CEO Assessment) 6 Excellent 5 4 Average 3 Satisfactory Score 2 1 0 A B C D E F G H Agencies Assessed 5.3 General Observations Most of the County’s current Business Continuity Plans are embedded in the agency-level emergency response and recovery plans. The primary focus of these plans is, properly, on emergency response. Consequently, business recovery does not receive as much attention and is less complete. Los Angeles County opted to create separate Business Continuity Plans to assure that they are complete, tested, and maintained. While agency-specific Business Continuity processes and plans are appropriate, County- level leadership, processes, and plans are also needed. Leadership by the Board of Supervisors would ensure use of a standard Business Continuity Management process Strategic Priorities” for 2005. .

Findings In accordance with California Penal Code Sections 933 and 933.05, each finding will be responded to by the government entity to which it is addressed. The responses are to be submitted to the Presiding Judge of the Superior Court. The 2005-2006 Orange County Grand Jury has arrived at the following findings: 6.1 Current Business Continuity Plans are incomplete: Plans were developed without a formal process and are incomplete. A lack of focus on the most critical business functions and omission of business recovery centers for people, processes, and backup computer facilities are serious deficiencies in most plans. 6.2 Current Business Continuity Plans either have not been tested or testing is not complete: Though Orange County is a leader in conducting table-top exercises for emergency responses to disasters, its agencies have not conducted formal tests of their plans for resuming internal operations after a disaster. Some plans were partially tested in response to local disasters and recovery alternatives such as emergency power generators are routinely tested. 6.3 Current Business Continuity Plans are not integrated: Some integrative elements are missing from the existing plans. For example, there is no prioritization of business functions across all agencies and no coordinated approach to defining and acquiring common support elements such as business recovery centers for people and backup facilities for computers. The CEO has initiated such coordination recently but is working at lower organizational levels without much visibility at the executive level. Responses to Findings 6.1 through 6.3 are required from the Orange County Board of Supervisors and requested from the County Executive Officer.

Recommendations In accordance with California Penal Code Sections 933 and 933.05, each recommendation will be responded to by the government entity to which it is addressed. The responses are Business Continuity to be submitted to the Presiding Judge of the Superior Court. Based on the findings, the 2005-2006 Orange County Grand Jury makes the following recommendations: 7.1 Active leadership from the Board of Supervisors: The Board of Supervisors should ensure the Business Continuity planning effort at County-level by guiding the planning, providing the necessary funding, and exercising oversight over the project. (See Findings 6.1 through 6.3.) 7.2 Use of best practice processes and project structure: The Board of Supervisors and CEO should require the use of best practice processes and project structure for further Business Continuity Plan development. (See Findings 6.1 and 6.2.) 7.3 Executive Steering Committee: The Board of Supervisors should direct the CEO to establish an executive steering committee to carry out its policies and to guide the Business Continuity project. The committee, led by the CEO, will ensure a collaborative effort by the agencies and a uniform approach to plan development including: guidelines, training, and use of standard Business Continuity software tools. (See Findings 6.1 through 6.3.) 7.4 Development of a County-level plan: The Board of Supervisors and CEO should require the development of a County-level Business Continuity Plan. This plan should include prioritization of critical business functions across all county agencies and use of common support elements such as backup facilities for both people and computer systems (See Findings 6.3.) Responses to Recommendations 7.1 through 7.4 are required from the Orange County Board of Supervisors. Responses to Recommendations 7.2 through7.4 are requested from the County Executive Officer.