County Project Management – a Case Study*

The Grand Jury found that project management across the County is inconsistent in its use of ISO/PMI best practices even though the expertise and resources to employ those practices are available through ITS. The EHR project governance, as defined by PMI, should have begun with a qualified project manager and a County project plan as early as 2009, when the search for a qualified vendor began. It did not. (FA-01, FA-02, FA-04, FA-05, FA-06, FA-07, FA-08, FA-13, FA-14, FA-15)

The Grand Jury found that the County maintained a website with PMI-compliant forms, policies, and procedures during the EHR project. Not all project managers were aware of the website's existence and those who knew were not required to use any of the recommended procedures or documents. As of May 2015 no countywide policy, applicable to all agencies, identifies required project management standards. Merely posting recommended templates and documents on a website is not a clearly stated policy. (FA-04, FA-05, FA-06, FA-07, FA-08)

The EHR is an enterprise-level project based on its cost and its effect on other departments and agencies. Such effects include the integration with the County data network and the interface with the Auditor-Controller's office. Enterprise projects require formal risk assessments prior to a project start date. The risk assessment documented and approved on the APAQ did not meet the level of detail expected for an enterprise-level project. By limiting risk assessment to federal incentive reimbursement and fines, the County ultimately failed to address the impact the system could have on patient safety; the daily workflow of doctors, nurses, lab technicians, pharmacists, billing, and registration; and the effect on VCHCA's efficiency and profitability. (FA-01, FA-02, FA- 12, FA-13, FA-14)

Scheduled quarterly ITC status reports for the EHR project were infrequent and inadequate, allowing the project to expand in both scope/size and cost. Failure to have quantitative project status and Estimate at Completion (EAC) reports caused ITC, the governing group, to miss opportunities to identify problems and take corrective action throughout the EHR project. (FA-09, FA-10, FA-11, FA-13, FA-14, FA-23, FA-24)

Information contained in the EHR project and plan documentation provided to the Grand Jury was incomplete and inaccurate and did not reflect effective project management. These documents did not accurately allocate or report the amount of time needed or spent on projects by County employees. They did not track the progress and cost of the individual tasks assigned to County resources. (FA-13, FA-14, FA-15)

The Grand Jury found that County labor required for the EHR project, as documented in the APAQ, was severely underestimated and ultimately proved more costly than originally proposed. The Grand Jury could not accurately verify the labor hours and cost due to the tracking methodology employed during the project. (FA-13, FA-14, FA-15, FA-17, FA-19, FA-22, FA-23)

VCHCA knew from its research and the ITC knew from its status reports that there was a shortage of County personnel assigned to the project, but both failed to take the necessary and timely corrective action. (FA-14, FA-25)

VCHCA failed to develop a project plan to reflect the hours and resources necessary to integrate with the Cerner production schedule. (FA-01, FA-15, FA-16) Page | 10 Grand Jury Report Response: County Project Management – A Case Study

The ITC allowed the adoption of an APAQ that gave priority to meeting a "Meaningful Use" date. Setting this priority distracted from establishing a PMI-compliant project plan. VCHCA adopted the vendor's "Event Driven Methodology," which covered Cerner's production but did not integrate into any County plan. (FA-12)

The ITC had no way to quantitatively measure successful completion of the APAQ goals. The project cannot be called complete until all outstanding substantive issues related to satisfaction of the APAQ goals are resolved and accurate project performance data is produced. (FA-12, FA-20, FA-23, FA-24) RESPONSE: Partially Agree. The ITC did not have quantitative measurements related to specific goals. However, the goals identified in the APAQ were all successfully met, given the aforementioned oversight of the project by the HCA Executive Steering Committee; and Meaningful Use was achieved to a laudable degree. Additionally, as noted in response to finding FA-13, the quarterly ITC project review process includes specific questions to be answered by each responsible project manager on the status of: 1) Project costs; 2) Project deliverables/schedule; and 3) New or unidentified risks, among other questions.

plus R-01, R-02, R-03, R-04, R-05 and R-06) NOTE: Also includes responses to FA-08, FA-11, FA-13, FA-14, FA-15, FA-17, FA-20, FA-22 and FA-24. Required Auditor-Controller Respondent: (FI-01, FI-03, FI-04, FI-05, FI-06, FI-08, FI-10 and FI-11 plus R-03 and R-04) (for information) County Executive Officer Requested Respondents: (FI-01, FI-02, FI-03, FI-04, FI-05, FI-06, FI-07, FI-08, FI-09, FI-10 and

The October 3, 2011 Automated Project Assessment Questionnaire (APAQ) for the Cerner EHR project presented to the ITC identified three goals, one measurement for success, and a minimal risk assessment. Goal 1: To replace VCHCA's clinical record system with a single system which complies with the HITECH provision of ARRA. Goal 2: To automate and integrate the patient accounting and supply chain management with the new clinical record system. Goal 3: To automate and integrate billing and claim management for leveraging information across the enterprise. Measurement: The single measure of this project's success would be achieving its first • attestation in accordance with federal requirements under the "Stage 1Meaningful Use" (MU) criteria by September 1, 2013. Risk assessment: Risk would be limited to the loss of federal reimbursement allocations and the issuance of fines if the project was not started by January 1, 2012, and completed by September 1, 2013. RESPONSE: Partially Agree. We agree with the listed goals. However, we disagree with the inference that the only risk assessment done was the above initial statement contained within the original project APAQ. Risk assessment was conducted on an ongoing basis, throughout the entirety of the project. A risk management plan consists of an assessment of the project, identification of the risks, determination of the impacts and development of a mitigation plan, where required. At the time the contract was signed, HCA's primary responsibilities were to provide staff and end-user computing devices. The staffing risks were as follows: acquiring a full- time project manager, informatics analysts (i.e. clinically-trained systems analysts) and subject-matter experts. Each of these were identified as potential risks and a mitigation strategy was put into place. These included the request in the Board letter dated July 24, 2012, wherein funding was requested for a full-time project manager; additional allocations of internal staff to the project in both a Subject Matter Expert (SME) and Analyst capacity; and contract Informatics staff from Novacoast to augment internal staff, mitigate project risk, and ensure project All of these resources were subsequently put into place on the completion. project. The use of Novacoast contract staff, as opposed to hiring permanent County employees, avoided an estimated 6- to 12-month recruitment process and allowed for the accelerated hiring of project staff to ensure timely project completion. Additionally, end-user computing devices were identified as a risk area, both in terms of ability to procure (funding) and usability. Based on an Page |2 Grand Jury Report Response: County Project Management - A Case Study analysis by HCA IT, user focus groups, and a review of several other local hospitals, an equipment list was prepared and sent to the HCA Cerner Steering Committee and, subsequently, the Board of Supervisors for funding.

As indicated in its agenda, status reports on open projects need not be provided at ITC meetings unless requested by the ITC. Of those reports given, a quantitative assessment is not required. The reports reviewed for EHR were qualitative and thus could not be monitored with any metrics. They also failed to provide status for goals stated in the APAQ. RESPONSE: Disagree. All projects approved by the ITC are required to submit quarterly written status reports until completed. In-person status reports are optional and/or at the request of the ITC. Additionally, as specified in the Project Management Institutes Book of Knowledge (PMBOK) Guide 5th Edition - sections 4.4.1.2, 4.4.1.3, 11.3 and 11.4 - both schedule and cost are key components in project monitoring and control; and both qualitative and quantitative measures may be used for assessing risk. It should be noted that the quarterly ITC project review process includes specific questions to be answered by each responsible project manager on the status of: 1) Project costs; 2) Project deliverables/schedule; and 3) New or unidentified risks, among other questions. Additionally, the County's Information Technology Committee was only one of three entities monitoring the status of the Cerner project. The HCA Cerner Project Executive Steering Committee had primary oversight; and the HCA Hospital Oversight Committee also monitored the ongoing progress of the Cerner Project.

Project status reports for Cerner presented to the ITC on October 18, 2012; January 9, 2013; April 11, 2013; and October 9, 2013, were primarily qualitative; i.e. no quantitative progress was provided which would support a performance-based prediction of total project costs at the completion date and a performance-based prediction of the completion date. All reports stated that there was a shortage of County labor and thus milestones would not be met. RESPONSE: Disagree. As noted in response to finding FA-13, the quarterly ITC project review process includes specific questions to be answered by each responsible project manager on the status of: 1) Project costs; 2) Project deliverables/schedule; and 3) New or unidentified risks, among other questions. Evidence that actionable reporting was sufficient was demonstrated through HCA's immediate hiring of Novacoast contract staff. The use of Novacoast contract staff, as opposed to hiring permanent County employees, avoided an estimated 6- to 12-month recruitment process and allowed for the timely hiring of project staff to address the resource shortage noted in the project status report. As a result of these staffing actions, the project went live on schedule and the County was able to qualify for 19 million dollars in American Recovery and Reinvestment Act Stage 1 Meaningful Use incentive funds by the required September 1, 2013 Stage deadline.

The Grand Jury requested all project files that demonstrate how the Cerner project was conducted, including activities that cover the pre-Cerner- installation plan, the Cerner integration efforts, and the planned post-"Go-Live" maintenance tasks. The Grand Jury only received and Page | 3 Grand Jury Report Response: County Project Management – A Case Study reviewed the following project plans created by Cerner: Project file number one (submitted to the Grand Jury on January 21, 2015, as a Microsoft Project plan file), spanned the period from October 24, 2011 to September 30, 2013, and represented Phase 1 of the EHR System project implementation. Project file number two (submitted to the Grand Jury on January 21, 2015, as a Microsoft Project file, spanned the period from August 26, 2013, to March 18, 2014, and represented Phase 2 of the EHR System project implementation. Project file number three (submitted to the Grand Jury on January 21, 2015) spanned project period from "9/11/14 to 2/13/2014" [sic] and represented the project tasks to be completed, achieving "Federal Core Measures." Project file number four (submitted to the Grand Jury on January 21, 2015 spans the period from March 1, 2014, to December 27, 2019, and includes Cerner implementation of the Cerner Remote Hosting option and maintenance. The plans reviewed did not have the PMI-required data for County labor needed to perform specific tasks. Resources such as equipment and labor for the project milestones were missing. The Grand Jury did not receive a County PMI-compliant plan for the Cerner project that was referenced in the BOS response to a 2013-2014 Ventura County Grand Jury report: "HCA did use a formal structured policy management plan for the Cerner System Implementation. The plan presented was the project plan required of Cerner in their contract with the County. This plan followed an Event Driven Project Management methodology." The project plans presented to the Grand Jury were developed by Cerner and covered only Cerner tasks. [Re 01, Re 07, Re 08] RESPONSE: Disagree. The submitted project files documented the hundreds of specific, detailed tasks required of the project team to complete the project, and covered all phases of the project. The files provided evidence that, consistent with the County's response to the original June 2013-2014 Grand Jury Report on the Cerner project, a formal, structured project-management plan was utilized, i.e., Cerner's Event-Driven methodology. The County did not assert that the utilized plan was a PMBOK-based plan; or that an independent County plan had been developed; or that one was required.

According to the BOS response to the 2013-2014 Grand Jury Report, Cerner uses a management method it calls "Event Driven Project Management Methodology." This methodology cannot be found in the standard ISO or PMI Management literature. It is used by Cerner as its own production planning tool because Cerner's work is driven by the customer's response (known as an "event") to its requirements. Cerner has no authority over the customer's workforce. Therefore, Cerner cannot predict when the customer will respond to its requests or how complete the response will be. The VCHCA, by contrast, was being driven by a very firm scheduled due date, the "Go-Live" date of July 1, 2013. Issues arise when event-driven methodologies are used on schedule-driven projects. A Federal Aviation Administration case study concludes that scope creep/project expansion and budget overruns are a consequence of this mismatch. (Ref-01, Ref- 07, Ref-08, Ref-09, Ref-10, Ref-11, Ref-12) Page |4 Grand Jury Report Response: County Project Management – A Case Study RESPONSE: Agree. We agree, in that the FAA study included these statements and reached this conclusion; however, we did not experience the same result with the Cerner Project, i.e. creep/project expansion or budget overruns caused by such creep/project expansion.

As of May 1, 2015, the total cost of the EHR system had not been publicly disclosed. The Grand Jury requested audit data from the Ventura County Auditor-Controller in order to evaluate implementation performance. At the time of the request, no audit had been completed. However, the Grand Jury received the following information: County employee project time was not always charged to the project. • Equipment purchased had not been inventoried or not charged to the project. Payments to vendors could not always be tracked to the project. • The true cost of EHR could not be ascertained. RESPONSE: Partially Disagree. While totals project costs continue to be finalized the statements above reflect preliminary (i.e., draft) findings, some of which will be clarified, or addressed, by the Health Care Agency and, we anticipate, not be present in the final report.

The EHR funding process began with a VCHCA letter to the BOS dated October 4, 2011, that recommended the purchase of an EHR system with the vendor Cerner for $32,466,000. The recommended project had an 18- month duration with a set of activities as follows: Month 1. Contract Execution Month 2. Hire Implementation Team Month 3. Project Kickoff Month 4-5 Training, Design Month 5-12 Build, Validate Month 13 Integration Test 1, Integration Test 2, Go-Live • Month 14-18 Post Live System Check, Begin Production Maintenance Activities • [Ref-14, Ref-15] RESPONSE: Agree. However, it was clearly identified within this initial Board letter that this amount did not represent the total project cost or needs; and that additional project components and costs, including project equipment and staffing/consulting costs, would be forthcoming.

The BOS gave approval of and authorization for EHR expenses to be increased from the initial $32,466,000 on October 4, 2011, to over $71,000,000 as of May 7, 2015. [Ref-13, Ref-14] The history of BOS authorizations is as follows: April 10, 2012 for $5,110,980 Cerner charges for services [Ref-15] July 24, 2012 for $5,748,500 for staffing services, computerized physician order entry software and 5% contingency; $4,450,000 to Nova Coast Corp. For staffing support during Page | 5 Grand Jury Report Response: County Project Management – A Case Study EHR (July 24, 2012 to June 30, 2013); $125,000 to Lynx Corporation for physician order set creation services (July 24, 2012 to June 30, 2013) [Ref-16, Ref-17] May 21, 2013 for $6,337,196 for medical equipment, end user equipment, licenses and • software necessary to integrate patient care into EHR [Ref-18] June 24, 2014 for $19,160,788 Cerner Remote Hosting [Ref-19] • September 23, 2014 for $514,560 not to exceed $599,880 for annual hosting of the Health Portal, portal reporting database, and Helping Hands Level 1 support [Ref-20] November 4, 2014 for $99,900 with the ability to expand to $570,000; Second amendment to contract with Barnard Howard LLC [Ref-21] December 2, 2014 for $379,000 for first amendment to contract with Luminous Technology Group (January 3, 2014 to June 30, 2015) [Ref-22] RESPONSE: Disagree. The $71 million cost identified as approved by the Board for the project includes approximately $20 million which are not appropriately associated with project implementation costs, as they are for remote-hosting costs extending 10 years into the future. This project was initiated under unusual time constraints associated with qualifying for significant and unprecedented federal incentives of $19 million. These time constraints included tight deadlines for automating all patient care; operations; and financial and patient account/billings functions across our entire integrated system of care, which is much larger than most health systems and the largest in Ventura County, and which includes two hospitals as well as thirty-five outpatient clinics located throughout the County. It is acknowledged that total project costs were not able to be fully identified prior to initial Board approval of the project. However, with the exception of the $20 million for remote hosting; 3 smaller post-implementation contracts for billing support; and related tasks for approximately $1 million, the initial and subsequent Board letters indicated that future Board letters would describe necessary additional costs. This occurred over the course of three subsequent Board letters, bringing the total planned implementation costs to approximately $50 million.

The Cerner contract estimated Cerner's travel expenses at $346,000. As of December 31, 2014, the County had reimbursed Cerner Corporation over $834,000 for travel expenses. [Ref-13] RESPONSE: Agree. However, during the project an amendment to the contract was signed, decreasing other costs and increasing the travel allocation from $346,000 to $696,000. Final travel expenses are being reconciled as part of the post-project audit being conducted by the Auditor-Controller and may, in fact, exceed the modified $696,000 amount.

As of the July 1, 2013 "Go-Live" date, over 100,000 County employee hours had been used for Phase 1 of the EHR implementation. RESPONSE: Disagree. We are unable to verify this number. Also, a number within this range would have to include system training for line staff not assigned to the project. The total number of County employee hours at go-live for employees dedicated Page | 6 Grand Jury Report Response: County Project Management – A Case Study to the project is approximately 63,000.

The Ventura County Star, citing the EHR as a contributing factor, reported an $8,550,000 loss at the Ventura County Medical Center (VCMC) in fiscal year 2014. [Ref-23] RESPONSE: Agree. However, the operating loss for 2014 was due to a number of one-time events and expenditures, only one of which was related to the Cerner EHR project. As a comparison, operating profits for the year 2015 are estimated to be approximately $15 million. The portion of the fiscal year 2014 operating loss attributable to the Cerner project was due to a planned 25% initial reduction in staff productivity following system implementation.

Problems stemming from the implementation of the EHR system affected the VCHCA's ability to bill correctly, resulting in an operational loss. These problems created a large volume of uncollectable patient billing (estimated at $40,000,000) that, even after being corrected, still left a substantial portion unrecoverable. RESPONSE: Disagree. We are unable to determine from where the "$40 million in uncollectible patient billing" stated is derived. VCHCA's figure for uncollected patient billing is a significantly small percentage of this amount, and none of the Cerner-related billing backlog is deemed to be "uncollectible". Additionally, one of the many benefits of the system is that the cash-collection trend at VCMC has continually improved since system implementation (increasing from $92 million during the first six months following Cerner implementation, to more than $122 million during the most recent six months), and now actually exceeds pre-Cerner system implementation levels. Also of note, the current loan balance to VCMC as of today stands at $14.5 million, down from a high of $73 million. FINDINGS