Bos, CEO and HCA*

The Grand Jury found that after Go-Live a significant level of concern was raised by clinical staff to IT regarding potential impacts of observed EHR-related risks on patient well-being. (FA- 27, FA-37, FA-42, FA-43, FA-44)

The Grand Jury found systemic deficiencies in the process used by VCHCA to develop and vet the adequacy of the EHR project requirements specification. For example: The "number of simultaneous users" specification was clearly developed using an inadequate analysis strategy, and the specification reasonableness was not validated by appropriate independent EHR SMEs. A performance requirement for a maximum window update time was not developed. VCHCA failed to develop a mutually agreeable specification with Cerner in the contract, as part of an EHR acceptance requirement. VCHCA did not have an effective mechanism to gauge the comprehensiveness and quality of the EHR implementation and its test development process. VCHCA did not specify the minimum required FTE staffing level that IT/Informatics . management and an independent EHR SME agreed was both necessary and sufficient to fully accomplish the goals of the project. Without this staffing it was not possible to conduct rigorous testing in the time period specified by the Cerner Event Driven Project file.

The Grand Jury found no evidence that project requirements were formally specified, which precluded generating a complete and quantifiable test plan to verify overall EHR quality throughout the Implementation stage. The actual project was guided primarily by untestable goals to meet the federal stage 1 attestation. (FA-02, FA-17, FA-22, FA-41, FA-42)

The Grand Jury found no effective independent review of the EHR project before the release of the RFP, before contract signing, nor continuing periodically during the course of the project. Such an independent review would include SMEs from outside the VCHCA who have HER Implementation experience and also clinical staff with experience in the VCHCA. (FA-02, FA-05, FA-17, FA-28, FA-29)

The Grand Jury found that the lack of an effective Risk Management Plan resulted in significant impact on project quality and cost. Developing and maintaining such a plan would have exposed potential problems and triggered mitigations that could have avoided or lessened the undesirable consequences. For example, training did not satisfactorily address learning retention losses with timely hands-on refresher courses using an EHR domain and more robust training materials. Nor did it adequately stress the importance of accuracy using discipline-specific examples of correct vs. incorrect situations (e.g., data entry accuracy). (FA-03, FA-IO, FA-17, FA- 27, FA-39, FA-46)

The Grand Jury found that EHR project execution was directed solely by the Cerner Event Driven Methodology and key events and dates in the Cerner Microsoft Project file-to the exclusion of other important VCHCA-specific considerations. The EHR Implementation had significant undiscovered problems at Go-Live caused by issues such as: the inflexible July 1, 2013 Go-Live date; the 14-month integration schedule; the lack of slack in the schedule; and the lack of documented testable requirements before proceeding to the next stage. As a consequence, waiting to address residual quality issues (e.g., software bugs) until after Go-Live made patient care more challenging in the interim. However, due to alert staff, temporary workarounds were developed to maintain patient care standards. (FA-12, FA-13, FA-22, FA-26, FA-43)

The Grand Jury found that, by failing to have quantitative data to predict impacts on the Go-Live date, project management was unable to convince VCHCA administration to support the project staffing levels and ordering dates of materials necessary to deliver an operationally acceptable product. EHR project management did not utilize industry-accepted best practices FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks project management techniques (e.g., PMI) for project planning and quantitative reporting of VCHCA labor and material schedules, nor for status against those schedules. (FA-12, FA-13, FA- 14, FA-15, FA-24, FA-26, FA-28, FA-29, FA-31, FA-32, FA-33, FA-36, FA-37, FA-38, FA-39, FA- 40, FA-41, FA-42, FA-48, FA-49)

The Grand Jury found that VCHCA research and ITC status reports both indicated a shortage of personnel assigned to the EHR project. However, VCHCA and ITC failed to take the necessary and timely corrective action. (FA-26, FA-28, FA-29)

The Grand Jury found that VCHCA failed to develop a project plan to reflect VCHCA staffing hours and resources necessary to integrate with the Cerner production schedule. (FA-13) FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks

Cerner would not agree to any requirement on window update time in the contract. Most clinical staff users consider any update time exceeding 2 to 3 seconds unacceptable because it affects concentration and degrades productivity. Response: Partially Disagree. While there are not service level agreements around the response time, this is not uncommon in the industry and Cerner does provide guidelines that it will adhere to for transaction performance. The reason Cerner (or other vendors) do not agree to an end-user performance agreement is because there are many client-owned components between the user and Cerner's hardware they don't control. Examples include: the workstation (age, memory), web browser version (and plugins), Citrix Receiver (interfaces to Cerner platforms), network equipment, and proximity to wireless antennas (where applicable). In addition to these technical components, there are also user work habits to consider, such as opening 10 or more applications concurrently. It is common practice in the Information Technology Industry today to not provide these commitments. In the later part of the 20th Century, it was common for a vendor to own the mainframe, desktop device, and network. This is still true in some locations; however, current technology is more distributed with no single vendor owning the entire technology infrastructure. Therefore, commitments can only be made based on what the vendor can control.

The contract required Cerner to develop a "Work Plan" that would describe mutual expectations and work to be performed by Cerner and VCHCA during the EHR delivery. The Cerner Work Plan was supposed to contain detailed information, including but not limited to schedule, tasks, estimates, durations, deliverables, critical events, task dependencies, resource assignments, specifications, and payment schedules. No provision of the Cerner EHR contract limited VCHCA to exclusively use the Cerner Work Plan for managing VCHCA labor and/or material. Response: Disagree. It is true there was no provision preventing the use of an HCA developed plan; however, the above statement infers the HCA should have provided a separate resource plan. This was unnecessary due to the way in which the solution project teams were formed. Additionally, on large projects, the County traditionally works with IT vendors off a single, common work plan to avoid duplication and the issues associated with keeping two work plans synchronized. The budgeted costs were developed based on the Cerner recommended approach (which the Cerner plan/methodology supported) of assigning a fixed number of resources to teams at a specific level of effort. This is a proven PMI approach to resource management. Then from a project management oversight level, the project manager monitors that resources are applied at the proper, agreed upon level and the solution lead assures the specific tasks are being completed. This approach worked as expected where either the project manager or the solution lead raised an issue that tasks were falling behind due to insufficient focus/availability of resources. These were escalated to organization management and the Cerner Steering Committee to be addressed. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks

The Work Plan Cerner delivered during the course of the project was documented in a Microsoft Project file. This file was described by VCHCA as reflecting the Cerner "Event Driven" Project Management Methodology. Cerner Event Driven Project files contain only Cerner-owned tasks, with scheduling and manpower loading. They do not contain any VCHCA labor hours. The key event in the schedule was the project Go-Live milestone of July 1, 2013. VCHCA's project manager was expected to ensure that VCHCA maintain this schedule in order to qualify for the financial incentives of HITECH stage 1 Meaningful Use. [Ref-03, Ref-04] Upon examining the Cerner Microsoft Project file for "Implementation" Phase 1 of the EHR project-spanning the time period from "contract signing" (October 2011) through "end of maintenance" (October 2013)—the Grand Jury observed that: Cerner did not "populate" the project file with any VCHCA labor tasks or hours. VCHCA did not augment the project file with its own staff resources and tasks. VCHCA did not create any independent project plan for the VCHCA staffing resources and tasks. Response: Disagree. As responded to in Fact FA-12, there is an inference that there was a fault in not maintaining a separate resource plan or providing that detail to the Cerner project plan. The HCA project team did maintain a list of resources assigned to the Cerner project teams. This list was used for initial HCA resource budgeting, the basis for weekly monitoring of resources applied to the project, as well as accounting for hours against the resource budget (in conjunction with staff charging to a Cerner specific charge code). Based on the way the resources were assigned and labor tracked, the project team and management had sufficient visibility into any staffing issues as well as resource costs.

The October 3, 2011 APAQ for the Cerner EHR project presented to the ITC identified three goals, one measurement for success, and a minimal risk assessment. Goal 1: To replace VCHCA's clinical record system with a single system that complies with the HITECH provision of ARRA Goal 2: To automate and integrate the patient accounting and supply chain management . with the new clinical record system Goal 3: To automate and integrate billing and claim management for leveraging information across the County Measurement: The single measure of this project's success would be achieving its first "attestation" in accordance with federal requirements under the "Stage 1 Meaningful Use" criteria by September 1, 2013. Risk assessment: Risk would be limited to the loss of federal reimbursement allocations and the issuance of fines if the project was not started by January 1, 2012, and completed by September 1, 2013. Response: Disagree. We agree with the listed goals; however, we disagree with the inference that the only risk assessment done was the above initial statement contained with the original project APAQ. Risk assessment was conducted on an ongoing basis, throughout the entirety of the project. As noted in the response to fact FA-03, the risks and challenges associated with healthcare technology projects are well-known to the professionals who work in the field, and there was significant involvement of such physicians, nurses and other medical clinicians and administrators in both selecting and implementing the Cerner system. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks Additionally, a risk management plan consists of an assessment of the project, identification of the risks, determination of the impacts and development of a mitigation plan where required. At the time the contract was signed, HCA's primary responsibilities were to provide staff and end-user computing devices. The staffing risks were to acquire a full time project manager, informatics analysts (i.e. clinically trained systems analysts) and subject matter experts. Each of these were identified as potential risks and a mitigation strategy was put into place. These included the request in the Board Letter dated July 24, 2012, where funding was requested for a full time project manager, additional allocations of internal staff to the project in both a Subject Matter Expert (SME) and Analyst capacity, and contract Informatics staff from Novacoast to augment internal staff, mitigate project risk, and ensure project completion. All of these resources were subsequently put into place on the project. The use of Novacoast contract staff, as opposed to hiring permanent County employees, avoided an estimated 6 to 12 month recruitment process and allowed for the accelerated hiring of project staff to ensure timely project completion. Additionally, end-user computing devices were identified as a risk area both in terms of ability to procure (funding) and usability. Based on an analysis by the HCA iT, user focus groups, as well as reviewing several other local hospitals, an equipment list was prepared and sent to the HCA Cerner Steering Committee and subsequently the Board of Supervisors or funding.

Cerner performed the overall EHR system design based on VCHCA's parameters (i.e., 600 simultaneous users, 56 Solutions, 2 hospitals, 40 clinics, and a Ventura-hosted server farm). Response: Disagree. The design of the hardware was for 1,200 concurrent users based on a contract requirement for 1,000 concurrent users.

EHR Project Kick-Off for VCHCA was originally planned by Cerner for month three of the contract timeline (January 2012) but did not happen until May 2012 (month seven)—a four-month schedule slip. Response: Agree. This project was initiated under unusual time constraints associated with qualifying for significant and unprecedented federal incentives of $19 million. These time constraints included tight deadlines for project staffing and kickoff; automating all patient care; operations; and financial and patient account/billings functions across our entire integrated system of care, which is much larger than most health systems and the largest in Ventura County, and includes two hospitals as well as thirty-five outpatient clinics located throughout the County. It is acknowledged that while the official kickoff date for the project was indeed delayed due to the greater than anticipated time required to identify and put in place both internal and external (contract) project staff resources, the project was ultimately delivered successfully on time and both met and exceeded all meaningful use attestation requirements.

VCHCA did not provide sufficient analyst and SME staff to meet scheduled key dates: Many other comparable-size Cerner customer institutions employ over 50 Informatics support staff. Cerner's original estimate for VCHCA's labor for Implementation was 31.5 experienced Full-Time Equivalent (FTE) staff (analysts and SMEs). [Ref-04] The APAQ for the EHR stated that approximately 30 dedicated clinical analysts would be needed. [Ref-11] VCHCA provided on average 24 FTE staff to support Phase 1 Implementation: FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks 14 full-time VCHCA staff (=14 FTE) . 22 part-time "borrowed" VCHCA staff (=5 FTE) . 5 full-time contractors (=5 FTE) VCHCA management and staff did not have prior hands-on experience with Cerner system Solutions Implementation, Build, or Maintenance. Limiting staffing to less than Cerner-recommended and IT-requested levels helped VCHCA hold down costs. It also delayed efforts to identify and fix EHR quality issues (e.g., "bugs") until after Go-Live. Response: Disagree. The most important measure of sufficiency of the analysts in terms of numbers or skills is whether or not the key dates were met with the needed quality. All of the key dates were met with quality confirmed by the experienced Cerner solution leads. The numbers shown are the staff at implementation. The HCA's staffing was changing monthly as contractors were brought on board and released or HCA staff was replaced, which is typical during a project of this size, scope and duration. It is true that HCA staff had no prior Cerner experience; however, this is typical for new system implementations. The contractors in conjunction with Cerner's team provided the knowledge to support, bringing HCA staff up to speed during the project. Additionally, staff knowledge did not prevent the identification of bug fixes until after go-live, as Cerner controlled the system build from inception until after the go-live was complete, such that HCA staffing had no involvement in bug fixes until 2 weeks after go-live.

Throughout the EHR Implementation in 2012 and 2013, the required ITC quarterly Project Status Reports indicated the following concerns (without quantitative supporting backup): The project experienced delays with the design of a few modules due to lack of personnel . allocations. Additional staff would have been needed to make up the lost time. Delays in approval for additional staff impacted the ability to meet milestones for the design phase. Response: Partially Disagree. The statement infers no corrective actions were considered or taken. (See also response to FA-29.) All large projects experience variations in individual task schedules. This is acceptable as long as the task is not on the critical path. One of the features of Cerner's Event Driven Methodology is that each team essentially has its own schedule managed by the Cerner Solution Architects. All of the solution design schedules had built in slack and therefore could tolerate some delays. In cases where concerns were registered by the project manager (and noted in the status report), additional resources, typically Subject Matter Experts (SMEs) from the agency, temporarily worked more hours on the project to ultimately allow the design to be completed in time and the project to go-live on schedule.

Neither ITC nor VCHCA took corrective action regarding the risks resulting from staff shortages and the related consequences as documented in the quarterly ITC reports. Response: Disagree. The HCA's Cerner project manager reported to the HCA Cerner Steering Committee biweekly on staffing status from the beginning. Members of the Steering Committee assisted the project manager by identifying needs and allocating staff as needed. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks Staff resource constraints were noted very early in the project and this was the primary reason, among others, that HCA hired Novacoast contract staff to expedite the hiring of resources, as opposed to going through a lengthy internal recruitment process. These resource constraints were subsequently addressed and the HCA's actions in this regard allowed the project to go-live on schedule.

VCHCA did not perform simulated or actual load testing before Go-Live. Testing could possibly have exposed storage capacity limitations, response time problems, and other limitations in the EHR system. Response: Disagree. The HCA did conduct actual load testing with over 100 concurrent users on all week days from mid-April through June, occurring in 9 training centers concurrently across the agency. There were no significant performance issues noted. In this regard, the performance testing took into account the actual workflows and loads expected to occur in production.

Beginning at Go-Live on July I, 2013, and for several weeks thereafter, much of the staff had difficulty logging into the EHR system to access patient records. To overcome this situation VCHCA had to rapidly purchase and install an additional 600 Citrix licenses and triple the number of servers in the server farm by July 30, 2013. VCHCA acknowledged this situation was a direct result of underestimating the number of simultaneous users at 600. Response: Partially Disagree. While we agree there were issues with logging into the system, this was a licensing issue and not an issue with system capacity. Also the contract called for 1,000 concurrent users, not 600. The servers were added to address system response time as opposed to issues associated with logging into the system.

After adding the 600 Citrix licenses and tripling the servers in July 2013, a new problem became apparent and lingered until VCHCA abandoned its Ventura server farm and switched to Cerner Remote Hosting (RHO) in April 2015. The new problem was that the "Order Entry" window response time, initially several minutes, was intolerable for most users. One of the causes was system design limitations in the server farm (e.g., the Storage Area Network (SAN) did not have enough ports) due to VCHCA's underestimating the number of simultaneous users. Response: Partially Disagree. Cerner delivered the solution sized to handle up to 1200 concurrent users. Based on an analysis by the HCA IT, Cerner and Hewlett Packard (providers of the SAN), the way in which the SAN connections were configured by HP/Cerner, as opposed to the underestimation of the number of system users, did not support database access as the database grew and expanded across the SAN. This is why the issue was not apparent until months after go live. Because HCA IT was responsible at this time for system performance and tuning, they worked with HP to conduct a reconfiguration of the SAN to allow improved access to the database and reduce response times until the system could be migrated to Remote Hosting.

Both VCHCA's and Cerner's system administrators managed to speed up response time slightly while the EHR was still hosted in Ventura by adjusting system software parameters. However, they were never able to get response time to acceptable levels. VCHCA decided not to pursue further hardware upgrades to the server farm in Ventura. Instead servers and server support were switched to Kansas City by purchasing Cerner's RHO option. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks Response: Disagree. The rationale for moving from the HCA hosted system to the Cerner remote hosted system (RHO) were documented thoroughly in the Cerner Remote Hosting Board Letter, presented at the June 24, 2014 Board of Supervisors meeting. The main reasons included substantial cost savings, improved reliability, and automatic hardware refreshes.

For six months after Go-Live, there were occasional planned and unplanned downtimes when the EHR network would be unavailable. During such intervals clinical staff had to temporarily revert to paper recordkeeping and then enter the paper information into the EHR when it came back online. Response: Partially Disagree. System outages are anticipated in new system implementations and procedures were in place for these events. All downtimes follow the HCA's standard downtime procedure, which includes the use of paper documentation from the agency's downtime procedures binder. The downtime procedures are in line with all other healthcare agencies nationwide. When recovering from a downtime, critical clinical (medications given, orders) and financial (charges) information is quickly loaded into the EHR while non-critical information that was charted is sent to medical records for scanning into the EHR to become part of the medical record.

To protect against an outage of the EHR, Cerner has the capability to periodically backup patient records (e.g., medication prescribed/ administered, lab results) "locally" in the hospitals, independent of the central EHR server farm. These backup "724 systems" are read-only to be used for retrieval of recent patient records during a system outage. At Go-Live, these 724 systems had not been configured and activated. After the Go-Live date, over a period of several months, thirty 724 systems were deployed by IT at strategic locations throughout the hospitals. Response: Agree. The 724 system was not completely built at go-live and the HCA determined the 724 backup system could be delayed. The decision to delay was based on the fact that the agency had been running on a paper system for decades and the possible need to revert to paper for short periods of time was part of well-established procedures (see response to previous fact) and would be a reasonable mitigation plan until the 724 system was available.

The Wi-Fi network at the Ventura County Medical Center was not adequately assessed and tested before Go-Live. The network experienced intermittent problems beginning at Go-Live and for several months thereafter. This condition interfered with staff productivity and led to frustration. Response: Disagree. The wireless system was designed based on Cerner best practices using an industry accepted modeling tool. The system was installed and tested for coverage and signal strength. There were no observed issues. While there were intermittent complaints of performance issues, which were attributed to the wireless system, both network based performance monitoring tools and network staff deployed to the specific areas observed excellent wireless performance at almost every area. One area, 3-North, did seem to have wireless issues. After a detailed analysis by the IT Services, it was determined that area was being exposed to some form of electromagnetic interference on the G and N wireless bands. A decision was made to move the entire FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks system to the A band for Cerner because it had more resistance to the particular form of interference observed. This immediately led to a resolution of the issues at 3-North.

VCHCA personnel discovered that the standard Cerner-formatted prescription label did not contain all the content/dosage information that the compounding pharmacist and administering nurse needed. This deficiency and many other issues considered high priority by hospital staff were duly reported to the Help Desk and to management as patient care issues. The Pharmacy label format issue was not resolved for nine months. Response: Partially Disagree. Shortly before go-live, the Pharmacy requested an additional label that was not part of the original scope. Cerner agreed to do the work at no additional cost. With all of the efforts around go-live, along with the substantial changes to the Pharmacy workflows, it did take several months to agree on the design changes, build the label, and test it. There were several issues reported to the help desk once the system went live. This is normal. There were no unresolved showstopper issues found by analysts, subject matter experts, other testing staff, those trained or management at go-live. Critical issues were addressed, following standard Help Desk protocol, which is to escalate up the organization until the issue is addressed.

Before hardware was ordered, focus groups were used to gauge end-user hardware preferences. At these sessions, selected staff got to view and touch a variety of end-user equipment, but the equipment was not tested in a live environment as it would be used in the hospital. Users did not have an opportunity to evaluate the hardware as it would be used in their normal work environment. For example, tablets were selected as a choice for nurses. But after Go-Live, nurses tried to use them for charting but found they were inappropriate for a variety of reasons (e.g., the charting area was too small with the current Cerner Solutions; the pop-up on- screen keyboard covered valuable chart area; battery life was only a couple of hours). The tablets had to be replaced with alternative hardware. In addition, the laptops with built-in scanners were focus group selected, but in practice with the EHR system they were impractical to use and had to be replaced. Response: Disagree. The tablets were never designed for charting. The tablets were selected for two reasons: to perform the medication administration function, and to collect patient vitals and other key health indicators. Neither of these require extensive keyboard use. There were no laptops with built-in scanners procured or put into operation. The resolution to the tablet concerns were to procure laptops with external, high quality scanners attached. These have worked well and are still in use today.

The purchase requisitions for end-user hardware needed to support the EHR Go-Live event were forwarded to VCHCA administration in December 2012 by the VCHCA IT organization. But the end-user hardware was not ordered until May-June 2013. Thus a significant amount of equipment was unavailable to be properly configured and in place for staff to use for check-out and refresher training in their work environment before Go-Live. Response: Disagree. It is acknowledged the equipment was ordered later than desired due to the HCA working on the best funding mechanism for the large order; however, all diam'r. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks of the equipment arrived and was deployed in time for implementation. All of the devices ordered and installed use the Windows operating system and the standard Cerner application. The only differences in the equipment was the lack of a physical key board on the tablets which is what made the documentation difficult. Therefore, all of the workstations in the HCA were capable of running Cerner and available to staff for checkout and refresher training. In fact, staff was encouraged to use their training access when returning to their work environment to become more familiar with Cerner before going live.

Due to inadequate planning, a significant number of workstations and tablets had to be ordered after Go-Live. Response: Disagree. A significant amount of planning was conducted by super users, clinical management, and IT to ensure sufficient and appropriate equipment was ordered. All of the equipment requested was ordered and installed prior to implementation. After the system had been used for several weeks, clinical management re-evaluated the workflows based on their experiences and allocation of equipment. Additional equipment was then ordered and deployed as requested.

At Go-Live, many of the computer printer assignments were incorrectly configured by IT technicians. Printouts were directed to out-of-area printers that potentially exposed critical data until the default destination printer was located and the printout picked up by the requester. It took many weeks to get all associated printer problems fixed. Response: Partially Disagree. The HCA IT had implemented the printing configurations based on their understanding of how the Cerner system handles printing. There were some prints misdirected but all misdirected prints were still on premises and in staff controlled areas. Staff was notified of the situation and told to place unrequested prints into secure shredding containers. Printers defined incorrectly were fixed with the first 30 days.

There were EHR Implementation related concerns regarding potential risks due to a variety of factors. Issues of concern included: Due to the frequent early EHR instability, staff had to temporarily administer medical care without access to recent patient records; they had to fall back to handwritten paper recordkeeping; and then, retroactively, update the EHR when it became accessible again. Saturation of EHR login capacity led to frequent staff login failure attempts, a condition that went unresolved during the first several weeks after Go-Live. Frequent crashes of the EHR during first 6 months after Go-Live Incomplete/inadequate/inconsistent data entry windows, order sentences, and pick-list choices used by physicians, nurses, pharmacists, and other healthcare staff to select from in the various Solution charts Sluggish response times for users launching/updating Solution window displays Printer queues (particularly label printers used by the Pharmacy and Labs) frequently stalled and stopped printing labels. Pharmacy staff had to resort to handwriting labels— usually for several hours. On third shift or weekends, IT support was not readily available to fix the blockage. The handwritten labels used to work around EHR outages precluded the automatic checks normally performed by the EHR when verifying correct medication/patient administration. This situation was not resolved for over nine months after Go-Live when IT reconfigured printer servers in the server farm. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks Response: Partially Disagree. As there are in any large system implementation there were issues; however, clinicians performed workarounds in a manner consistent with their training and their long held practices of providing healthcare prior to implementation of the EHR. Periods of system instability and downtimes were addressed using industry standard, paper-based downtime procedures. Login saturation was addressed by correcting the Citrix licensing and the number of Citrix servers as previous discussed. Frequent crashes were addressed by progressive and extensive tuning of the system following go-live, with further improvements realized by eventually moving to Cerner's Remote Hosting Option. Input areas of the system were designed by the HCA subject matter experts. Like all 8 design efforts, not all will agree with the way something was designed. When testing was completed, the project team instituted a change control process which is still in use today. If a change is requested, regardless of the source, a larger group of clinical and IT staff review it to ensure it is the best change for all parties. Sluggish response times were addressed progressive and extensive tuning of the system following go-live. Although not the primary reason for migrating to Cerner Remote Hosting Option (RHO), further improvements realized by eventually moving to RHO. The pharmacy label printing issue was intermittent and difficult to troubleshoot. The HCA IT and Cerner were eventually able to trap it and a solution was found.

While there are no reported incidents of harm to patients because of EHR problems, there are documented occasions that potentially could have put patients in danger if alert clinical staff had not taken corrective actions with workarounds. Response: Disagree. There were system outages and performance issues, however HCA has highly trained, experienced, licensed and certified staff that know how to take care of serious patient care issues. Clinical staff have been continually trained, (long before Cerner), to care for patients in the absence of automation, and HCA continues to balance the merits of improvements in automation such that the system never completely takes over for the clinician's expertise. The EHR system is a tool to assist the clinician in providing patient care. One of the challenges of introducing an EHR system is balancing the skills of the clinical staff with the value of the automation.

During the EHR Implementation, the communication paths within VCHCA's organizational structure became ambiguous. IT problems involving patient care tended to be reported to IT personnel and may not have reached clinical management. Response: Disagree. Throughout the project and especially at the time of go-live all analysts and subject matter experts responsible for clinical solutions reported to a clinical manager. Every known concern for patient care was reported to a clinical manager or one of the analysts who then reported it to their clinical manager. The manager made sure their executive member was kept informed.

The user training did not include competency testing before Go-Live. It was also noted that training did not satisfactorily address learning retention losses with timely hands-on refresher courses using an EHR domain and more robust training materials. Nor did it adequately stress FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks the importance of accuracy using discipline-specific examples of correct vs. incorrect situations (e.g., data entry accuracy). Response: Disagree. Competency testing was included in the nursing training program. Competency testing is rarely used as part of training for new technology systems. Typically the student attends a hands-on class (or classes) where they follow actual workflows. Since technology systems are usually unique to the specific business, the training is always provided by the staff involved with building the system. This provides the best domain knowledge but the fact these individuals have not been professional trainers can impact the quality of the training. Every person being trained received discipline specific examples of correct data entry. It became clear very shortly after go-live that a large portion of the staff needed refresher training. Clinic and registration staff were provided 4 hours every week for the first 2-3 months for help in registration and charges. Nurses attended regular unit huddles that were developed to focus on nursing workflows.

Immediately after EHR Go-Live, many of the VCHCA staff were not comfortable using the system in spite of the training opportunities that had been provided and the availability of experts to help. Many users were confused and frustrated-a situation that was compounded by unplanned system downtime, slow window response time, and frequent failure of login attempts. Response: Agree. All new systems experience some level of discomfort and frustration, normally proportional to the level of change associated with the new system. This was the largest automation change the HCA has experienced, moving from a primarily paper- based medical care system to one that is fully automated and integrated across the entire patient care process. A similar situation was encountered in the County's public safety departments during the implementation of the Ventura County Integrated Justice System (VCIJIS), when more than a thousand staff moved from what was also a primarily paper- based process to one that was fully automated across the entire justice lifecycle. As the EHR system approached implementation and continuing through the present, the HCA has been in contact repeatedly with other larger health care institutions. Virtually every institution had some of the same issues the HCA encountered. Performance issues were encountered regardless of whether or not the institution hosted it themselves or outsourced. Those that outsourced it had fewer issues after the initial startup. Those that had an existing older EHR and hence were already mostly automated had less staff concerns during implementation than those that had to migrate to a higher level of automated documentation. New workflows and concerns regarding adequate training to support those workflow appeared at every institution regardless of the amount of training effort provided and amount of communication of change sent to staff.

Many factors contributed to patient billing problems associated with the EHR: Some users did not consistently enter data correctly into Solution windows, which ultimately led to downstream uncollectable patient billing. FY 14-15 Grand Jury Report Response: Electronic Health Care Record Implementation Risks Beginning with Go-Live, much of the patient information used for billing by the EHR was in not accurate. Many bills produced from the EHR were rejected by the "Scrubber" checking process and simply set aside to be looked at later for diagnosis and correction. By second quarter 2014, the backlog of unresolved billing produced by the EHR was 9 to 10 months behind, due to rejected claims having incorrect/inconsistent/missing data on patient billing. o After a deep-dive analysis by VCHCA, the rejected claims were found to be due to a variety of problems, most notable being data entry issues such as: Ineffectual training Lack of attention by staff entering patient and treatment data into the EHR Lack of proper supervisory oversight Response: Partially Disagree. The new EHR and its attendant Patient Accounting module were a complete replacement both in terms of the technology and the workflows. Revenue Cycle is the term that applies to the entire business process. In the previous system, the workflow allowed for mistakes on the front end (registration, assessment, orders, etc.) to be made and these were corrected at the end of the process by Patient Accounting. The new, Cerner based workflows required the data to be entered correctly at the proper point and if any errors were made, the corrections were made at the original data entry point, not corrected downstream. Therefore: Some users did not perform as expected at the start. This is not unusual in a large system m with thousands of users. Some learn faster and in different ways. Follow up training is expected. The majority of patient information used for billing was accurate but there were errors in a . significant portion of it due to the just described individual performance/training issues. The primary purpose of the Scrubber entity is to find these kind of errors and set them aside in an error queue for the providing company to review, correct and resubmit. The Scrubber error queue did increase in depth after the implementation, which was expected. The error queue is currently about the same level it was at pre go-live. There were sufficient numbers of supervisors for the numbers of staff involved; however 8 the supervisors were subject to the same training curve and learning issues the rest of the staff had. As the supervisors' knowledge and experience have increased, the quality of supervision related to the new system has increased. Although there were charges and bills that remained in the "scrubber" backlog for 9 to 10 months, other charges and bills were successfully being produced and sent out during this period and as previously noted, the "scrubber" queue is currently at about the same level as it was prior to Cerner implementation. Current billings overall are significantly higher than pre-Cerner implementation, $122 million for a recent six month period as opposed to $91 million in the six month period prior to Cerner system implementation.

FINDINGS I (we) agree with the FINDINGS numbered: FI-01. = I (we) disagree wholly or partially with the FINDINGS numbered: FI-02, FI-03, FI-04, FI-05, -