Click to Enlarge
WASHINGTON, DC — Numerous reports have been issued and testimony given detailing the failures in VA’s Electronic Health Record Modernization project. VA officials and legislators have placed at least part of the blame on the original contract that VA signed with Cerner in 2018.
An Office of the Inspector General (OIG) report released last month not only reconfirms that there were significant weaknesses in the contract but details how those weaknesses led to dozens of incidents of VA staff being locked out of the system or the system going down entirely.
The report found that VA and Cerner did not have adequate controls in place to prevent system changes from causing major incidents. And, while stronger controls were put in place last year when VA renegotiated the contract with Oracle Health, which purchased Cerner in 2022, those controls could still be much stronger, investigators stated.
The EHR was first launched at the Mann-Grandstaff VAMC in Spokane, WA, in October 2020 and immediately began experiencing problems, some of which led to the system going offline for hours. Over the next several years, the system was rolled out to five other hospitals, many of which experienced similar issues.
The investigators audited 360 major performance incidents that occurred between launch and August 2022, while also reviewing incidents through March 2024. They eventually chose a sampling of 35 incidents to investigate further.
They found that the downtime created by those incidents was due largely in part to a lack of controls in the original contract. The most notable was a lack of configuration management controls, which are standard for federal agency IT systems. They require agencies to have processes in place so they can swap out hardware components, update software and make changes to their computer systems without causing major disruption to their users.
A lack of configuration controls in the contract accounted for 18 of those 35 incidents lasting a combined 65 hours and 49 minutes.
“On March 14, 2022, the Mann-Grandstaff VA Medical Center experienced incomplete functionality for 10 hours and four minutes,” the report explained. “This incident occurred because of an update that inadvertently corrupted some 870 users’ credentials and prevented them from accessing part of the EHR system.”
When the EHR goes down, this creates immediate risk to patient safety, since it impacts staff’s ability to provide care. Federal agencies are directed to make plans so, when the system does go down, there’s a risk-mitigation strategy in place. Investigators found that VA did not finalize the procedures clinicians should take when the system goes down until May 2024. Also, there are still gaps in provider outreach and training.
“Procedures still must be implemented and training provided,” the report noted. “Without these additional measures, [VA] facilities risk staff confusion about what to do and delays that would negatively affect patient care.”
The report also determined that VA needs a better backup system for providers to access patients’ medical records when the system goes down. The first option given to clinicians was the Joint Longitudinal Viewer (JLV). However, when the EHR system goes down, it cannot connect to the JLV, making it useless.
The second option was a read-only system provided by Oracle. That system only shows records from up to seven days prior. This can be extremely limiting, especially for outpatient facilities where patients might not have been seen within that time frame.
Investigators also found that there was a lack of controls in the contract that provide for continuous monitoring of the system. Five of the 35 incidents occurred because Oracle had not been monitoring the EHR, accounting for a total of 14 hours and 31 minutes of disruption. One such incident on Aug. 22, 2022 affected all five of the sites that were then nonoperational for 1 hour and 38 minutes.
How to Respond to Major Incidents?
The original contract also failed to include clear procedures for how VA and Oracle would respond to major incidents.
“According to the National Institute of Standards and Technology, the parties involved in operating a system should collaborate on developing incident response policy and procedures,” the report noted. “These controls should be applied consistently across the organization. However, VA and Oracle Health responded according to who owned the incident, each following their own guidance.”
That guidance included different criteria for prioritizing incidents. VA’s definition of a priority 1 incident was narrower than Oracle’s. This resulted in VA responding to fewer incidents than Oracle.
There was also a lack of a well-defined standard for how quickly VA or Oracle should respond to a major incident. From 2019 to 2021, VA guidance specified response times for VA-caused incidents depending on how serious they were. But in 2021, the department revised the guideline to specify an average notification and resolution time and applied it only to critical and high-priority incidents.
Because that average was calculated monthly, it effectively loosened the timeline standards for VA.
For the 28 incidents caused by Oracle that the audit encompassed, the contract established no clear timeliness goal stating that Oracle should respond in consultation with VA “immediately.”
“Given the inconsistencies and lack of clarity in the expectations for major incident response time, the audit team could not determine whether VA or Oracle Health complied with the stated procedure in most cases,” investigators said.
In May 2023, VA added requirements to its renegotiated contract with Oracle designed to fix some of these problems. Those included new monthly targets for how long the system should be free of incidents, an increase in target uptime for the system, and stronger financial incentives and deterrents for Oracle. VA also contracted with Oracle to provide a new read-only system for when the EHR goes down.
According to the OIG, VA and Oracle can do much more, including assessing what data is needed to accurately track major performance incidents and have Oracle contractually commit to real-time sharing of that data. VA should also focus more on a post-mortem of incidents, and work with Oracle to determine why they happened and how they can be prevented in the future.