WASHINGTON, DC - The House Energy and Commerce Committee today released new documents uncovered in its investigation of the botched rollout of the president’s health care law, highlighting serious concerns among top administration IT officials in the week before launch. The documents are from September 26 to September 30.
Documents released earlier this week indicated that the White House and top administration officials, including HHS Secretary Kathleen Sebelius, were told in late March and early April that meeting the October 1 launch date presented serious challenges. White House Press Secretary Jay Carney confirmed that President Obama had been briefed on this report, prepared by McKinsey & Co. But on Nov. 14, when asked about the website’s ongoing failures, President Obama said, “We would not have rolled out something knowing very well that it wasn’t going to work the way it was supposed, given all the scrutiny that we knew was going to be on the website." The email traffic days before launch between the administration’s top IT officials reveal the chaos behind the scenes make it clear that the system was not on track to work the way tech officials hoped or intended. Yet the launch proceeded as scheduled.
Chairman Fred Upton (R-MI) commented, “For over six weeks, we have been asking the administration if they did not know or did not disclose the problems that lay ahead. What we are learning is that the administration went out of its way to hide the chaos behind the scenes. We now know the president was made aware of McKinsey’s warnings in the spring, and the situation did not seem to improve through launch and beyond. Any American business with facts like these, from Amazon to Apple, to local small businesses, would have had enough sense to delay launch and get it right the first time. Instead, the administration rolled the dice and Americans are left to struggle with the broken website and the president’s broken promises. The frenzied lead up to Oct. 1, coupled with the inadequate testing and numerous systems failures, reveal an administration that was not up to the job despite over three years to prepare."
Oversight and Investigations Subcommittee Chairman Tim Murphy (R-PA) added, “Fully informed that HealthCare.gov was doomed to fail, the White House decision to move forward anyway was incredibly insensitive to the worries and anxieties of five million Americans who’ve lost their health insurance and have no easy way to get it back."
Key findings from the documents follow (emphasis added):
On Thursday, Sept. 26, Akhtar Zaman from CMS Office of Information Services emailed several CMS officials, including Henry Chao and Mark Oh, “We ran several Performance & Stress Test (PST) cycles (starting from 10 Vuser to 2000 Vuser) since last 3 days but the results are not good and not consistent at all. … Later in the Wednesday evening, we ran two more cycles of testing. We attempted two cycles at 2,000 Vusers. Neither was successful." After receiving this email, Henry Chao sent a warning to CMS staff stating, “I DO NOT WANT A REPEAT OF WHAT HAPPENED NEAR THE END OF DECEMBER 2005 WHERE MEDICARE.GOV HAD A MELTDOWN (THIS IS TO GET YOUR ATTENTION IF I DIDN’T HAVE IT ALREADY." (Doc # QSSI-ECC-0000023064 and QSSI-ECC-0000023065)
On Friday, Sept. 27, the Acting Director of the CMS Office of Enterprise Management David Nelson emailed Henry Chao to say, “The scripts are failing so far due to issues like load balancing, inefficient and defective code, and inefficient queries. We have not been successful in moving beyond 500 concurrent users filling applications without income verification." He goes on, “I realize everything is a priority but…the facts are that we have not successfully handed more than 500 concurrent users filling out applications in an environment that is similarly in size to Day 1 production. We cannot proactively find or replicate actual production capacity problems without an appropriately sized operational performance testing environment. And, we have not even started looking for tuning issues in the plan select and enroll parts of the application." (Doc # QSSI-ECC-0000036602 and Doc # QSSI-ECC-0000036603)
On Sunday, Sept. 29, the White House’s Chief Technology Officer, Todd Park emailed Henry Chao and Michelle Snyder, indicating he had no idea whether or when performance testing would occur. “Has the team run performance/diagnostic testing on the whole FFM … Are we going to run performance testing today and tomorrow on the growing aggregate collection of hardware … Are we testing to make sure that incoming traffic gets properly load balanced across the VMs/units? … What happens after the 16,000/60,000-70,000 threshold is reached? Is there gradual degradation of response time for users? Rapid degradation? Immediate crashing?" Park concludes, “Massive kudos again for the incredible progress the team is making!" (Doc # QSSI-ECC-0000066582)
Chao forwarded these questions to CMS staff asking, “Can you do this starting today (you are doing much of it already) and making some adjustment in how you lead this effort so that by tomorrow midday we have some semblance of a final picture of performance." (Doc # QSSI-ECC-0000066581)
“Tomorrow midday" would be just hours before HealthCare.gov was scheduled to launch.
Akhtar Zaman at CMS ultimately responded, “We’ll be continue to test today, tomorrow & beyond until we reach at least 10k concurrent users, and eventually 50k. So far, we only covered the Individual application in PST, and today we’ll be focusing on scripting Plan results, Plan compare, and enrollments. Also we’ll be performance testing FFM-EIDM integration today." (Doc # QSSI-ECC-0000066581)
NOTE: EIDM means Enterprise Identity Management. This is part of the account verification tool used when creating an account on HealthCare.gov.
A separate email chain resulted from the initial Todd Park September 29 questions also added, “Our plan is to have 54k applications/hr completed." (Doc # QSSI-ECC-0000058360)
Read the newly available documents online here.
