Thomas Dinsmore's Blog

A little over a year ago, on February 17, 2017, the Houston Chronicle reported that the University of Texas’ MD Anderson Cancer Center had halted an AI project for cancer diagnostics. The story revealed that MD Anderson spent $62 million over four years to build a system called the Oncology Expert Advisor (OEA), based on IBM Watson. As envisioned by its champions, OEA would help community oncologists provide quality care to patients unable to seek treatment directly from MD Anderson physicians.

A cascade of stories about the failed project ensued: in Forbes, The Wall Street Journal, MIT Technology Review, Medscape, The Cancer Letter, Health News Review, Health IT and CIO Review, ArsTechnica, and many others. Four themes emerged from the reporting:

(1) OEA was a poster child for bad project management.

An audit report published in November 2016 by the University of Texas Audit Office identified numerous exceptions to standard project management practice. According to the Audit Office, project leadership:

• Did not use proper contracting and procurement procedures
• Failed to follow IT Governance processes for project approval
• Did not effectively monitor vendor contract delivery
• Overspent pledged donor funds by $12 million

Other than that, the project was well-managed. 🙂

In a response to the audit report, project leader Lynda Chin argued that she was not required to follow IT Governance policies because the effort was a “research” project. This strikes me as a silly and self-serving argument. If you’re spending $62 million on a project intended for clinical use, you need to practice good project management. Calling the project “research” does not absolve you of that responsibility.

(2) Scope changes inflated project costs.

MD Anderson signed agreements with IBM and PwC in June and July 2012, respectively. Under these initial agreements, the scope of OEA included lower risk myelodysplastic syndrome (MDS) leukemia patients. The system would digest a broad range of whole exome, tissue, and other clinical data, produce new insight, and deliver physician decision-support services. The budget: just under $5 million.

Beginning in early 2013, MD Anderson radically expanded the scope of the project:

— Diseases added to OEA’s diagnostic capabilities: $23 million.

— Onboarding two partners to pilot the system: $29 million.

— Additional data sources: $5 million. 

Over the four-year life of the project, MD Anderson executed 7 agreements and 8 amendments with IBM and PwC. The auditors note that the contract value for many of these agreements was just below the threshold for Board approval, which suggests deliberate structuring to avoid scrutiny.

Interestingly, the massive expansion in project scope coincides with a $50 million pledge from “billionaire party boy” Low Taek Jho.

(3) OEA is not integrated with MD Anderson’s electronic health records (EHR) system.

IBM and its partners integrated the system with data from ClinicStation, the EHR system MD Anderson used previously. However, MD Anderson now uses Epic Systems for EHR; without live updates, OEA is unavailable for clinical use.

So, for $62 million, IBM and its partners built a custom demo.

(4) MD Anderson could not sell the system to partner hospitals.

MD Anderson planned from the beginning to use OEA as a way to provide high-quality cancer diagnostics to patients unable to seek treatment with MD Anderson physicians. Hence, business success of the project depended on MD Anderson’s ability to forge agreements with healthcare partners who would use the system. It was unable to do so.

Project leader Lynda Chin told the audit team that several factors prevented piloting with external partners, including “time needed for compliance and information security reviews of the cloud-based data repository,” and “lack of engagement or interest by network partners.” That’s two very different reasons. The former implies bad project estimating, poor delivery, or both; the latter implies an inability to sell the system.

---

Reports and analysis in the press raise as many questions as they answer.

Q. Why can’t Watson connect to MD Anderson’s EHR system?

Epic Systems is the leading EHR provider. A tool for medical diagnostics that cannot integrate with Epic is a like a tool for optimizing logistics that cannot integrate with SAP.

MD Anderson began the search for a new EHR system in late 2012 and announced that it had selected Epic in early 2013. IBM and its partners knew that OEA would have to integrate with Epic before it could go into production. Moreover, they knew this very early in the OEA development cycle.

IBM announced a partnership with Epic in 2015. Interestingly, MD Anderson is not among the 14 collaborating cancer centers.

Integrating Watson with Epic Systems, MD Anderson’s current EHR system, may be easy or it may be hard. It does not matter. It was a necessary step for OEA to go into production. IBM and PwC knew this.

Yet, they kept building on. Like an AWS commercial, but without brains.

Q. Why did MD Anderson contract the project piecemeal?

MD Anderson knew from the beginning that this project would cost a lot more to deliver than the $16 million budget approved by the Board in early 2013. Otherwise, why solicit a restricted gift of $50 million? Or are we expected to believe that Jho Low just happened to come up with that number by chance?

“Thank you for your interest, Mr. Jho. Our budget for OEA is $16 million.”

“Great! Here’s a check for $50 million.”

Moreover, MD Anderson also knew from the beginning that piloting OEA with partners was critical to success. So why wasn’t this task built into the original project plan? Expanding scope to cover this task nearly doubled the project budget.

There can be good reasons to contract a project in phases. It may be difficult to accurately estimate the cost of later phases before early phases are complete. Contracting serially keeps vendors “honest” and introduces the potential for competition in later phases.

Of course, MD Anderson did not keep IBM and PwC “honest.” No vendor other than IBM and PwC performed work on this project. The cancer center awarded $51.4 million in contract fees to the two vendors under non-competitive procurement. Moreover, per the audit report, it appears that MD Anderson paid IBM and PwC for work they did not do.

Q. Why couldn’t MD Anderson secure partners for the project?

IBM wants us to believe that Watson worked well and that OEA would be in use today if MD Anderson chose to continue the project. If that’s true, why couldn’t MD Anderson interest partners in piloting the system?

OEA may be the greatest breakthrough in medicine since the discovery of penicillin. There’s only one problem: nobody wants it.

Hello, sir, I just sunk a pile of money into this gold-plated veeblefetzer. Would you like to buy one?

IBM claims that OEA agrees with experts 90% of the time. That sounds impressive, but isn’t; for all we know, “community oncologists” perform as well or better.

For more than 90% of Super Bowl LII, the Eagles didn’t sack Tom Brady.

That 10% kills you every time.

A smart organization would gauge the market for partners before sinking money into OEA. Instead, MD Anderson built it, Field of Dreams style, and hoped that partners would come.

---

Here are a few closing thoughts and observations.

One failed project says little about a technology, product, or company.

Case in point: plenty of ERP projects went sidewise, sometimes with dire results. A botched ERP go-live in 1999 prevented Hershey from shipping $100 million in orders for inventory it had on hand. Despite this, firms continue to invest in ERP, for good reasons.

One failed project does not mean that Watson has no value, nor does it mean that IBM cannot successfully deliver solutions based on Watson. However, it highlights that Watson projects are high-risk IT projects. Customers must exercise good vendor and project management.

Much of the blame for this FAIL rests with MD Anderson.

OEA is a lock for the Pantheon of Bad Project Management. MD Anderson failed to practice competent vendor, contract, and project management. One can hardly blame IBM and PwC for feasting at the trough.

That said, are vendors responsible for customers’ bad project governance? The answer is an emphatic “yes” — as a matter of ethics, and as good business practice. Ethical vendors do not accept contracts that violate a customer’s procurement policies. They also do not initiate or continue projects that they know will fail to deliver the promised solution.

Don’t kid yourself. IBM and PwC knew their contracts violated MD Anderson’s procurement policies. Both vendors embed themselves deeply in organizations; they often know the customer’s policies better than the executives they serve.

They also knew that the project was a train wreck. They couldn’t possibly have not known.

Big expensive AI projects, like any other project, require a sound business case.

Do we still need to bang this drum? Apparently so. MD Anderson, it seems, thought it was smart to build OEA first and figure out a business case later. Oops.

Successful AI projects require solid data architecture.

AI without live data is worthless. Build your data platform first. That is all.

IBM’s claims about the project were <ahem> “aspirational.”

In early 2013, IBM announced in a press release that  MD Anderson “is using the IBM Watson cognitive computing system for its mission to eradicate cancer.”

It all depends on what the meaning of the word ‘is’ is.

Later that year, IBM planted a story in Scientific American, reporting that “M. D. Anderson Cancer Center is using Watson to help doctors match patients with clinical trials, observe and fine-tune treatment plans, and assess risks.”

There’s that pesky word “is” again.

In October 2014, IBM Watson Health CTO Rob High wrote that “Doctors at the MD Anderson Cancer Center in Houston are using Watson to drive a software tool called the Oncology Expert Advisor, which serves as both a live reference manual and a virtual expert advisor for practicing clinicians.”

IBM continued to speak “aspirationally” about the project after it was stone cold dead.

In September 2016, IBM ended work on OEA and declared it “not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited.” Two months later, IBM Watson Health’s Chief Health Officer Kyu Rhee touted Watson Health’s “collaboration with the world-leading MD Anderson Cancer Center in Houston, Texas. This project involves the rapid analysis of genomic information from cancer cells to provide personalized treatment for individuals.”

I guess Dr. Rhee didn’t get the memo.

The next time you hear IBM tout Watson, you may wonder if those claims, too, are “aspirational.”