It is funny that so many are quick to dismiss this because it is IBM, so "it must be just PR" (saw a few comments here). If Google did this or Elon Musk, everyone would be singing praises and musing about a better, brighter future.
Which is funny because it is Google / Facebook / Microsoft which collect and hold everyone's personal details and data. For better or for worse, IBM never seemed to have gotten into that business (yet?) of recording and selling every American's data for ads.
In general, I think healthcare is more important than self-driving Teslas or robots making us dinner and scratching our backs. So at least they seem to have focused on the right thing.
I think you are incorrect and the same cynicism would hold true if RMS and Linus themselves wrote the software. The frustration with this claim is that it's missing a lot of data and it's a claim without a lot of rather important backstory.
- Why was Watson being used for this? (That is, whose idea, whose plan, whose test)
- How many times was Watson wrong? (many others have asked this within the comments)
- Is there a statistical significance to Watson's diagnosis?
- How did Watson narrow it down? That is, did it provide a single conclusion of "it must be this form of leukemia" or was it just the most heavily weighted of several options?
The original ad-walled article from NDTV doesn't add a lot of information either on the specifics, though at least the siliconangle article is correction pointing out the limitations on this (needed a huge DNA repository to work with that would expose a lot of extremely private data, probably wouldn't work well for rarer ailments due to lack of data and inability to understand what it's actually looking at).
It is very good and I am happy that the lady was diagnosed correctly and promptly by Watson, and I do think this is important as something like Watson can cut down immensely on the grunt work that a lot of doctors go through. Already most jump on terminals in the patient room and pull up the relevant information from the hospital's internal libraries, so this just seems like the next natural step for modern healthcare; speed up those libraries.
But the cynicism comes from the lack of information, with how news like this gets portrayed (truthfully, I must credit to SiliconAngle for having a much more reserved approach to the article than the source they relied on), and IBM has been touting Watson as a miracle machine before, and they're certainly not curbing outrageous claims made by less careful publications and journalists.
That is the source of cynicism, and I really do think you'd find it as a result of just about any announcement like this from any person.
True, but I think it is sad that IBM uses the Watson name for everything related to AI, because this way they implicitly pretend to have found a unified approach to AI. I don't believe this is true. The AI doing the diagnosing is not the same AI as the one that played Jeopardy. But please correct me if I'm wrong.
As someone who's worked on/with teams building various Watson services/solutions over the past 3 years, I can confirm Watson is almost entirely a branding exercise without (from the engineering perspective) common leadership, coherent inter-team objectives, etc.
There isn't a 'unified approach' to anything here beyond the use of various open source machine learning libraries with huge amounts of medical data either licensed or acquired over the past 2 years.
Who exactly thinks its a good idea to hijack an existing brand for tangentially related, poorer quality products? Microsoft also does this with Skype and 'Skype' for business. Instead of a win-win scenario they hope for, they instead burn up the goodwill generated by the original product to bear the name. So instead of thinking "Watson, the amazing AI that won Jeorpardy", I now think "Watson, the horrible mishmash of IBM APIs"
IBM's primary sales channel is enterprise, where branding makes all the difference. Most of the time, the person authorizing the project has no idea on the technical details (and may or may not have solicited internal technical commentary).
These deals are honestly mostly about trust. Do they trust IBM / {vendor} has the expertise to deliver the project? Given that it's mostly bespoke integration, branding the entire IBM AI/ML area as "Watson" isn't too disingenuous.
The underlying algorithms may mostly be shared hence grouping them together as Watson just like Google uses Brain but I think it's more of a branding thing. It looks like they want to eventually make these systems work and run together to form a kind of"true AI"
You would see many of the same comments no matter who claimed some kind of first in medical diagnosis. Expert systems have been coming up with diagnoses doctors missed for decades.
It's a fat target for automation because much of the expertise is available in diagnostic protocols and few humans can hold all that information in their heads and keep it up to date. MYCIN only used a few hundred rules and it could keep track of which antibiotics to use better than MDs.
Yeah I do think the large data sets and processing power available to systems like Watson and Deepmind etc can and should improve detection of many diseases. More than 100k people die in the us alone due to hospital mistakes. 12million misdiagnosed a year is huge number
> IBM never seemed to have gotten into that business (yet?) of recording and selling every American's data for ads.
Maybe not Americans, but they did pretty well from doing it for Europerans from 1936-1945. [1] and rather than ads, the Hollerith cards IBM supplied by the million tabulated the census data from 1930s Europe into Jew / Not Jew rather nicely.
When the death camps were liberated there was a special unit assigned to makes sure the leased Hollerith machines were returned to the USA safely.
Thomas J. Watson was one of four Americans to receive the Order of the German Eagle [2]
This is an anecdote and could only be meaningful as statistics. Landing a booster stage once is an accomplishment, you can't guess you way to a stable landing. Making one diagnosis is not news. Especially since we can be sure they've tried to make many, and somehow they're only promoting the result of one.
Except, as person who doesn't plan on flying to space soon, but who will at some point in life need a good medical diagnosis, for purely selfish reasons, I would rather want the medical diagnosis to get more attention and improvement.
Google has a specific division that is looking into life sciences and health care. Recently they made deal with British health service nhs for processing anonymised eye scans of millions of people using deepmind so that they are better able to detect degenerative blindness.
I can't get past the adblocker spam, but I'm going to guess this woman had APL (acute promyelocytic leukemia, caused almost exclusively by a very specific chromosomal translocation) and the doctors didn't think to look for Auer rods and thus did not try FISHing for the translocation. APL is deadly (5% survival) if treated incorrectly but curable (95%) if treated correctly. However, something or other about that differential has been a standard question on the hematology medical boards for, what, 3 decades now?
I'm curious whether this case was confounded by cytogenetic complexity, which was part of the problem in that case. Ley later diagnosed Lukas Wartman, a fellow in his lab (or, more like, they worked together to figure out what to do), too:
Lukas did not look nearly as healthy as he does in that picture when I saw him last. He is a great scientist and what he had (adult acute lymphoblastic leukemia, ALL) is a nasty malignancy to treat. Children with ALL tend to do well, but their disease typically seems to arise from different underlying causes than adults, who do poorly.
(Not-so-ninja edit: Here is Lukas' own writeup of his experience. He was in a much tighter spot than I recalled: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4850894/
Even children in second relapse are viewed as bad news. Adults in second relapse are generally considered dead men walking. Anyways, read the paper if you're curious.)
If Watson caught that the woman had (say) Ph+-like ALL and suggested the right TKI, that will impress me quite a bit, because I've seen someone die from a wrong guess on the latter. On the other hand, if it was APL, they fucked up the differential somehow and are probably kicking themselves.
(Double not-so-ninja edit: It wasn't APL-vs-AML, or ALL subtyping. Another poster kindly pointed out the primary source for this news, and something doesn't add up.shrug)
It would definitely be impressive, but it doesn't mean much without knowing how many times it mis-diagnosed. IBM is very good at PR and Watson is milked for every little bit so I'm always a bit suspicious about it in triumphant announcements like these.
After all, winning the lottery is a lot less impressive if you've bought a few million tickets and this article says absolutely nothing about any kind of controls or whether or not this was a one-off performance or if it would work at scale.
They're definitely not free; doctors have to evaluate what Watson says, and if Watson is wrong a significant fraction of the time, then doctors will spend a lot of time and energy barking up the wrong tree.
Eh, it's a tool. Some are useful, some less so. I imagine this would be a tier two kind of thing. It's an unusual case, what suggestions does the robot have? Maybe something obscure and easy to forget.
Doctors have to evaluate what patients say, too, and they're forgetful or clueless at least as often as anyone else. If the cost-benefit analysis favors having Watson (or whatever) around, great, if not, great, just another decision to make.
>nothing about any kind of controls or whether or not this was a one-off performance or if it would work at scale.
I thought the 20 million oncology studies Watson cross referenced was the control and I arrived at the opposite conclusion as you about the usefulness at scale.
For example, I immediately thought that under the Affordable Care Act, the number of patient Electronic Health Records the Government collects eclipses what Watson had to work with in this case so at scale the potential is huge. Admittedly, until reading your comment I did not consider there were likely a number of times Watson misdiagnosed (which seems obvious now). Therefore, it would be interesting to rerun against the Electronic Health Records the Government has collected so far and test if Watson could diagnosis correctly in less attempts than with the 20 million record it used (or what I call the control). Though at that point, I think it would be the patient that is the control.
The "control" for the actual (machine) learning may be the test set derived from the available data, but the control of a medical study must be real life (new data) and not old data used for the machine learning algorithm.
You don't need 100% new data to get an idea of performance. It's meaningful (indeed, common) to select some arbitrary percentage of the real-world data as a "training set" then measure the performance of the algorithm on the rest. It's not proof against overfitting, of course, but it helps.
right, but there's a reason we run clinical trials before accepting that something is superior to standard of care. It is really, really hard to manipulate a preregistered clinical trial (not that people don't try) and incredibly easy to manipulate train/test/validate results.
> After all, winning the lottery is a lot less impressive if you've bought a few million tickets
I don't know, I'd still be impressed, wouldn't you? Buying a few million lottery tickets is impressive by itself, both logistically and financially. And even still, odds. It's a lot of risk for anybody to dangle.
Shit, buying every ticket is even more impressive. (I grant your point, you just posed a funny metaphor as comparison.)
I don't think buying every ticket is more impressive, it is pretty much the definition of a losing strategy.
Tests like these are all about the statistics and without statistics this result - while important to this particular individual - is almost meaningless.
That's pretty cynical of you. Perhaps you should give credit where credit is due instead of attributing it to just PR. Watson diagnosed an illness that was undetected by her doctors. This happens all of the time and people die because of this or live in misery because of it. If Watson can prevent even a fraction of the rampant misdiagnoses and aid in the actual determination of the illness then it's in our benefit to use the information it provides.
It's one thing to ask for additional information on how the test was conducted and another to automatically attribute it to a PR play that IBM likes to "milk every last drop of".
Its not cynical if you've actually had any dealings with IBM and Watson.
Edit: And when the hell did cynical become a bad word in silicon valley? I've had a number of posters on here respond to things "oh, stop being cynical!" as if that was some kind of contribution...
There's a difference between cynical and skeptical, from google:
"A skeptic doesn't believe anything without strong reasons, which is why it is also associated with doubt, especially when something hasn't been experienced yet. Cynicism is believing the worst of something or someone. It has nothing to do with evidence. It is an outlook on life."
If cynicism is believing the worst in something or someone, then believing that this article is just PR is by definition not cynical, since it is not the worst thing one could believe.
I'm sure the GP knows many things I don't. The only point of my post was the definition of the word cynical. It's probably fair to say that skepticism is an admirable trait while cynicism isn't. Feynman was a proponent of skepticism not cynicism.
The applicability of either moniker towards anyone in this debate, or any other, is up to the participants to hash out.
> to automatically attribute it to a PR play that IBM likes to "milk every last drop of".
No, that's history. IBM has quite a series of Watson 'breakthroughs' that ultimately went nowhere, to the point where when I see 'Watson' mentioned in an IBM press release it automatically gets discounted if there isn't a part where the subject has some statistics to go with the inevitable hype.
I see it as IBM trying to stay relevant in a world that needs it less and less and that's why they focus on anything that will grab headlines.
Winning jeopardy, curing cancer, what's not to like about IBM?
Very odd. To my knowledge, STAG2 deletions or mutations don't respond any differently to most other primary or secondary AMLs. (In fact, usually a secondary AML is a worse diagnosis as it means they have progressed from something like MDS, which itself has a shitty prognosis). Something is not quite right here.
One possibility is that Seishi Ogawa's group has figured out that something off-label can more effectively treat cohesin mutants and it is under review. But I'm not sure how Watson would have known about that. Interesting.
Probably my reading is flawed, but this doesn't make sense.
Somewhat interesting and related is that software that performs medical diagnoses better than doctors has existed since the 1970s (Mycin, an expert system). It seems like nowadays that this kind of solution can possibly be provided at scale at a consumer level (can process new medical knowledge by itself, be installed on commodity hardware), not an expert myself though so I don't know if totally true. What exciting times though.
"it proposed an acceptable therapy in about 69% of cases, which was better than the performance of infectious disease experts who were judged using the same criteria."
I see two important parts to this dialog now: 1) machine support for sifting through mountains of data, and 2) the increasing jeopardy of large corporations controlling data. For part 1, the dialog confirms what some of us already know: "Watson" is a branding exercise, and there is no overall coherent plan for strategic AI. But general open source components for literature-based discovery are being built. Part 2 is scarier, and as noted by two speakers from NSF last Friday (Peter Arzberger, Chaitan Baru), "data IS infrastructure," and so the increasing difficulty for public data sources to compete with Google, Yahoo, Facebook, Baidu, etc. is perhaps the more serious challenge for AI support of humans in the future. University research faculty can't tackle problems that require the de novo or nearly de novo creation of such data infrastructure, so we await public policy of funding agencies to clarify the construction and access to such. (Americans in this forum please note that you need to stop writing and thinking as if there is no other part of the planet ... I would be most grateful if Google brokered access to only American data ;-)
MYCIN was an early expert system that used artificial intelligence to identify bacteria causing severe infections, such as bacteremia and meningitis, and to recommend antibiotics
It and its ilk (expert systems, with hand-crafted rules) (it was the '70s) where commonly shown to outperform experts:
MYCIN was never actually used in practice but research indicated that it proposed an acceptable therapy in about 69% of cases, which was better than the performance of infectious disease experts who were judged using the same criteria.
Yet, we still don't have (flying cars) AI that can help doctors make better diagnoses- and note I'm by no means advocating replacing the experts with AI. That would open a whole other can of worms (what do you train your AI on when there's no more experts, because you replaced them all with AI?).
But- just having the tech doesn't auto-solve your problems as if by magic. You gotta beat dumb politics first.
Each generation of computer power enables new AI techniques. Expert systems require explicit distillation of "rules". That took a lot of human work. And they are considered brittle when they are outside their expertise. But expert systems could on the million times weaker computers of the 1980s.
Watson employes statistical searching of large knowledge bases. You dont have to explicity ferret out all the rules and relationships. Google Translate does this too. There is no preprogrammed language dictionary.
The next frontier is deep learning which requires powerful computing to operate in real time.
All these techniques have their limitations. No magic bullet yet.
I have a slightly unrelated question, but there may be medically trained individuals in this thread.
I'm a medical student, and it seems like most people in the field are rather cavalier when it comes to talking about the job outlook for physicians in most any specialty. Do you all think that some healthcare jobs will not be as vital in the next decade thanks to improvements in computing and AI?
Also a medical student with an interest in AI. Healthcare jobs that rely on visual recognition (dermatology, radiology, some pathology) are probably the most likely to benefit in the short term (see Enlitic). Presumably a lot of other jobs require advances in Natural Language Processing/Understanding, as one of the big problems in health is the mostly unstructured nature of the data.
It is also possible that many healthcare jobs are essentially AI-complete problems - in this scenario, subjective opinion is not really a reliable marker, but lots of AI specialists give around a 90% chance of human-level machine intelligence by 2070 (there's a table in Nick Bostrom's Superintelligence with the actual figures).
One correct diagnosis is simply not that significant, except for the woman concerned. The real news will be A/B testing on much larger data contrasting Watson's diagnoses with those of doctors sans Watson, and checking both against a ground-truth value, if that is possible. Newsworthy would be a marked bump in accuracy across many cases with Watson...
IBM's Watson scares me primarily because the source and data is not in the public domain. Something this powerful, and this rare should be given to humanity to better it.
That said, I'm unsure how doing something like that would provide the right incentives for the research needed to create the next breakthrough.
1. It is most likely that Watson is far less powerful than advertised. Given enough trials, a pigeon could probably detect some forms of cancer that a human expert can't. [1]
2. I too would prefer that more open-source work (and published research) would come out of Watson project than it has been the case so far. However, there's a devil's advocate point to be made that closedness encourages diversity of approaches and implementations. When a high-profile project becomes open-source (as recently happened with TensorFlow), it exerts a lot of pull on time/attention of other developers and researchers that could have been focused on trying out entirely different approaches.
The Quantum Fund by Soros and others, returned upwards of 30% per year for over three decades.
Steve Cohen averaged near 30% annual returns for two decades.
Buffett's investing track record is similarly off the charts.
And lastly, a monkey couldn't do what John Paulson (or Burry and Eisman) did with the 'greatest trade ever,' producing a radical outcome from an extremely intricate concentrated investment (some of which required them goading the opportunity into existence to begin with).
If you had said monkeys with darts can sometimes outperform the bottom half of traders on Wall Street, you might have been close.
As a research scientist at a face recognition company, Watson doesn't strike me as particularly powerful or rare. As mentioned in one of the other comments, IBM PR is pretty impressive though.
I'd say that is exactly why this matters... more automation would let the doctors do more research into new treatments, and let computers run the day to day algorithms (Symptom A gets Test B, followed by results C, which is labelled as diagnosis D, and treatment E.)
Having had years of struggle to get a diagnosis on my own problems, the most frustrating part of it all is that exact process, where you are just plugged into the machinery of our health care system, and run through to get a canned answer that needed no human judgment at all, just canned knowledge and a prescribed treatment plan.
So just my own opinion, but yes, please let us automate that portion of medicine, so the doctors can do more research and improve things.
You'd be surprised. Watson is nothing to write home about, it's an umbrella term for various disparate technologies (the chess version is not the same as the Jeopardy version, etc) and most of them are not that impressive in the first place.
It's not difficult to see how a computer - especially with a huge database - could do very well at differential diagnosis. It would require the correct inputs, but its advantages are very clear even with fairly simple algorithms.
Theranos hand-waved several huge problems with their approach by claiming proprietary magic. What they tried to do is probably impossible. So, big difference.
Well, IBM Watson will be surprised to find this little girl, Brittany Wenger built a " Global Neural Network Cloud Service for Breast Cancer " from her bedroom computer around 2012.
Which is funny because it is Google / Facebook / Microsoft which collect and hold everyone's personal details and data. For better or for worse, IBM never seemed to have gotten into that business (yet?) of recording and selling every American's data for ads.
In general, I think healthcare is more important than self-driving Teslas or robots making us dinner and scratching our backs. So at least they seem to have focused on the right thing.