{"id":335426,"date":"2017-03-10T01:51:16","date_gmt":"2017-03-09T17:51:16","guid":{"rendered":"http:\/\/techcrunch.com\/?p=1462658"},"modified":"2017-03-10T01:51:16","modified_gmt":"2017-03-09T17:51:16","slug":"deepmind-says-no-quick-fix-for-verifying-health-data-access","status":"publish","type":"post","link":"https:\/\/people.utm.my\/asmawisham\/deepmind-says-no-quick-fix-for-verifying-health-data-access\/","title":{"rendered":"DeepMind says no quick fix for verifying health data access"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=738\" class=\"\"\/><\/p>\n<p>Why should you trust an advertising giant with the\u00a0most sensitive personal\u00a0data you possess: aka your medical\u00a0records? That\u2019s the hugely sticky\u00a0issue\u00a0Google-owned DeepMind is facing\u00a0as it seeks\u00a0to embed itself\u00a0into the UK\u2019s healthcare space \u2014 a big push publicly announced in February last year.<\/p>\n<p>DeepMind is\u00a0now fleshing out in <a target=\"_blank\" href=\"https:\/\/deepmind.com\/blog\/trust-confidence-verifiable-data-audit\/\">a little more detail<\/a> how it hopes to futureproof patient trust in commercial\u00a0access to and monetization of their\u00a0health data, via a blog post that puts a little more meat on the bones of a plan for a technical audit infrastructure first discussed\u00a0last <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/11\/22\/patient-data-api-pivotal-to-deepminds-push-into-uks-nhs\/\">November<\/a>\u00a0\u2014 when DeepMind\u00a0also confirmed it was\u00a0building an\u00a0access infrastructure for\u00a0National Health Service (NHS) patient medical records. And the world-famous AI company is not sounding hugely\u00a0confident of being able to build a verifiable\u00a0audit system\u00a0for health data \u2014 move over <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/03\/17\/google-defeating-go-champion-shows-ai-can-find-solutions-humans-dont-see\/\">Alpha Go<\/a>! There\u2019s a new challenge for DeepMind to apply its collective wits\u00a0to.<\/p>\n<p>The end-game of DeepMind\u2019s NHS access infrastructure plan is for the company to\u00a0have ownership of a standard interface which\u00a0could be rolled out to other NHS Trusts, enabling both DeepMind\u00a0and third-party developers to more easily deliver apps into\u00a0the UK\u2019s healthcare system (but with DeepMind positioned to be able to charge other app-makers for access to the access API it\u2019s building, for example).<\/p>\n<p>On its own account, it\u00a0has said that where AI and health intersect its\u00a0future ambition is to be able to\u00a0<a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/09\/20\/deepmind-wants-its-healthcare-ai-to-charge-by-results-but-first-it-needs-your-data\/\">charge by results<\/a>. But in the meantime it needs patient data to train its AIs. And it\u2019s that scramble\u00a0for data that got DeepMind into early hot water last year.<\/p>\n<p>Since 2015, the company\u00a0has inked multiple agreements with UK NHS\u00a0Trusts to gain access to patient data for various purposes, <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/07\/05\/deepmind-partners-with-nhs-eye-hospital-to-conduct-ai-research\/\">some<\/a> but <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/11\/21\/deepmind-health-inks-new-deal-with-uks-nhs-to-deploy-streams-app-in-early-2017\/\">not all<\/a>\u00a0for AI research. The most wide-ranging of DeepMind\u2019s\u00a0NHS\u00a0data-sharing arrangements to date, with the Royal Free NHS Trust \u2014 to build an app wrapper for an NHS algorithm to identify acute kidney injury \u2014 caused <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/05\/04\/concerns-raised-over-broad-scope-of-deepmind-nhs-health-data-sharing-deal\/\">major controversy<\/a> when an\u00a0FOI request\u00a0revealed the scope of\u00a0identifiable patient data the company\u00a0was receiving. DeepMind and the Trust in question had not publicly detailed how much data was being shared.<\/p>\n<p>Patient consent in that instance is assumed (meaning patients are\u00a0not asked to consent), based on an interpretation of NHS healthcare data-sharing guidelines for so-called \u2018direct patient care\u2019 that\u00a0has been <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/11\/22\/patient-data-api-pivotal-to-deepminds-push-into-uks-nhs\/\">questioned<\/a> by\u00a0<a target=\"_blank\" href=\"https:\/\/www.ft.com\/content\/f6bcce6e-b099-11e6-9c37-5787335499a0\">data protection experts<\/a>,\u00a0and criticized by health data privacy advocacy group, <a target=\"_blank\" href=\"https:\/\/medconfidential.org\">MedConfidential<\/a>.<\/p>\n<p>The original DeepMind-Royal Free data-sharing arrangement (it\u2019s\u00a0<a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/11\/21\/deepmind-health-inks-new-deal-with-uks-nhs-to-deploy-streams-app-in-early-2017\/\">since been reinked<\/a>) also remains under investigation by the UK\u2019s national data protection agency, the ICO. And under review by the National Data Guardian, the government-appointee tasked with ensuring citizens\u2019 health data\u00a0is safeguarded and used properly.<\/p>\n<p>Despite the ongoing\u00a0probes, the\u00a0app DeepMind\u00a0built with London\u2019s Royal Free NHS Trust has been deployed in\u00a0the latter\u2019s\u00a0three hospitals.\u00a0So you could say the AI-company thinking about\u00a0a health data access audit infrastructure at this point in proceedings is\u00a0akin to a coach-driver talking about putting an as yet unconstructed cart on a horse that\u2019s already been released to run around the fields \u2014 while simultaneously asking those being saddled up\u00a0to trust it. (See also: DeepMind wasting no time\u00a0<a target=\"_blank\" href=\"http:\/\/www.wired.co.uk\/article\/deepmind-nhs-ai-kidney-royal-free\">PRing the apparent benefits<\/a> of the Streams app created\u00a0after it gained liberal access to Royal Free patients\u2019 medical\u00a0records.)<\/p>\n<p>The overarching issue\u00a0here is trust \u2014 trust that the sensitive healthcare data of patients is not being\u00a0shared without the proper authorizations, and\/or with patient consent. And that patients are not left in the dark about who is being allowed to access their personal information and for what purposes.<\/p>\n<p>DeepMind\u2019s answer to the trust issue \u2014 and the controversy caused by how it went about acquiring\u00a0NHS patient data in the first place \u2014 appears primarily to be\u00a0a technical one. Though building an audit infrastructure after you\u2019ve already gained access to data\u00a0does not satisfy legal or privacy experts. And such a topsy-turvy trust trajectory\u00a0may be unlikely to\u00a0impress\u00a0patients either. (Albeit DeepMind has also <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/09\/20\/deepmind-wants-its-healthcare-ai-to-charge-by-results-but-first-it-needs-your-data\/\">started engaging with patient groups<\/a>, even if only\u00a0after the controversy arose.)<\/p>\n<p>In a <a target=\"_blank\" href=\"https:\/\/deepmind.com\/blog\/trust-confidence-verifiable-data-audit\/\">blog post<\/a> entitled \u2018Trust, confidence and Verifiable Data Audit\u2019, DeepMind paints a picture of a technical audit infrastructure\u00a0that uses \u201cmathematical assurance\u201d and open source respectability to deliver \u201cverifiable\u201d data access audits that \u2014 it presumably\u00a0hopes \u2014 will euthanize\u00a0the trust issue, down the road. In a nearer time frame, its\u00a0hope looks to be trying to kick the can of scrutiny far\u00a0away from the \u2018trust us\u2019 reality of how it is currently utilizing\u00a0patient data (i.e. without a verifiable\u00a0technical infrastructure to prove its\u00a0claims, and while still under review by UK data protection bodies).<\/p>\n<p>The Google-owned AI company writes:<\/p>\n<blockquote readability=\"11\">\n<p>Imagine a service that could give mathematical assurance about what is happening with each individual piece of personal data, without possibility of falsification or omission. Imagine the ability for the inner workings of that system to be checked in real-time, to ensure that data is only being used as it should be. Imagine that the infrastructure powering this was freely available as open source, so any organisation in the world could implement their own version if they wanted to.<\/p>\n<\/blockquote>\n<p>Of course\u00a0that\u2019s just introductory <a target=\"_blank\" href=\"https:\/\/www.youtube.com\/watch?v=DVg2EJvvlF8\">mood music<\/a>. The meat of the post contains\u00a0few concrete assurances, beyond a\u00a0repeatedly stated conviction of how tough it will be for DeepMind to build a\u00a0\u201cVerifiable Data Audit for DeepMind Health\u201d, as it describes the planned audit infrastructure.<\/p>\n<p>This is \u201creally hard, and the toughest challenges are by no means the technical ones\u201d it writes \u2014 presumably an oblique reference to the fact that it needs to get buy-in from all the various healthcare and regulatory stakeholders. Ergo,\u00a0it needs to gain their trust in its approach. (Which in turn explains the\u00a0mood music, and the tight\u00a0PR game.)<\/p>\n<p>Timing and viability for the technical audit infrastructure also remain vague. So while, as noted above, the DeepMind-built Streams app is again in use in three London\u00a0hospitals, its\u00a0slated trust-building audit system has not yet even begun\u00a0to be constructed.<\/p>\n<p>And with the blog post replete with\u00a0warnings about the challenges\/difficulty of building\u00a0the hoped for infrastructure, the subtext sounds a lot like: \u2018NB, this may actually not be possible.\u2019<\/p>\n<p>\u201cOver the course of this year we\u2019ll be starting to build out Verifiable Data Audit for DeepMind Health,\u201d it writes early on. But by the end of the post it\u2019s talking about \u201choping to be able to implement the first pieces of this later this year\u201d \u2014 so it\u2019s shifted from \u201cstarting\u201d to \u201choping\u201d within the course of the same\u00a0blog post.<\/p>\n<p>We\u2019ve reached out to DeepMind to ask for clarity on its timeline for building the audit infrastructure\u00a0and will update this post with any response.<\/p>\n<p>In terms of additional details of how the\u00a0audit infrastructure might work, DeepMind says the aim\u00a0is to build on the existing data logs it creates when its systems interact with health data via\u00a0an append-only \u201cspecial digital ledger\u201d \u2014 not a decentralized blockchain (which it claims would be wasteful in terms of resources) but by a DeepMind-controlled ledger that has a tree-like structure, meaning new entries generate\u00a0a cryptographic hash that summarizes both\u00a0the latest entry and all of the previous values \u2014 with the idea being to make\u00a0entries tamper-proof, as the ledger grows. It says an entry would record: \u201cthe fact that a particular piece of data has been used, and also the reason why \u2014 for example, that blood test data was checked against the NHS national algorithm to detect possible acute kidney injury\u201d.<\/p>\n<p>Notably it does not specify whether\u00a0ledger entries will log when\u00a0patient data is being used\u00a0to train any AI models \u2014 something DeepMind and the Royal Free have <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/06\/08\/nhs-memo-details-googledeepminds-five-year-plan-to-bring-ai-to-healthcare\/\">previously said they aim to do<\/a> \u2014 nor whether an audit trail will be created of how patient data\u00a0changes AI models, i.e. to enable data inputs to be compared with patient outcomes and allow for some\u00a0algorithmic\u00a0accountability in future. (On that topic DeepMind, probably the world\u2019s most famous AI company, remains markedly\u00a0silent.)<\/p>\n<p>\u201cWe\u2019ll build a dedicated online interface that authorised staff at our partner hospitals can use to examine the audit trail of DeepMind Health\u2019s data use in real-time,\u201d it\u00a0writes instead. \u201cIt will allow continuous verification that our systems are working as they should, and enable our partners to easily query the ledger to check for particular types of data use. We\u2019d also like to enable our partners to run automated queries, effectively setting alarms that would be triggered if anything unusual took place. And, in time, we could even give our partners the option of allowing others to check our data processing, such as individual patients or patient groups.\u201d<\/p>\n<p>Looping patients into audits might sound nice and inclusive, but DeepMind goes on to caveat the difficulties of actually providing\u00a0any access for patient\u00a0groups\/individual\u00a0patients as one of the major technical challenges standing in the way of building the system \u2014 so again this is\u00a0best filed under \u2018mood music\u2019 at this nascent point.<\/p>\n<p>Discussing \u00a0the \u201cbig technical\u00a0challenges\u201d \u2014 as it sees it \u2014 the first problem DeepMind flags is being able to ensure that all access to data is logged by the ledger. Because, obviously, if the system\u00a0fails to capture any\u00a0data interactions the entire audit falls apart. So really that\u2019s not so much a \u201cchallenge\u201d as a massive\u00a0question-mark\u00a0about the feasibility of the entire endeavor.<\/p>\n<p>Yet on\u00a0this DeepMind merely\u00a0rather tentatively writes (emphasis mine):<\/p>\n<blockquote readability=\"12\">\n<p>As well as designing the logs to record the time, nature and purpose of any interaction with data, we\u2019d also <strong>like to be able to prove<\/strong> that there\u2019s no other software secretly interacting with data in the background. As well as logging every single data interaction in our ledger, we will also need to use formal methods as well as code and data centre audits by experts, to prove that every data access by every piece of software in the data centre is captured by these logs. We\u2019re also interested in efforts to guarantee the trustworthiness of the hardware on which these systems run \u2013 an active topic of computer science research!<\/p>\n<\/blockquote>\n<p>Frankly, I\u2019d argue\u00a0that\u00a0there being zero chance of\u00a0\u2018secret software\u2019 getting clandestine access to people\u2019s sensitive\u00a0medical records\u00a0would have to be a requirement, not an optional extra, for the proposed\u00a0audit system\u00a0to have a shred of credibility.<\/p>\n<p>Notably, DeepMind\u00a0also does not specify whether or not\u00a0the \u201cexperts\u201d it here envisages being needed to audit its data centers\/infrastructure would be independent of the company\u00a0itself. But obviously they would need to be \u2014 or again any audits the\u00a0system delivers would not be worth the paper they\u2019re\u00a0written on.<\/p>\n<p>We\u2019ve reached out to DeepMind with questions about its intentions vis-a-vis open sourcing the technical audit infrastructure, and again will update this post with any response.<\/p>\n<p>As with end-to-end encryption protocols, for example, it\u2019s clear that for any technical audit solution to be credible DeepMind would need to open it up entirely \u2014 publishing detailed whitepapers and fully open sourcing all components, as well as having expert outsiders perform a thorough\u00a0audit of its\u00a0operation (likely on an ongoing basis, as the infrastructure gets updated\/upgraded over time).<\/p>\n<p>Nothing short of a full open sourcing would be required. Remember:\u00a0this is a data processor <em>itself<\/em> proposing to build an audit system for the health data it is being granted access to by the data controller. So the conflict is very clear.<\/p>\n<p>DeepMind does not make that point, of course, rather\u00a0it concludes its\u00a0blog with a vague hope of getting help in realizing its vision\u00a0from any generally interested others. \u201cWe hope that by sharing our process and documenting our pitfalls openly, we\u2019ll be able to partner with and get feedback from as many people as possible, and increase the chances of this kind of infrastructure being used more widely one day, within healthcare and maybe even beyond.\u201d<\/p>\n<p>But if the company\u00a0really wants to inculcate\u00a0trust in its vision for overhauling healthcare\u00a0delivery it will need to make itself\u00a0and its processes a lot more transparent and accountable than they <a target=\"_blank\" href=\"https:\/\/techcrunch.com\/2016\/07\/20\/deepminds-first-nhs-health-app-faces-more-regulatory-bumps\/\">have been thus far<\/a>.<\/p>\n<p>For example, it could start by\u00a0answering questions\u00a0such as what is the legal basis for\u00a0DeepMind processing the sensitive data of healthy patients who will never go on to develop AKI?<\/p>\n<p>And why it\u00a0and the Royal Free did not\u00a0pursue a digital integration solution for the Streams app that\u00a0only pulls in a sub-set of data on patients who\u00a0might be vulnerable, rather than the\u00a0much broader swathe of medical records that are passed to DeepMind\u00a0under the\u00a0data-sharing arrangement?<\/p>\n<p>Asked for comment\u00a0on DeepMind\u2019s\u00a0audit infrastructure plans,\u00a0Phil Booth, coordinator of MedConfidential, raised just such\u00a0unanswered questions regarding\u00a0the\u00a0original data-sharing arrangement \u2014 pointing out that the ongoing\u00a0issue\u00a0is how and why the company\u00a0got so much\u00a0patient identifiable data in the first place, rather than quibbles over how data access might\u00a0be managed after the fact.<\/p>\n<p>Discussing\u00a0the proposed audit infrastructure, Booth said:\u00a0\u201cIn the case of Google\u2019s dodgy deal with the Royal Free, this will eventually demonstrate to a patient that data was copied to Google \u2018for direct care\u2019 when they were nowhere near the hospital at the time. It should irrevocably log that Google got data that they were not entitled to access, and now refuse to answer questions about.\u201d<\/p>\n<p>\u201cIt\u2019s like a blackbox recorder for a flight,\u201d he added, of the audit infrastructure. \u201cYou always hope it\u2019s not necessary, but if something goes wrong, its reassuring to know someone can figure out what happened after your plane flew into a mountain.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" width=\"680\" height=\"453\" src=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=680\" class=\"attachment-large size-large wp-post-image\" alt=\"\" srcset=\"https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=680 680w, https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=1360 1360w, https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=150 150w, https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=300 300w, https:\/\/tctechcrunch2011.files.wordpress.com\/2016\/12\/deepmind2mustafa-suleyman.jpg?w=768 768w\" sizes=\"(max-width: 680px) 100vw, 680px\" \/>&nbsp;Why should you trust an advertising giant with the most sensitive personal data you possess: aka your medical records? That&#8217;s the hugely sticky issue Google-owned DeepMind is facing as it seeks to embed itself into the UK&#8217;s healthcare space &#8212; a big push publicly announced in February last year. <a href=\"https:\/\/techcrunch.com\/2017\/03\/09\/deepmind-says-no-quick-fix-for-verifying-health-data-access\/?ncid=rss\">Read More<\/a><\/p>\n<div class=\"feedflare\">\n<a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:2mJPEYqXBVI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=2mJPEYqXBVI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:7Q72WNTAKBA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=7Q72WNTAKBA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:yIl2AUoC8zA\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=yIl2AUoC8zA\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:-BTjWOF_DHI\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=eoUpLFMQhP0:jXo87vACWmk:-BTjWOF_DHI\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:D7DqB2pKExk\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?i=eoUpLFMQhP0:jXo87vACWmk:D7DqB2pKExk\" border=\"0\"><\/img><\/a> <a href=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?a=eoUpLFMQhP0:jXo87vACWmk:qj6IDK7rITs\"><img decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~ff\/Techcrunch?d=qj6IDK7rITs\" border=\"0\"><\/img><\/a>\n<\/div>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/feeds.feedburner.com\/~r\/Techcrunch\/~4\/eoUpLFMQhP0\" height=\"1\" width=\"1\" alt=\"\"\/> <\/p>\n","protected":false},"author":5817,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[25],"tags":[59,66,26],"class_list":["post-335426","post","type-post","status-publish","format-standard","hentry","category-technology","tag-media","tag-techcrunch","tag-technology"],"_links":{"self":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/335426","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/users\/5817"}],"replies":[{"embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/comments?post=335426"}],"version-history":[{"count":0,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/posts\/335426\/revisions"}],"wp:attachment":[{"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/media?parent=335426"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/categories?post=335426"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/people.utm.my\/asmawisham\/wp-json\/wp\/v2\/tags?post=335426"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}