Support Centre

You have out of 5 free articles left for the month

Signup for a trial to access unlimited content.

Start Trial

Continue reading on DataGuidance with:

Free Member

Limited Articles

Create an account to continue accessing select articles, resources, and guidance notes.

Free Trial

Unlimited Access

Start your free trial to access unlimited articles, resources, guidance notes, and workspaces.

International: AI, privacy, and security - part three: Addressing bias, siloed governance, and data breach risks in healthcare

In part one of this insight series, Dr. Paolo Balboni, Noriswadi Ismail, Davide Baldini, and Kate Francis, of ICT Legal Consulting, delved into the growing influence of artificial intelligence (AI) in areas such as recruitment, talent management, and cybersecurity. In part two, they outlined potential concerns that may arise from the use of AI in the provision of health services. In part three, they explore the imperative of addressing bias, siloed governance, and data breach risks in healthcare, emphasizing the critical need for comprehensive mitigation strategies and interdisciplinary collaboration to ensure AI's responsible integration into healthcare systems.

Rost-9D / Essentials collection / istockphoto.com

Unpacking and unboxing complexities

Risk of bias

As suggested in the previous part of this series, the use of AI in healthcare has the potential to improve care and efficiency in various ways. However, AI is notoriously prone to bias, which may be in terms of race, age, gender, disability, religion, sexual orientation, or socioeconomic background. In healthcare, bias could lead to significant harm to patients, including discrimination, poor care outcomes, and even the loss of life. Bias can exacerbate existing inequalities, underscoring the urgent need for organizations developing and deploying AI solutions in healthcare to prioritize mitigation measures and accountability. However, promising advancements are being made to reduce racial bias and enhance the screening potential of new systems.  

Bias in AI has implications in relation to the principle of fairness established in Article 5(1)(a) of the General Data Protection Regulation (GDPR). As guided by the UK Information Commissioner's Office (ICO), in the context of data protection law, fairness entails both non-discrimination and fair treatment. Bias may result from AI systems trained on imbalanced data or data that reflects past cases of discrimination which then again leads to discriminatory outcomes  The very design and use of AI systems may also lead to discrimination.

To limit bias in AI systems, organizations must have a good level of accountability which can be expressed in the form of documentation. When it comes to AI, data controllers should document the algorithmic logic used, the metrics used to train the model, verifications made to detect bias and related corrective measures, their appropriateness, and related risks, among others. Furthermore, the quality and amount of data used should be evaluated and any potential issues should be mitigated.

In relation to algorithmic fairness - defined as mathematical techniques to measure how machine learning (ML) models treat individuals from different groups in potentially discriminatory ways and reduce them - the ICO stresses the necessity for organizations to adopt a holistic approach to fairness. Specifically, organizations should evaluate power imbalances between themselves and those whose data is processed, the underlying structures and dynamics of the environment in which the AI will be deployed, the presence of self-reinforcing feedback loops, the scale and nature of potential harms resulting from processing, and how decisions are made, favoring rationality and causality over correlation, etc.  

Appropriate mitigation measures will depend on the specific context and domain of AI operation and must therefore be identified on a case-by-case basis. Organizations should also document their approach to mitigating bias and discrimination across the AI lifecycle, establish clear policies and procedures, and ensure that data is representative of the population to which the system will be applied. Trade-offs of different approaches, safeguards implemented for specific groups, and the performance of the system as well as how it was expected to perform should also be documented. As substantively outlined in the Maastricht University Data Protection as a Corporate Social Responsibility Framework (UM-DPCSR Framework), organizations should also evaluate their own level of diversity and cases of bias and discrimination.  

Risk of continued silo

One of the prevailing challenges faced not only by healthcare organizations but by any organization willing to incorporate AI solutions within its workflows is the tendency to compartmentalize AI governance and other frameworks which are also related to AI. However, given the particular risks associated with healthcare, potential shortcomings in AI governance are greatly increased. In particular compartmentalization, or 'siloed governance,' occurs when cybersecurity, data privacy, data governance, and AI strategies are developed and implemented independently, without a cohesive and integrated approach. While these components may individually excel in their objectives, the lack of coordination and alignment can give rise to unforeseen risks that permeate throughout the organization.

Siloed governance creates many pitfalls and shortcomings for organizations that incur in them. From a cybersecurity perspective, it may result in fragmented security postures, where individual AI components are fortified independently and without an overall, overarching, cybersecurity aim. AI systems, as any other software, are typically composed of many different elements: some of these components are not AI-related (e.g., the hosting infrastructure, backups, interface, etc.) and some are (i.e., the AI model itself). While the latter is usually delivered by the AI producer who has provided the AI system, some components of the former may instead be deployed by the organization itself (e.g., the infrastructure). As a result, if the cybersecurity governance of one or more components is allocated to different teams within the organization, the resulting cybersecurity posture will be implemented incoherently and inconsistently. This approach therefore neglects the interconnectedness of AI systems components and, consequently, vulnerabilities may emerge at the intersections of the different components, leading to potential breaches and unauthorized access. In sum, it is essential to embrace a comprehensive strategy that aligns with the principles of security in depth and security by design to guarantee the compliance of AI systems with cybersecurity requirements.

Siloed governance can lead to other privacy shortcomings. Given that AI systems used in healthcare (e.g., for diagnostics) typically process personal data, including sensitive data, both during training and deployment. Healthcare organizations are subject to strict privacy regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) in the US, the General Data Protection Regulation (GDPR) in the EU, the Personal Information Protection Law (PIPL) in China, and the General Data Protection Law (LGPD) in Brazil, just to name a few. The application of a siloed compliance approach to each jurisdiction, namely, treating each country's legal requirements as completely autonomous and independent from each other, may result in inconsistencies in data handling practices, jeopardizing compliance, and exposing organizations to legal and financial repercussions. To avoid a similar outcome, it is imperative that healthcare organizations address the data protection compliance of their AI systems in the context of their Group-Data Protection Compliance Framework (G-DPCF). In this respect, the implementation of a G-DPCF should already provide the organization with a sound methodology to address all relevant local data privacy legislations where the organization operates.

At a general level, and beyond sector-specific shortcomings, operating in silos often leads to redundant investments and resource allocations. Duplication of efforts in developing separate strategies for cybersecurity, data privacy, and AI can be inefficient, diverting valuable resources away from holistic initiatives that could enhance overall organizational resilience.

Overall, effective AI implementation in healthcare demands interdisciplinary collaboration among cybersecurity experts, data privacy professionals, data governance specialists, and AI practitioners. Siloed governance inhibits this collaboration, hindering the development of comprehensive solutions that address the multifaceted challenges posed by AI in healthcare. To mitigate the emerging risks associated with siloed governance in AI and healthcare, organizations must adopt a holistic and integrated approach to their AI strategy. Cross-siloed governance should be implemented, emphasizing collaboration between cybersecurity, data privacy, data governance, and AI teams. This approach ensures that these critical components are developed and implemented cohesively, aligning with the broader organizational goals and targeted operating models.

Risk of endless data and security remodeling

The iterative, data-intensive, and dynamic nature of AI development, particularly for emerging AI technologies such as generative AI, provides a fertile ground for innovation and rapid prototyping. However, the agility of data and security remodeling in the AI field, often driven by use cases still in the sandbox phase, introduces a distinctive risk, namely the potential for endless cycles of data and security remodeling, which pose risks in terms of uncertainty, emerging bias, ethics, and responsibility.

The sandbox phase of AI development is inherently characterized by experimentation, exploration, and learning, which may often lead to uncertain and unpredictable outcomes. As a result, the constant quest for optimization and innovation may lead to a cycle of endless data and security remodeling. As new insights emerge and technologies evolve, organizations may find themselves in a perpetual state of adjustment, grappling with the uncertainty of when to freeze configurations and protocols. This ongoing remodeling introduces a layer of unpredictability, making it challenging to establish robust and consistent data privacy and security measures.

Furthermore, the continuous remodeling of AI applications may inadvertently contribute to the accumulation of biases, as discussed in the section on the risk of bias above.

Another risk of endless data and security remodeling concerns the consideration of ethical implications in AI, which becomes increasingly challenging in an environment of continuous remodeling. The constant evolution of use cases and applications may lead to ethics fatigue, where ethical considerations take a back seat to the urgency of rapid development inherent in AI. This fatigue can undermine the commitment to upholding ethical standards in AI applications.

While data and security remodeling are inherent to AI innovation, awareness of the risks associated with perpetual remodeling is essential. This should be done by strategically balancing the need for ongoing improvements with the establishment of stable foundations and responsible development practices.

In practice, in order to mitigate relevant risks connected to this phenomenon, it is important to implement critical freeze points in the AI development lifecycle, where data and security configurations are solidified. This helps mitigate the risk of perpetual remodeling and provides a stable foundation for ongoing, systematic assessment and improvement. Furthermore, regular ethical reviews should be integrated into the development process to ensure that ethical considerations remain a priority, prevent ethics fatigue, and foster a culture of responsible AI development.

Implementing clear accountability frameworks that delineate responsibilities throughout the data and security remodeling cycles may further ensure that potential risks are addressed promptly and that the right stakeholders are accountable for their decisions.

How AI, cybersecurity, data privacy, and governance minimize healthcare data breach

Pain points and practical implications

Personal data breaches in the healthcare context typically pose high to very high risks to the rights and freedoms of data subjects, as AI systems used in healthcare often process special category data, the processing, let alone the breach, is considered to be inherently risky by most data privacy legislations worldwide. For the same reason, the processing of health data, and its breach is heavily regulated in many jurisdictions, sometimes with specific privacy legislation for the healthcare sector, such as in the US with the Health Insurance Portability and Accountability Act (HIPAA) regulation, and sometimes with encompassing data privacy legislation, such as the Personal Information Protection Law (PIPL) in China, the GDPR, and the General Data Protection Law (LGPD) in Brazil.

One of the most risky situations can arise when data breaches occur in the context of AI-powered clinical trials, where the AI system often processes health data. This event may present many pain points, some typical of AI technologies and some unique to the healthcare sector, which might occur simultaneously or not, depending on the exact dynamics of the event.

Supply chain vulnerabilities, including breaches of third parties involved in clinical trials, are often a common cause of data breaches, including in healthcare. Parties other than the healthcare provider may be involved both in the context of the IT infrastructure that is part of the AI system and in the clinical trial itself if the third party is providing the trial resources. Furthermore, in the case of AI-as-a-service applications, any vulnerability presented by the AI provider will inevitably be reflected in the clinical study.

Further threats may come from the very AI systems used for clinical trials, which are typically based on machine learning (ML) algorithms and may therefore rely on technologies that are relatively new, or even at the sandbox or proof-of-concept stage: given the young age of these technologies, not all AI-specific threats are already fully understood. Many new vulnerabilities are in fact inherent to machine learning, such as data poisoning, model stealing, and model inversion, just to name a few. While cybersecurity institutions, communities, and literature are undoubtedly catching up with these new threats, AI-specific threats are still not fully understood and countered.

Moreover, it should not be overlooked that traditional cybersecurity threats still exist in the context of novel AI systems: although well-known preventive and reactive measures already exist for threats such as human error, social engineering, software vulnerabilities, etc., similar threats still need to be taken into account, assessed, and mitigated by the healthcare organization.

Despite an organization's best efforts to implement security measures, when a data breach occurs in the context of AI-powered healthcare solutions, the usual steps to investigate, remedy, mitigate, and document the occurrence should apply here as well. However, given the complexity entailed by the use of novel technologies, all these phases risk being lengthier than usual, especially when the vulnerability giving rise to the breach is inherent to the AI system itself and, therefore, the technical complexity of the system is likely to encumber the data breach management process. Despite this, it is paramount that the affected organization is able to complete all the necessary steps legally required timeframe for documenting and, if necessary, reporting breaches to competent regulators or even affected individuals; if this is not possible, the organization may still be able to leverage the possibility of reporting the breach in multiple phases (e.g., a first notification within the legally-mandated timeframe, and further integrations as soon as reasonably feasible)

Equally, and for the same reasons, investigations into the breach carried out by the competent regulators, which may be triggered by the notification of the event, might take longer than usual. This is especially the case where the breach has a transnational reach, typically when it affects individuals residing or located in multiple jurisdictions.

Consider the following example of a data breach occurring in the context of AI-powered processing of globally sourced health data, suffered by a data controller established in an EEA country, but affecting data belonging to individuals located not only in multiple EEA Member States but also in other non-EEA countries. In this case, several authorities could be involved in the assessment of the breach, albeit from different perspectives:

  • the data protection authority of the organization's Member State of establishment, as lead supervisory authority in the context of the GDPR one-stop-shop mechanism;
  • each national EEA data protection authority, which may claim the role of 'concerned authority' when one or more data subjects reside within their Member State of competence;
  • non-EEA data protection authorities, which are likely to have jurisdiction in each case where one or more affected individuals reside within their country; and
  • the competent market surveillance authority under the EU Artificial Intelligence Act (AI Act), to whom providers of high-risk AI systems are required to report any 'serious incident' concerning their systems.

While all of the elements outlined above highlight the many new layers of complexity introduced by the use of AI in healthcare, which exacerbate the already difficult task of managing compliance with multiple data and AI-related legislations worldwide, the following section will explore some strategies aimed at successfully managing such new risks, including by means of prior engagement with the regulators.

Calibration and recalibration

Despite the complex nature of AI-related data breaches in healthcare, there are strategies that can both reduce the risk of such events ex-ante and allow for efficient and effective management of breaches ex-post, should they occur.

Once again, as also seen in the section on pain points and practical implications, the first line of defense against all types of breaches should typically be found in the organization Group-Data Protection Compliance Framework (G-DPCF). At a general level, the effective implementation, within the organization's activities, of the recognized principles of data protection and security by design, purpose limitation, data minimization, and storage limitation, all contribute to enhancing the cybersecurity posture of the organization, thereby greatly lessening the risks of occurrence of breaches.

Concrete and practical applications of these principles could include, for example, the introduction of robust pseudonymization of the personal data used to train and/or operate the AI systems (or even full anonymization, including the use of synthetic data, where this does not run counter to the purpose of the processing), as well as the presence of adequate technical and organizational security measures surrounding the AI systems, including the regular conduct of audits on the AI supply chain.

In particular, the organization's data breach management procedure should be carefully reviewed and amended to take into account the AI-related threats outlined above, especially when operating on a global dataset (see, in this regard, the example seen at the end of the previous section). In doing so, in line with a holistic approach, this activity should be carried out with the involvement of several actors, such as the legal and cybersecurity teams, ethics boards/advisors, the AI governance team, etc., depending on the organization's structure. In this way, the procedure will incorporate the necessary steps and considerations necessary to successfully manage a data breach occurring in this area.

In line with the principle of accountability, every single data breach suffered (even false positives) should be taken into account to recalibrate both the organization's cybersecurity posture and its data breach management procedure. This is especially important when suffering AI-related breaches, given the novel and often still-obscure nature of many threats.

Finally, organizations should consider proactively engaging with regulators when developing or deploying a new AI system in order to mitigate risks from the outset and thereby reduce the likelihood of breaches. An interesting example where this engagement could take place is in the context of 'regulatory sandboxes,' which are controlled spaces where authorities engage organizations to test innovative products or services that challenge existing legal frameworks, for a limited amount of time. Regulatory sandboxes have recently emerged in the context of AI applications, initially in the FinTech sector, as a tool to promote legal certainty and the uptake of safe and trustworthy AI solutions. Participation and admission to the sandbox varies, depending on the legal instrument that envisages it but, generally speaking, they have two types of benefits. On the one hand, they allow regulators to gain insights into the functioning of AI systems, thereby acquiring knowledge and expertise of new technological trends and solutions which they are then called to supervise. On the other hand, organizations can obtain valuable real-time guidance from the regulator, at the development stage of the solution. Sometimes, successful participation in the sandbox can even allow the organization to obtain a relative presumption of conformity to its technological solution. In light of this, healthcare organizations could particularly benefit from submitting their new AI solution to a regulatory sandbox, given the increased risk inherent to the handling of health data, thereby greatly reducing risks associated with updating AI solutions.

Dr. Paolo Balboni Founding Partner
[email protected]
Noriswadi Ismail Of Consultant & Advisor
[email protected]
Davide Baldini Partner
[email protected]
Kate Francis Privacy & Ethics Researcher, Development & Communication Specialist
[email protected]
ICT Legal Consulting