Simon Reader, Senior Policy Officer, discusses some of the key considerations for organisations undertaking data protection impact assessments for Artificial Intelligence (AI) systems.
This post is part of our ongoing on developing the ICO framework for auditing AI. We encourage you
to share your views by leaving a comment below or by emailing us at .
the blogs in this series have referred to the importance of carrying out a Data
Protection Impact Assessment (DPIA) for AI systems that will process personal
data. DPIAs offer an organisations an opportunity to consider how and why using
AI systems to process personal data and what the potential risks could be.
The ICO has
produced detailed guidance on DPIAs that explains when they
are required and how to complete them. This blog sets out some of the things organisations
should think about when carrying out a DPIA for the processing of personal data
in AI systems.
under the General Data Protection Regulation (GDPR)
are required (at least)
- before the deployment of innovative technological solutions;
- for the processing of special category personal data at large scale; or
- for automated decision-making, profiling, or for the expected denial of a service to an individual.
for processing personal data will therefore usually meet the legal requirement
for completing a DPIA.
result of an assessment indicates residual high risk to individuals that cannot
be reduced, data controllers must consult with the ICO.
should be assessed in a DPIA
needs to describe the nature, scope, context and purposes of any processing of
be used to process the data. It will need to detail:
- how data will be collected, stored and used;
- the volume, variety and sensitivity of the input data;
- the nature of the data controller’s relationship with data
- the intended outcomes for individuals or wider society and for the
if undertaken at the earliest stages of project development. It should feature,
at a minimum, the following key components.
1. A systematic description of the processing
should include a systematic description of the processing activity, including
data flows and the stages when AI processes and automated decisions may produce
effects on individuals. It can also explain any relevant variation or margins
automated decisions are subject to human intervention or review, the processes
being implement to ensure this is meaningful and that decisions can be
overturned should also be detailed.
there is a good reason not to do so, organisations should seek and document the
views of individuals, or their representatives, on the intended processing
operation during a DPIA. It is therefore important to be able to describe the
processing in a way that is accessible to those who are consulted.
can be difficult to describe the processing activity of a complex AI system. It
may be appropriate therefore, to maintain two versions of an assessment. The
first presenting a thorough technical description for specialist audiences. The
second containing a more high-level description of the processing and
explaining the logic of how the personal data inputs relate to the outputs
should set out the roles and obligations of the data controller and any
processors. Where AI systems are partly or wholly outsourced to external
providers, both organisations should also assess whether joint controllership has
been established under Article 26 of the GDPR; and if so, to collaborate in the
DPIA process as appropriate.
data processor is used, some of the more technical elements of the processing activity
can be illustrated in a DPIA by reproducing information from that processor. For
example, a flow diagram from a processor’s manual. However, the data controller
should generally avoid copying large sections of a processor’s literature into
their own assessment.
necessity and proportionality
deployment of an AI system to process personal data needs to be driven by the
proven ability of that system to fulfil a specific and legitimate purpose; not by
the availability of the technology. By assessing necessity in a DPIA, an
organisation can evidence that these purposes couldn’t be accomplished in
another reasonable way.
undertaking a DPIA, organisations can also demonstrate that the processing of
personal data by an AI system is a proportionate activity. When assessing
proportionality, the interests of the organisation need to be weighed up
against the rights and freedoms of individuals. In relation to AI systems, organisations
need to think about any detriment to data subjects that could follow from bias
or inaccuracy in the algorithms and data sets being used.
proportionality element of a DPIA, organisations need to assess whether data
subjects would reasonably expect the processing to be conducted by an AI system.
If AI systems complement or replace human decision-making, it should be
documented in the DPIA how the project might compare human and algorithmic
accuracy side-by-side to better justify its use.
should also describe any trade-offs that are made, for example between accuracy
and data minimisation, and document the methodology and rationale for these.
risks to rights and freedoms
personal data in the development and deployment of AI systems may not just pose
risks to individual’s privacy and data protection rights.
instance, machine learning systems may reproduce discrimination from historic
patterns in data, which could fall foul of equalities legislation. Similarly, AI
systems that stop content being published based on the analysis of the creator’s
personal data could impact their freedom of expression. In such contexts, data
controllers should consider the relevant legal frameworks beyond data
process will help organisations to objectively identify the relevant risks. A
score or level should be assigned to each risk, measured against the likelihood
and the severity of the impact on data subjects.
to address the risks
important that data protection officers and other information governance
professionals are involved in AI projects from the earliest stages. Clear and
open channels of communication must be established between them and the project
teams. This will ensure that risks can be identified and addressed early in the
protection should not be an afterthought, and a DPO’s professional opinion
should not come as a surprise at the eleventh hour.
be used to document the safeguards put in place to ensure the individuals
responsible for the development, testing, validation, deployment, and
monitoring of AI systems are adequately trained and have an appreciation for
the data protection implications of the processing.
measures to ensure that appropriate training is in place to mitigate risks
associated with human error can also be evidenced in a DPIA. Along with the
technical measures designed to reduce risks to the security and accuracy of an
measures have been introduced to mitigate the risks identified, the DPIA should
document the residual levels of risk posed by the processing. These must be
referred to the ICO for prior consultation if they remain high.
must be carried out before the processing of personal data begins, they should
be considered a ‘live’ document. This means they are subject to regular review
or re-assessment should the nature, scope, context or purpose of the processing
alter for any reason.
instance, depending on the deployment, it could be that the demographics of the
target population may shift, or that people adjust their behaviour over time in
response to the processing itself.
We are very interested to hear about
how organisations are approaching DPIAs in the context of AI. We would
particularly like to know:
you feel your organisation needs to take substantially different approach
to DPIAs in relation to AI systems?
your organisation has a policy of publishing DPIAs for AI systems and why
you do/ do not?
comment below or by emailing us at AIAuditingFramework@ico.org.uk.