Ofcom’s proactive technology measures: principles-based or vague?

Ofcom has published its long-expected consultation on additional measures that it recommends U2U platforms and search engines should implement to fulfil their duties under the Online Safety Act.  The focus, this time, is almost entirely on proactive technology: automated systems intended to detect particular kinds of illegal content and content harmful to children, with a view to blocking or swiftly removing them.

The consultation marks a further step along the UK’s diverging path from the EU Digital Services Act. The DSA prohibits the imposition of general monitoring obligations on platforms. Those are just the kind of obligations envisaged by the Online Safety Act’s preventative duties, which Ofcom is gradually fleshing out and implementing.

Ofcom finalised its first Illegal Harms Code of Practice in December 2024. For U2U services the Code contained two proactive technology recommendations: hash and URL matching for CSAM. The initial consultation had also suggested fuzzy keyword matching to detect some kinds of fraud, but Ofcom did not proceed with that. The regulator indicated that it would revisit fraud detection in a later, broader consultation. That has now arrived.

The new U2U proposals go beyond fraud. They propose perceptual hash-matching for visual terrorism content and for intimate image abuse content. They suggest that content should be excluded from recommender feeds if there are indications that it is potentially illegal, unless and until it is determined via content moderation to be legal. 

Most ambitiously, Ofcom wants certain relatively large platforms to research the availability and suitability (in accordance with proposed criteria) of proactive technology for detection of fraud and some other illegal behaviour, then implement it if appropriate. Those platforms would also have to review existing technologies that they use for these purposes and, if feasible, bring them into line with Ofcom’s criteria.

Ofcom calls this a ‘principles-based’ measure, probably because it describes a qualitative evaluation and configuration process rather than prescribing any concrete parameters within which the technology should operate.

Freedom of expression

Legal obligations for proactive content detection, blocking and removal engage the fundamental freedom of expression rights of users. Obligations must therefore comply with ECHR human rights law, including requirements of clarity and certainty.

Whilst a principles-based regime may be permissible, it must nevertheless be capable of predictable application. Otherwise it will stray into impermissible vagueness. Lord Sumption in Catt said that what is required is a regime the application of which is:

“reasonably predictable, if necessary with the assistance of expert advice. But except perhaps in the simplest cases, this does not mean that the law has to codify the answers to every possible issue which may arise. It is enough that it lays down principles which are capable of being predictably applied to any situation."

In Re Gallagher he said that:

“A measure is not “in accordance with the law” if it purports to authorise an exercise of power unconstrained by law. The measure must not therefore confer a discretion so broad that its scope is in practice dependent on the will of those who apply it, rather than on the law itself. Nor should it be couched in terms so vague or so general as to produce substantially the same effect in practice.”

Typically these strictures would apply to powers and duties of public officials. The Online Safety Act is different: it requires U2U service providers to make content decisions and act (or not) to block or remove users’ posts. Thus the legal regime that requires them to do that has to provide sufficient predictability of their potential decisions and resulting acts.

In addition to fraud and financial services offences, Ofcom’s proposed principles-based measures would apply to image based CSAM, CSAM URLs, grooming, and encouraging or assisting suicide (or attempted suicide).

Any real-time automated content moderation measure poses questions about human rights compatibility. The auguries are not promising: proactive technology, armed only with the user’s post and perhaps some other on-platform data, will always lack contextual information. For many offences off-platform information can be the difference between guilt and innocence.  Decisions based on insufficient information inevitably stray into arbitrariness.

Then there is the trade-off between precision and recall. Typically, the more target content the automated tool is tuned to catch, the more false positives it will also throw up. False positives result in collateral damage to legitimate speech. It does not take many false positives to constitute disproportionate interference with users’ rights of freedom of expression.

Lord Grade, the Chairman of Ofcom, said in a recent speech that the aims of tackling criminal material and content that poses serious risks of harm to children’s physical or emotional health were not in conflict with freedom of expression. Indeed so, but focusing only on the aim misses the point: however worthy the end, it is the means - in this case proactive technology - that matters.

Prescribed by law

Ofcom’s Proactive Technology Draft Guidance says this about proportionality of the proposed measures:

“Proactive technology used for detection of harmful content involves making trade-offs between false positives and false negatives. Understanding and managing those trade-offs is essential to ensure the proactive technology performs proportionately, balancing the risk of over-removal of legitimate content with failure to effectively detect harm.” (para 5.14)

Proportionality is a requirement of human rights compliance. However, before considering proportionality a threshold step has to be surmounted: the ‘prescribed by law’ or ‘legality’ condition. This is a safeguard against arbitrary restrictions - laws should be sufficiently precise and certain that they have the quality of law.

The prescribed by law requirement is an aspect of the European Convention on Human Rights. It has also been said to be a UK constitutional principle that underpins the rule of law:

"The acceptance of the rule of law as a constitutional principle requires that a citizen, before committing himself to any course of action, should be able to know in advance what are the legal consequences that will flow from it." (Lord Diplock, Black-Clawson [1975])

The Constitutional Reform Act 2005 refers in S.1 to:

“the existing constitutional principle of the rule of law”.

For content monitoring obligations the quality of law has two facets, reflecting the potential impact of the obligations on the fundamental rights of both platforms and users.

The platform aspect is written in to the Act itself:

“the measures described in the code of practice must be sufficiently clear, and at a sufficiently detailed level, that providers understand what those measures entail in practice”. (Schedule 4)

The user aspect is not spelled out in the Act but is no less significant for that. Where a user’s freedom of speech may be affected by steps that a platform takes to comply with its duties, any interference with the user’s right of freedom of expression must be founded on a clear and precise rule.

That means that a user must be able to foresee in advance with reasonable certainty whether something that they have in mind to post is or is not liable to be blocked, removed or otherwise affected as a result of the obligations that the Act places on the platform.

That is not simply a matter of users themselves taking care to comply with substantive law when they consider posting content. The Act interpolates platforms into the process and may require them to make judgements about whether the user’s post is or is not illegal. Foreseeability is therefore a function both of the substantive law and of the rules about how a platform should make those judgements.

If, therefore, the mechanism set up by the Act and Ofcom for platforms to evaluate, block and take down illegal content is likely to result in unpredictable, arbitrary determinations of what is and is not illegal, then the mechanism fails the ‘prescribed by law’ test and is a per se violation of the right of freedom of expression.

Equally, if the regime is so unclear about how it would operate in practice that a court is not in a position to assess its proportionality, that would also fail the ‘prescribed by law’ test. That is the import of Lord Sumption’s observations in Catt and Gallagher (above).

A prescriptive bright-line rule, however disproportionate it might be, would satisfy the ‘prescribed by law’ test and fall to be assessed only by reference to necessity and proportionality. Ofcom’s principles-based recommendations, however, are at the opposite end of the spectrum: they are anything but a bright-line rule. The initial ‘prescribed by law’ test therefore comes into play.

How do Ofcom’s proposed measures stack up?

Service providers themselves would decide how accurate the technology has to be, what proportion of content detected by the technology should be subjected to human review, and what is an acceptable level of false positives.

Whilst Ofcom specifies various ‘proactive technology criteria’, those are expressed as qualitative factors to be taken into account, not quantitative parameters. Ofcom does not specify what might be an appropriate balance between precision and recall, nor what is an appropriate proportion of human review of detected content.

Nor does Ofcom indicate what level of false positives might be so high as to render the technology (alone, or in combination with associated procedures) insufficiently accurate.

Examples of Ofcom’s approach include:

“However, there are some limitations to the use of proactive technology in detecting or supporting the detection of the relevant harms. For example, proactive technology does not always deal well with nuance and context in the same way as humans.

However, we mitigate this through the proactive technology criteria which are designed to ensure proactive technology is deployed in a way that ensures an appropriate balance between precision and recall, and that an appropriate proportion of content is reviewed by humans.” (Consultation, para 9.92)

“Where a service has a higher tolerance for false positives, more content may be wrongly identified. … The extent of false positives will depend on the service in question and the way in which it configures its proactive technology. The measure allows providers flexibility in this regard, including as to the balance between precision and recall (subject to certain factors set out earlier in this chapter).” (Consultation, paras 9.135, 9.136)

“… when determining what is an appropriate proportion of detected content to review by humans, providers have flexibility to decide what proportion of detected content it is appropriate to review, however in so doing, providers should ensure that the following matters are taken into account…” (Consultation, para 9.19)

“However, in circumstances where false positives are consistently high and cannot be meaningfully reduced or mitigated, particularly where this may have a significant adverse impact on user rights, providers may conclude that the proactive technology is incapable of meeting the criteria.” (Proactive Technology Draft Guidance, para 5.19)

How high is high? How significant is significant? No answer is given, other than that the permissible level of false positives is related to the nature of the subsequent review of detected content. As we shall see, the second stage review does not require all content detected by the proactive technology to be reviewed by human beings. The review could, seemingly, be conducted by a second automated system.

The result is that two service providers in similar circumstances could arrive at completely different conclusions as to what constitutes an acceptable level of legitimate speech being blocked or taken down. Ofcom acknowledges that the flexibility of its scheme:

“could lead to significant variation in impact on users’ freedom of expression between services”. (Consultation, para 9.136)

That must raise questions about the predictability and foreseeability of the regime.

If the impact on users’ expression is not reasonably foreseeable, that is a quality of law failure and no further analysis is required. If that hurdle were surmounted, there is still the matter of what level of erroneous blocking and removal would amount to a disproportionate level of interference with users’ legitimate freedom of expression. 

Proportionality?

Ofcom concludes that:

“Having taken account of the nature and severity of the harms in question, the principles we have built into the measure to ensure that the technology used is sufficiently accurate, effective and lacking in bias, and the wider range of safeguards provided by other measures, we consider overall that the measure’s potential interference to users’ freedom of expression to be proportionate.” (Consultation, para 9.154)

However, it is difficult to see how Ofcom (or anyone else) can come to any conclusion as to the overall proportionality of the recommended principles-based measures when they set no quantitative or concrete parameters for precision versus recall, accuracy of review of suspect content, or an ultimately acceptable level of false positives.

Ofcom’s discussion of human rights compliance starts with proportionality. While it notes that the interference must be ‘lawful’, there is no substantive discussion of the ‘prescribed by law’ threshold.

Prior restraint

Finally, on the matter of human rights compatibility, proactive detection and filtering obligations constitute a species of prior restraint (Yildirim v Turkey (ECtHR), Poland v The European Parliament and Council (CJEU)).

Prior restraint is not impermissible. However, it does require the most stringent scrutiny and circumscription, in which risk of removal of legal content will loom large. The ECtHR in Yildirim noted that “the dangers inherent in prior restraints are such that they call for the most careful scrutiny on the part of the Court”.

The proactive technology criteria

Ofcom’s proactive technology criteria are, in reality, framed not as a set of criteria but as a series of factors that the platform should take into account.  Ofcom describes them as “a practical, outcomes-focused set of criteria.” [Consultation, para 9.13]

Precision and recall One criterion is that the technology has been evaluated using “appropriate” performance metrics and

“configured so that its performance strikes an appropriate balance between precision and recall”.  (Recommendation C11.3(c))

Ofcom evidently must have appreciated that, without elaboration, “appropriate” was an impermissibly vague determinant. The draft Code of Practice goes on (Recommendation C11.4(a)):

“when configuring the technology so that it strikes an appropriate balance between precision and recall, the provider should ensure that the following matters are taken into account:

i) the service’s risk of relevant harm(s), reflecting the risk assessment of the service and any information reasonably available to the provider about the prevalence of target illegal content on the service;

ii) the proportion of detected content that is a false positive;

iii) the effectiveness of the systems and/or processes used to identify false positives; and

iv) in connection with CSAM or grooming, the importance of minimising the reporting of false positives to the National Crime Agency (NCA) or a foreign agency;”

These factors may help a service provider tick the compliance boxes – ‘Yes, I have taken these factors into account’ - but they do not amount to a concrete determinant of what constitutes an appropriate balance between precision and recall.

Review of detected content Accuracy of the proactive technology is, as already alluded to, only the first stage of the recommended process. The service provider has to treat a detected item as providing ‘reason to suspect’ that it is illegal content, then move on to a second stage: review.

“Where proactive technology detects or supports the detection of illegal content and/or content harmful to children, providers should treat this as reason to suspect that the content may be target illegal content and/or content harmful to children.

Providers should therefore take appropriate action in line with existing content moderation measures, namely ICU C1 and ICU C2 (in the Illegal Content User-to-user Codes of Practice) and PCU C1 and PCU C2 (in the Protection of Children User-to-user Code of Practice), as applicable.” (Consultation, para 9.74)

That is reflected in draft Codes of Practice paras ICU C11.11, 12.9 and PCU C9.9, 10.7. For example:

“ICU C11.11 Where proactive technology detects, or supports the detection of, target illegal content in accordance with ICU C11.8(a), the provider should treat this as reason to suspect that the content may be illegal content and review the content in accordance with Recommendation ICU C1.3.”

‘Review’ does not necessarily mean human review. Compliance with the proactive technology criteria requires that:

“...policies and processes are in place for human review and action is taken in accordance with that policy, including the evaluation of outputs during development (where applicable), and the human review of an appropriate proportion of the outputs of the proactive technology during deployment. Outputs should be explainable to the extent necessary to support meaningful human judgement and accountability.” (Emphasis added) (draft Code of Practice Recommendation ICU C11.3(g))

The consultation document says:

“It should be noted that this measure does not itself recommend the removal of detected content. Rather, it recommends that providers moderate detected content in accordance with existing content moderation measures (subject to human review of an appropriate proportion of detected content, as mentioned above).” (Consultation, para 9.147)

And:

“Providers have flexibility in deciding what proportion of detected content is appropriate to review, taking into account [specified factors]” (Consultation, para 9.145)

Ofcom has evidently recognised that “appropriate proportion” is, without elaboration, another impermissibly vague determinant. It adds (Recommendation C11.4(b)):

“when determining what is an appropriate proportion of detected content to review by humans, the provider should ensure that the following matters are taken into account:

i) the principle that the resource dedicated to review of detected content should be proportionate to the degree of accuracy achieved by the technology and any associated systems and processes;

ii) the principle that content with a higher likelihood of being a false positive should be prioritised for review; and

iii) in the case of CSAM or grooming, the importance of minimising the reporting of false positives to the NCA or a foreign agency.”

As with precision and recall, these factors may help a service provider tick the compliance boxes but are not a concrete determinant of the proportion of detected content to be submitted to human review in any particular case.

Second stage review – human, more technology or neither?

The upshot of all this appears to be that content detected by the proactive technology should be subject to review in accordance with the Code of Practice moderation recommendations; and that an ‘appropriate proportion’ of that content should be subject to human review.

But if only an ‘appropriate proportion’ of content detected by the proactive technology has to be subject to human review, how is the rest to be treated? Since it appears that some kind of ‘appropriate action’ is contemplated in accordance with Ofcom’s content moderation recommendations, the implication appears to be that moderation at the second stage could be by some kind of automated system.

In that event it would seem that the illegal content judgement itself would be made by that second stage technology in accordance with Recommendation C1.3.

Recommendation C1.3, however, does not stipulate the accuracy of second stage automated technology. The closest that the Code of Practice comes is ICU C4.2 and 4.3:

“The provider should set and record performance targets for its content moderation function, covering at least:

a) the time period for taking relevant content moderation action; and

b) the accuracy of decision making.

In setting its targets, the provider should balance the need to take relevant content moderation action swiftly against the importance of making accurate moderation decisions.”

Once again, the path appears to lead to an unpredictable balancing exercise by a service provider.

Curiously, elsewhere Ofcom appears to suggest that second stage “complementary tools” could in some cases merely be an ‘additional safeguard’:

“What constitutes an appropriate balance between precision and recall will depend on the nature of the relevant harm, the level of risk identified and the service context. For example, in some cases a provider might optimise for recall to maximise the quantity of content detected and apply additional safeguards, such as use of complementary tools or increased levels of human review, to address false positives. In other cases, higher precision may be more appropriate, for example, to reduce the risk of adverse impacts on user rights.” (Proactive Technology Draft Guidance, para 5.18)

If the implication of ‘in some cases’ is that in other cases acting on the output of the proactive technology without a second stage review would suffice, that would seem to be inconsistent with the requirement that all detected content be subject to some kind of moderation in accordance with Recommendation C1.3.

Moreover, under Ofcom’s scheme proactive technology is intended only to provide ‘reason to suspect’ illegality. That would not conform to the standard stipulated by the Act for an illegal content judgement: ‘reasonable grounds to infer’.

Conclusion

When, as Ofcom recognises, the impact on users’ freedom of expression will inevitably vary significantly between services, and Ofcom’s documents do not condescend to what is or is not an acceptable degree of interference with legitimate speech, it is difficult to see how a user could predict, with reasonable certainty, how their posts are liable to be affected by platforms’ use of proactive technology in compliance with Ofcom’s principles-based recommendations.

Nor is it easy to see how a court would be capable of assessing the proportionality of the measures. As Lord Sumption observed, the regime should not be couched in terms so vague or so general as, substantially, to confer a discretion so broad that its scope is in practice dependent on the will of those who apply it. Again, Ofcom's acknowledgment that the flexibility of its scheme could lead to significant variation in impact on users’ freedom of expression does not sit easily with that requirement.  

Ofcom, it should be acknowledged, is to an extent caught between a rock and a hard place. It has to avoid being overly technology-prescriptive, while simultaneously ensuring that the effects of its recommendations are reasonably foreseeable to users and capable of being assessed for proportionality. Like much else in the Act, that may in reality be an impossible circle to square. That does not bode well for the Act’s human rights compatibility.

[Amended 6 August 2025 to add ‘principles-based’ to the first paragraph of the Conclusion.]

Public order: from street protest to the Online Safety Act

Assiduous readers of this blog will know of my fondness for working through concrete examples to illustrate how, once they come into force (now likely to be in Spring next year), platform illegal content duties under the UK Online Safety Act 2023 (OSA) might pan out in practice.

A recurring theme has been that making judgements about the legality or illegality of user content, as platforms are required to do by the OSA, is not a simple matter. The task verges at times on the impossible: platforms are required to make complex legal and factual judgements on incomplete information. Moreover, the OSA stipulates a relatively low threshold for a platform to conclude that content is illegal: reasonable grounds to infer. The combined result is that the OSA regime is likely to foster arbitrary decisions and over-takedown of legal user content.

The newest opportunity to hypothesise a concrete example is presented by the acquittal of Marieha Hussain, who was charged with a racially aggravated public order offence for carrying, at a pro-Palestine demonstration, a placard depicting Rishi Sunak and Suella Braverman as coconuts.  The prosecution alleged that this was a well-known racial slur. The district judge held that it was part of the genre of political satire, and that the prosecution had not proved to the criminal standard that it was abusive.

Ms Hussain was prosecuted for an offence in a public street, to which the Online Safety Act would not directly apply. However, what if an image of the placard appeared online? If displaying the placard in the street was sufficient to attract a criminal prosecution, even if ultimately unsuccessful, could the OSA (had it been in force) have required a platform to take action over an image of the placard displayed online? 

As it happens the prosecution in Marieha Hussain’s case was prompted by someone posting a photograph of the placard online, accompanied by a critical comment. That was followed by a response from the Metropolitan Police, who were tagged in the post:

 

If the Online Safety Act duties were in force (and assuming that the court had not yet delivered its acquittal verdict), how would a service provider have to go about deciding whether an online post of a photograph of the placard should be treated as illegal? How would that differ from the court process? Could the differences lead a service provider to conclude that a post containing an image of the placard should be removed? Could (or should) the fact that a prosecution had been instigated for display of the placard in the street, or (before that) that the police had indicated an interest, affect the platform’s illegality judgement?

The prosecution

As far as can be understood from the press reports, Ms Hussain was prosecuted for a racially aggravated offence under Section 5 of the Public Order Act 1986. The Section 5 offence (so far as relevant to this example) is:

“(1) A person is guilty of an offence if he—

(a) uses… abusive words or behaviour…, or

(b) displays any writing, sign or other visible representation which is… abusive,

within the hearing or sight of a person likely to be caused harassment, alarm or distress thereby.

(2) An offence under this section may be committed in a public or a private place, except that no offence is committed where the words or behaviour are used, or the writing, sign or other visible representation is displayed, by a person inside a dwelling and the other person is also inside that or another dwelling.

(3) It is a defence for the accused to prove—

(a) that he had no reason to believe that there was any person within hearing or sight who was likely to be caused harassment, alarm or distress, or

(b) that he was inside a dwelling and had no reason to believe that the words or behaviour used, or the writing, sign or other visible representation displayed, would be heard or seen by a person outside that or any other dwelling, or

(c) that his conduct was reasonable.

Additionally, someone is guilty of the offence only if they intend their words or behaviour, or the writing, sign or other visible representation, to be… abusive, or are aware that it may be… abusive.

The racially aggravated version of the offence (which carries a larger fine) applies if the basic offence is committed and:

“(a) at the time of committing the offence, or immediately before or after doing so, the offender demonstrates towards the victim of the offence hostility based on the victim’s membership (or presumed membership) of a racial …  group; or

(b) the offence is motivated (wholly or partly) by hostility towards members of a racial…  group based on their membership of that group.”

The ‘victim’ for the purpose of (a) is the person likely to be caused harassment, alarm or distress.

Both offences are triable only in the magistrates’ court. If the defendant is acquitted of the racially aggravated offence the court may go on to consider the basic offence, but only if it is charged in the alternative (which the CPS Charging Guidance says it should be).

Priority offences

Both the basic offence under Section 5 and the racially aggravated version are within the scope of the Online Safety Act. They are listed in Schedule 7 as ‘priority offences’. As such, not only is a service provider required swiftly to take down illegal content if it becomes aware of it (OSA Section 10(3)(b)), but it may be required to take proportionate proactive prevention measures (OSA Section 10(2)(a)).

The Section 5 offence attracted attention during the Online Safety Bill’s passage through Parliament. On 19 May 2022 the Chair of the Joint Parliamentary Committee on Human Rights, Harriet Harman MP, wrote to the then Secretary of State, Nadine Dorries. She said:

“It is hard to see how providers, and particularly automated responses, will be able to determine whether content on their services fall on the legal or illegal side of this definition”.

She went on:

“…how will a provider of user-to-user services judge whether particular words or behaviour online are “abusive” rather than merely offensive and whether or not they are likely to cause someone “distress” sufficient to amount to a criminal offence?”

and

“Will the inclusion of section 5 Public Order Act 1986 within the category of priority illegal content, in practice, result in service providers removing content that does not meet the criminal threshold, potentially resulting in an interference with the Article 10 rights of users?”

The DCMS Minister, Chris Philp MP, replied on 16 June 2022. In response to the specific questions about Section 5 he recited the general provisions of the Bill.

JUSTICE, in its Lords Second Reading Briefing, elaborated on the concerns of the Joint Human Rights Committee and called for Section 5 to be removed from the category of priority illegal content. That did not happen.

So far, so clear. Now the picture starts to get foggy, for a variety of reasons.

Making an Illegal Content Judgement

First, is either version of the Section 5 offence capable of applying online at all? Inclusion of the Section 5 offence in Schedule 7 is not conclusive that it can be committed online. The reason for inclusion of offline offences is that, in principle, it is possible to encourage or assist online an offence that can only be committed offline. Such inchoate offences (plus conspiracy, aiding and abetting) are also designated as priority offences. (Parenthetically, applying the inchoate offences to online posts presents its own problems in practice – see here.)

One potential obstacle to applying the Section 5 offences online is the requirement that the use or display be: “within the hearing or sight of a person likely to be caused harassment, alarm or distress thereby”. Does this require physical presence, or is online audibility or visibility sufficient? If the latter, must the defendant and the victim (i.e. the person likely to be caused harassment, alarm or distress) be online simultaneously? The Law Commission considered the simultaneity point in its consultation on Modernising Communications Offences, concluding that the point was not clear.

Ofcom, in its draft Illegal Content Judgements Guidance, does not address the question expressly. It appears to assume that the “within hearing or sight” condition can be satisfied online. That may be right. But it is perhaps unfortunate that the Act provides no mechanism for obtaining an authoritative determination from the court on a point of law this kind.

Second, which offence should be considered? CPS practice is to charge the more serious racially aggravated offence if there is credible evidence to prove it. Under the Online Safety Act, the opposite applies: the simpler, less serious offence should be the one adjudged. The Ofcom consultation documents explain why:

“In theory, in order to identify a racially aggravated offence, the service would not only need to identify all the elements of the Public Order Act offence, but also all the elements of racial or religious aggravation. But in practice, in order to identify the content as illegal content, the service would only need to show the elements of the underlying Public Order Act priority offence, because that would be all that was needed for the takedown duty to be triggered. The racial aggravation would of course be likely to make the case more serious and urgent, but that would be more a matter of prioritisation of content for review than of identifying illegal content.” [26.81]

Third, how strong does the evidence of an offence have to be?

In court, a criminal offence has to be proved beyond reasonable doubt. The district judge in the Hussain case concluded that the placard was: “part of the genre of political satire” and that as such, the prosecution had “not proved to the criminal standard that it was abusive”. The prosecution had also not proved to the criminal standard that the defendant was aware that the placard may be abusive. The court reached those decisions after a two day trial, including evidence from two academic expert witnesses called by the defence to opine on the meaning of ‘coconut’.

A service provider, however, must treat user content as illegal if it has “reasonable grounds to infer” that it is illegal. That is a lower threshold than the criminal standard.

Could that judgement be affected by the commencement of a criminal prosecution? The Director of Public Prosecutions’ Charging Guidance says that for a criminal prosecution to be brought the prosecutor: “must be satisfied that there is sufficient evidence to provide a realistic prospect of conviction…” It must be “more likely than not” that “an objective, impartial and reasonable jury, bench of magistrates or a judge hearing a case alone, properly directed and acting in accordance with the law, would convict the defendant of the charge alleged.”

Whether “reasonable grounds to infer” is a lower threshold than the “more likely than not to convict” Charging Guidance test for commencing a prosecution is a question that may merit exploration. If (as seems likely) it is lower, or even if it is just on a par, then a platform could perhaps be influenced by the fact that a prosecution had been commenced, in the light of the evidential threshold for that to occur. However, it does not follow from commencement of a prosecution for a street display that the charging threshold would necessarily be surmounted for an online post by a different person.

The more fundamental issue is that the lower the service provider threshold, the more likely that legal content will be removed and the more likely that the regime will be non-compliant with the ECHR. The JUSTICE House of Lords briefing considered that ‘reasonable grounds to infer’ was a ‘low bar’, and argued that provisions which encourage an overly risk-averse approach to content removal, resulting in legitimate content being removed, may fall short of the UK’s obligations under the ECHR.  

The Ofcom consultaion observes:

“What amounts to reasonable grounds to infer in any given instance will necessarily depend on the nature and context of the content being judged and, particularly, the offence(s) that may be applicable.” [26.15]

The significance of context is discussed below. Notably, the context relevant to criminal liability for a street display of a placard may be different from that of an online post of an image of the placard by a third party.

The service provider’s illegal content judgement must also be made on the basis of “all relevant information that is reasonably available” to it. Self-evidently, a service provider making a judgement about a user post would not have the benefit of two days’ factual and expert evidence and accompanying legal argument, such as was available to the court in the Hussain prosecution. The question of what information should be regarded as reasonably available to a service provider is a knotty one, implicating data protection law as well as the terms of the OSA. Ofcom discusses this issue in its Illegal Harms consultation, as does the Information Commissioner’s Office in its submission to the Ofcom consultation. The ICO also touches on it in its Content Moderation Guidance.

In order for the Section 10(3)(b) swift takedown obligation to be triggered, the service provider must have become aware of the illegal content. Ofcom’s consultation documents implicitly suggest that the awareness threshold is the same as having reasonable grounds to infer illegality under Section 192. That equation is not necessarily as clear-cut as might be assumed (discussed here).

Fourth, whose awareness?

Ms Hussain’s placard was held not to be abusive. The court also held that she did not have the necessary awareness that the placard may be abusive. A service provider faced with an online post of an image of a placard would have to consider whether it had reasonable grounds for an inference that the placard was abusive and that the person who posted it (rather than the placard bearer) had the necessary awareness.

When it comes at least to reposting, Professor Lorna Woods, in her comments on the Ofcom Illegal Content Judgements Guidance, has argued that a requirement to evaluate the elements of an offence for each person who posts content is too narrow an interpretation of the OSA:

“The illegal content safety duties are triggered by content linked to a criminal offence, not by a requirement that a criminal offence has taken place. … The requirement for reasonable grounds to infer a criminal offence each time content is posted … presents an overly restrictive interpretation of relevant content. Such a narrow perspective is not mandated by the language of section 59, which necessitates the existence of a link at some stage, rather than in relation to each individual user. … There is no obligation to look at the mental state of each individual disseminator of the content”

Professor Woods gives as an example the reposting of intimate images without consent.

S.59 (which defines illegal content) has expressly to be read together with S.192 (illegal content judgements). S.192, at first sight, reads like an instruction manual for making a judgement in relation to each individual posting. Be that as it may, if Professor Woods’ argument is correct it seems likely for many kinds of offence (even if not for the intimate images offence) to reintroduce the problems that the Independent Reviewer of Terrorism Legislation identified with S.59 (then Clause 52). The Bill was subsequently amended to add S.192, it is assumed in response to his criticisms:

“2. ...Intention, and the absence of any defence, lie at the heart of terrorism offending. ... 

16. The definition of “terrorism content” in clause 52(5) is novel because under terrorism legislation content itself can never “amount to” an offence. The commission of offences requires conduct by a person or people.

17. Clause 52(3) attempts to address this by requiring the reader of the Bill to consider content in conjunction with certain specified conduct: use, possession, viewing, accessing, publication or dissemination.

18. However, as Table 1 shows, conduct is rarely sufficient on its own to “amount to” or “constitute” a terrorism offence. It must ordinarily be accompanied by a mental element and/or take place in the absence of a defence. …

23. … It cannot be the case that where content is published etc. which might result in a terrorist offence being committed, it should be assumed that the mental element is present, and that no defence is available.

24. Otherwise, much lawful content online would “amount to” a terrorist offence.”

My own subsequent submission to the Public Bill Committee analysed Clause 52, citing the Independent Terrorism Reviewer's comments, and concluded in similar vein:

"Depending on its interpretation the Bill appears either:

6.21.1 to exclude from consideration essential ingredients of the relevant criminal offences, thereby broadening the offences to the point of arbitrariness and/or disproportionate interference with legitimate content; or

6.21.2 to require arbitrary assumptions to be made about those essential ingredients, with similar consequences for legitimate content; or

6.21.3 to require the existence of those ingredients to be adjudged, in circumstances where extrinsic factual material pertaining to those ingredients cannot be available to a filtering system.

In each case the result is arbitrariness (or impossibility), significant collateral damage to legal content, or both.”

An interpretation of the OSA that increases the likelihood of lawful content being filtered or taken down also increases concomitantly the risk of ECHR incompatibility. (See also, ‘Item by Item Judgements’ below)

On a different point, Ofcom appears to suggest that the wider and more general the audience for a controversial post, the greater the likelihood of awareness being inferred:

“A service must also draw an inference that the person posting the content concerned was at least aware that their behaviour may be abusive. Such awareness may reasonably be inferred if the abusive behaviour is very obviously likely to be distressing to most people and is posted somewhere with wide reach.” [A3.77]

In contrast:

“It is less likely to be reasonably inferred if content is posted to a place where, for example, only persons sharing similar sorts of content themselves are likely to see it.” [A3.77] 

Fifth, any defence?

As to the Section 5 defence of reasonable conduct, the district judge said that had it been necessary to go that far, she would have found Ms Hussain's conduct to be reasonable in that she was exercising her right to freedom of expression, and the judge would not have been satisfied that the prosecution was a proportionate interference with her right, or necessary in a democratic society. 

Our hypothetical assumes that no court ruling has been made. If the service provider has concluded that there are reasonable grounds to infer abusive content and awareness, how should it evaluate the possibility of a defence such as reasonable conduct?

When making an illegal content judgement a service provider can only base a judgement on the availability of a defence if it positively has some reason to infer that a defence to the offence may be successfully relied upon. That is the effect of OSA S.192(6)(b):

“(6) Reasonable grounds for that inference exist in relation to content and an offence if … a provider—

(a) has reasonable grounds to infer that all elements necessary for the commission of the offence, including mental elements, are present or satisfied, and

(b) does not have reasonable grounds to infer that a defence to the offence maybe successfully relied upon.”

An obvious instance of positive grounds to infer a Section 5 reasonable conduct defence on the part of the poster would be a comment added to the image.

In a different context (terrorism), Ofcom has reached the same conclusion as to the need for positive grounds:

“There is a defence of ‘reasonable excuse’ which may be harder for services to make reasonable inferences about, but they only need to consider it if there are positive grounds to do so.” [26.93]

Similarly, for the offence of stirring up racial hatred:

“In cases where there are no reasonable grounds to infer intent it is a defence for a person to show that he was not aware that the content might be insulting or abusive. However, positive grounds to infer this would need to be available to the service.” [A3.90]

As to the Section 5 “reasonable conduct” defence, a service provider hypothetically considering the original online post of the Marieha Hussain placard in the absence of a court judgment would have to consider whether, if it considered that there were reasonable grounds to infer that the placard was abusive and that the post satisfied the other elements of the offence, the comment by the poster (in addition to anything inferrable from the nature of the posted image) provided reasonable grounds to infer that a defence of reasonable conduct might be successfully relied upon. 

It might also be relevant to consider whether there were reasonable grounds to infer that the original placard holder could have have a reasonable conduct defence for the street display, as the judge in the Hussain case held that she would have done. However, the defence is specific to the conduct of each defendant, not a finding about the nature of the content. 

As the judge's remarks demonstrate, consideration of the reasonable conduct defence can result in the service provider making judgements about the necessity and proportionality of the interference with freedom of expression. 

Ofcom’s Illegal Content Judgements Guidance says:

“Services should take a common-sense approach to considering whether the behaviour displayed in the content could be considered reasonable. For example, it may be reasonable (even if unwise) to abuse someone in response to abuse.” [A3.68]

Common sense also comes to the aid of the harassment and distress element of the Section 5 offence:

“Services should consider any information they hold about what any complainant has said about the emotional impact of the content in question, and take a common-sense approach about whether it is likely to cause harassment or distress.” [A3.27]

Appeals to common sense bring to mind the Oxford Reference definition of palm tree justice: 

“Ad hoc legal decision-making, the judge metaphorically sitting under a tree to make rulings based on common sense rather than legal principles or rules.”

The perceived value of guidance based on common sense may also depend on whether one shares the William O. Douglas view that ‘Common sense often makes good law’ or that of Albert Einstein: “Common sense is the collection of prejudices acquired by age eighteen”.

In addition to reasonable conduct, Section 5 of the Public Order Act provides a defence “that he had no reason to believe that there was any person within hearing or sight who was likely to be caused harassment, alarm or distress”.

Ofcom suggests that a post that is legal may be rendered illegal through the poster being deprived of the defence as the result of a notification:

“it is a defence if it is reasonable to infer that the person had no reason to believe that there was any person within hearing or sight who was likely to be caused harassment or distress. This is most likely to be relevant where a user is challenging a takedown decision (but of course if the person becomes aware as a result of the takedown decision that such a person was within hearing or sight, the content would become illegal content).” [A3.33]

That and Ofcom’s comment on the relationship between awareness and wide reach are both reminiscent of the concerns about the “harmful communications” offence that was originally included in the Bill, then dropped.

Sixth, what is the significance of context? The Hussain decision appears to have turned on the court’s finding of what was ‘abusive’ in the context of the display of the placard (albeit that the racially aggravated element of the alleged offence inevitably focused attention on whether the placard was specifically racially abusive).

The Ofcom Illegal Judgments Guidance on the Section 5 offence emphasises the significance of context:

“However, the context should be taken into account carefully, since abusive content may also carry political or religious meaning, and will be more likely to be a reasonable exercise of the right to freedom of expression if it is.” [A3.79]

While some of the context available to a service provider may be the same as that available to a court (for instance it is apparent on the face of the image of the Hussein placard that it was a political comment), much of the available context may be different: different person, different place, different audience, additional comments, no expert witnesses. Add to that a different standard of proof and a different statutory framework within which to judge illegality, and the possibility of a different (most likely more restrictive) conclusion on legality from that which a court would reach (even if considering the same version of the offence) is significant.

The last word on context should perhaps go to Ofcom, in its Illegal Content Judgements Guidance on Section 5:

“We have not given any usage examples here, due to the particularly strong importance of context to these judgements.” [A3.81]

Item by item judgements?

While some may argue that the OSA is about systems and processes, not content, there is no doubt (pace Professor Woods’ argument noted above) that at least some of its illegality duties require platforms to make item by item content judgements (see discussion here). The duties do not, from a supervision and enforcement point of view, require a service provider to get every individual judgement right. They do require service providers to make individual content judgements.

Ofcom evidently expects service providers to make item by item judgements on particular content, while noting that the function of the online safety regime is different from that of a court:

“The ‘beyond reasonable doubt’ threshold is a finding that only UK courts can reach. When the ‘beyond reasonable doubt’ threshold is found in UK courts, the person(s) responsible for the relevant illegal activity will face criminal conviction. However, when services have established ‘reasonable ground to infer’ that content is illegal according to the Act, this does not mean that the user will necessarily face any criminal liability for the content and nor is it necessary that any user has been prosecuted or convicted of a criminal offence in respect of such content. When services make an illegal content judgement in relation to particular content and have reasonable grounds to infer that the content is illegal, the content must however be taken down.” [26.14]

Critics of the OSA illegality duty have always doubted the feasibility or appropriateness of requiring platforms to make individual content legality judgements, especially at scale. Those coming at it from a freedom of expression perspective emphasise the likelihood of arbitrary judgements, over-removal of legal content and consequent incompatibility with the European Convention on Human Rights.

The ‘systems and processes’ school of thought generally advocates harm mitigation measures (ideally content-agnostic) in preference to item-by-item content judgements. Relatedly, the Online Safety Network recently suggested in a Bluesky post that “the government needs to amend the Act to make clear that - once content has been found to be illegal content – it should continue to be categorised that way”. That would reduce the need for successive item-by-item illegality judgements in relation to the same content, and would make explicit what Professor Woods has argued is already the proper interpretation of the Act (see above).

The comments of the Online Safety Network were made in the specific context of the non-consensual intimate image offence. For offences where the gravamen lies in the specific nature of the prohibited content, and the role of any mental element, other condition or defence is secondary (such as ensuring only that accidental behaviour is not criminalised), there may be some force in the suggestion that the same content should always be treated in the same way (at least if the initial finding of illegality has been verified to a high standard). Ofcom’s proposed CSAM image filtering duties, for instance, would operate on that basis.

Elevated to a general principle, however, the suggestion becomes problematic. For offences where the conduct element is broad or vague (such as the Section 5 offence), or where context is significant, or where the heavy lifting of keeping the offence within proper bounds is done by the mental element or by defences, it would be overreaching (and at serious risk of ECHR incompatibility) automatically to deem the same item of content to be illegal regardless of context, intention or of any other factors relevant to illegality. In the terrorism field filtering algorithms have had trouble distinguishing between illegal terrorist content and legal news reports of the same content. To deem that content always to be illegal for the purpose of filtering and takedown duties would be controversial, to say the least.

The Online Safety Network went on to comment that “the purpose of the regime is not to punish the person sharing the content, but to control the flow of that content.” It is true that the safety duties do not of themselves result in criminal liability of the user. But “don’t worry, we’re only going to suppress what you say” does not feel like the most persuasive argument for an interference with lawful freedom of expression.

[The original version of this post stated: "Since Ms Hussain’s placard was held not to be abusive, it appears that the magistrates’ court did not rule on any available defences." Now updated, with some consequential additions to the discussion of the reasonable conduct defence, in the light of Professor Augustine John's fuller account of the judge's ruling. [21 September 2024) 

The Online Safety Act: proactive illegality duties, safeguards and proportionality

Part 4 of a short series of reflections on Ofcom’s Illegal Harms consultation under the Online Safety Act 2023 (OSA). 

A significant proportion of the consultation’s discussion of Ofcom's proposed Code of Practice recommendations — especially those involving proactive monitoring and detection of illegal content — is taken up with enumerating and evaluating safeguards to accompany each recommended measure.

That is to be expected, for two reasons. First, the OSA itself provides in Schedule 4 that measures recommended in a Code of Practice must be designed in the light of the importance of protecting the privacy of users and the right of users to freedom of expression within the law, and (where appropriate) incorporate safeguards for the protection of those matters.

Second, the potential interference with users' fundamental rights (notably freedom of expression and privacy) brings into play the European Convention on Human Rights (ECHR) and the Human Rights Act (which, following the UK's recent general election, we can assume will be with us for the foreseeable future).

The first step in the ECHR analysis is to consider whether the interference is “prescribed by law”. This is a threshold condition: if the interference fails that test, it is the end of the matter. When considering whether an interference contained in a statute is prescribed by law, it is not enough that the law has been passed by Parliament and is publicly accessible. It also has to have the “quality of law”: it must be sufficiently clear and precise that someone potentially affected by it can foresee in advance, with reasonable certainty, how the law will apply to them.

Requirements (strictly speaking, in the case of the OSA, Ofcom recommendations) for automated proactive detection, filtering and removal of user content present a particularly high risk of arbitrary interference with, and over-removal of, legal content. They can also be seen as a species of prior restraint. The European Court of Human Rights observed in Yildirim that "the dangers inherent in prior restraints are such that they call for the most careful scrutiny on the part of the Court".

Compatibility with the ECHR operates at two levels: the legislative measure and individual decisions taken under it. A court will regard itself as well placed, since it has the facts to hand, to determine whether an individual decision is or is not a justified interference with a Convention right. It is far less willing to declare a legislative measure per se incompatible, unless it is clear that when applied in practice it will result in a breach of Convention rights in most or all cases. If the measure is capable of being operated in a way that does not breach the Convention, then it will not be per se incompatible.

However, there is an important rider: the UK courts have said that in order to protect against arbitrary interference there must be safeguards which have the effect of enabling the proportionality of the interference to be adequately examined.

In the case of legislation such as the OSA, where the Act frames the duties at a very high level and a regulator is authorised to flesh them out, the necessary safeguards have to be provided in Ofcom's Codes of Practice and its statutory guidance. If such safeguards are not provided, or if they are not sufficient, then the regime will fall at the first Convention hurdle of not being prescribed by law. The ECHR compatibility of the regime on this score is thus heavily dependent on Ofcom's work product.

Much judicial ink has been expended on explaining the precise underlying rationale for the “capable of being adequately examined" test. It is safest to regard it as an aspect of the prescribed by law (a.k.a. “legality”) principle: the reason why legislation must be reasonably clear and precise is in order to prevent arbitrariness and the abuse of imprecise rules or unfettered discretionary powers. If the impact of the scheme is foreseeable, then its proportionality is capable of being assessed. If its impact across the board is not discernible, then its impact will be arbitrary.

Lady Hale said in the Supreme Court case of Gallagher:

“The foundation of the principle of legality is the rule of law itself - that people are to be governed by laws not men. They must not be subjected to the arbitrary - that is, the unprincipled, whimsical or inconsistent - decisions of those in power. 

This means, first, that the law must be adequately accessible and ascertainable, so that people can know what it is; and second, that it must be sufficiently precise to enable a person - with legal advice if necessary - to regulate his conduct accordingly. The law will not be sufficiently predictable if it is too broad, too imprecise or confers an unfettered discretion on those in power. 

This is a separate question from whether the law in question constitutes a disproportionate interference with a Convention right -the law in question must contain safeguards which enable the proportionality of the interference to be adequately examined. 

This does not mean that the law in question has to contain a mechanism for the review of decisions in every individual case: it means only that it has to be possible to examine both the law itself and the decisions made under it, to see whether they pass the test of being necessary in a democratic society.”

In the final analysis it may be said that safeguards have to provide sufficient protection against arbitrariness.

The courts have stressed that challenging an entire regime ex ante on proportionality grounds presents a high hurdle and will rarely succeed, compared with a challenge by an individual who claims that their rights have been violated in a particular instance. Nevertheless, the safeguards proposed by Ofcom have to pass the prescribed by law test. If they do pass, then the actual proportionality of a given interference can be considered should a case arise.

The impact of the legality requirement and the nature of the required safeguards have to be considered in the light of the triangular structure of the Online Safety Act regime. We are not here dealing with a discretionary power vested in a state official to direct a user to take down their post. The OSA regime places legal obligations on intermediary service providers. The steps that they take to comply with those obligations have the potential to affect users' rights, particularly freedom of expression. 

Foreseeability requires that a user should be able to predict, with reasonable certainty, whether their contemplated online post is liable to be affected by actions taken by a service provider in discharging its obligations under the Act.

The safeguards stipulated by Ofcom should therefore provide the requisite degree of predictability for users in respect of blocking and removal actions to be taken by service providers when carrying out Ofcom's recommended measures.

As regards the consultation’s general approach to ECHR compliance, two points stand out. The first is that there is virtually no discussion of the “prescribed by law” requirement. Its existence is recited in many places, but the substantive discussion of ECHR compatibility proceeds directly to discussion of legitimate aim, necessity and proportionality of the recommended measures. Para 1.14 of the consultation may provide a clue as to why that is:

“In passing the Act, Parliament has set out in legislation the interferences prescribed by law and which it has judged to be necessary in our democratic society.”

Similarly in para 12.64:

“…our starting point is that Parliament has determined that services should take proportionate steps to protect UK users from illegal content. Of course there is some risk of error in them doing this, but that risk is inherent in the scheme of the Act.”

There is possibly a hint here of regarding the fact that Parliament has passed legislation as being sufficient of itself to satisfy the “prescribed by law” requirement. That may be the starting point, but it is not the end point.

The second point is that insofar as Ofcom has focused on the need for clarity and certainty, it has done so from the perspective of providing clarity to service providers. The Act requires this. Schedule 4 provides that the measures described in a Code of Practice must be:

“sufficiently clear, and at a sufficiently detailed level, that providers understand what those measures entail in practice;”

That, however, does not detract from the ECHR requirement that the potential for interference with users’ privacy and freedom of expression must also be reasonably clear and precise.

The two requirements do not necessarily go hand in hand. A provision may be clear as to the amount of discretion that it gives to a service provider, yet unforeseeable in its effect on the freedom of expression of users.

Several aspects of Ofcom's proposed safeguards in relation to automated detection and related takedowns give pause for thought on the question of capability to assess the proportionality of the interference. The recommendations (which would apply only to some services) are: 

• Perceptual hash matching against a database of known CSAM material (draft U2U Code of Practice, A4.23)
• URL matching against a list of known CSAM URLs (draft U2U Code of Practice, A4.37)
• Fuzzy keyword matching to detect articles for use in fraud (draft U2U Code of Practice, A4.45)

The concerns are most apparent with the fraud keyword proposal, albeit they are not entirely absent with CSAM hash and URL matching. 

URL matching presents the fewest challenges. Ofcom's proposed safeguards relate entirely to the process for establishing and securing the list of URLs. They provide that the service provider should source the list from: 

“a person with expertise in the identification of CSAM, and who has arrangements in place to [inter alia] secure (so far as possible) that URLs at which CSAM is present, and domains which are entirely or predominantly dedicated to CSAM, are correctly identified before they are added to the list; to review CSAM URLs on the list, and remove any which are no longer CSAM URLs” [draft Code of Practice, A4.40]

By way of further safeguards, both the person with expertise and the service provider should secure the list from unauthorised access, interference or exploitation (whether by persons who work for the provider or are providing a service to the provider, or any other person).

The reasonable assumption is that the technology is capable of accurately matching detected URLs with the list, such that no further safeguards are required on that score.

If there were any concern about adequacy of these safeguards, it would probably be whether "a person with expertise in the identification of CSAM" is sufficiently precisely articulated.

For CSAM hash matching the draft Code of Practice contains equivalent safeguards to URL matching for establishment and security of the hash database. However, further safeguards are required since the recommendation of perceptual hashing introduces an element of judgement into the matching process, with the concomitant risk of false positives and consequent blocking or removal of legal user content.

Here the adequacy of the proposed safeguards may be open to more serious debate. The draft Code of Practice states that the perceptual hashing technology should be configured so that its performance strikes "an appropriate balance between precision and recall".

Precision and recall refers to the incidence of false positives and missed hits. There is typically a trade-off: fewer missed hits means more false positives.  As to what is an appropriate balance between them, the draft Code of Practice stipulates that the provider should ensure that the following matters are taken into account: 

- The risk of harm relating to image-based CSAM, as identified in the risk assessment of the service, and including in particular information reasonably available to the provider about the prevalence of CSAM content on its service.

- The proportion of detected content that is a false positive; and

- The effectiveness of the systems and/or processes used to identify false positives.

Annex 15 to the Consultation suggests various further factors that could point towards striking the balance towards either precision or recall.

The draft Code of Practice stipulates that human moderators should review “an appropriate proportion” of material detected as CSAM, and sets out principles that the service provider should take into account in deciding what proportion of detected content it is appropriate to review - for instance that the resource dedicated to review of detected content should be proportionate to the degree of accuracy achieved by the perceptual hash matching technology. It also provides various periodic review and record-keeping recommendations.

Annex 15 sets out Ofcom’s reasons (related to differences between perceptual hash technologies) for not setting a threshold which should be used to determine whether an image is a match.

The substantive balancing and proportionality decisions are thus parked firmly on the desk of the service provider. However, neither the draft Code of Practice nor the Act itself contains any indication of what is to be regarded as a proportionate or disproportionate level of interference with legal user content.

The result is that two different service providers could readily apply the stipulated safeguards in equivalent factual situations, follow the prescribed process and reach significantly differing conclusions about what is an appropriate balance between precision and recall, or about what resource should be devoted to human review. Consequently it can be argued that the effect on user content cannot be predicted. That smacks of arbitrariness. 

The safeguards for fuzzy keyword detection of articles for use in fraud are more extensive, as would be expected for a technology that is inherently more likely to throw up false positives. The consultation document points out that the recommendation:

"...differs from our proposed measures regarding CSAM hashing and the detection of CSEA links which focus on the detection of positive matches with content (or URLs that provide access to content) that has already been determined to be illegal." [Annex 15, A15.121]

Unlike with CSAM URL and hash matching the draft Code of Practice envisages that the service provider may compile its own list of fraud keywords. It contains safeguards around establishment, testing, review and security of the list. It contains equivalent provisions to perceptual hash matching for configuration of the technology so as to strike “an appropriate balance between precision and recall”, stipulating equivalent matters to be taken into account. Ofcom envisages that the safeguards will mean that it will be ‘highly likely’ that a keyword hit will correspond to an offence:

“In light of the above, we would expect any content detected as a result of applying this [keyword technology] measure to be highly likely to amount to an offence concerning articles for use in frauds.” [Volume 4, para 14.249]

It goes on:

“We recognise however that the keyword detection measure we are considering will enable services to identify content about which no prior illegal content judgment or determination has been made and that it may result in false positives. It may identify legitimate content (such as news articles or academic articles) which discuss the supply of articles for use in fraud. It is for this reason that we are not recommending that services take down all content detected by the technology, and are instead recommending that it be considered by services in accordance with their internal content moderation policies.” [ibid]

As with perceptual hash matching the draft Code of Practice provides for after the event periodic human review of some detected content. Whereas for perceptual hash matching this has to be ‘an appropriate proportion’, for fraud detection it has to be ‘a reasonable sample’. Again, it sets out principles to be taken into account in deciding what is a reasonable sample. These bear some similarities to, but are not identical to, those for perceptual hash matching. For instance there is no stipulation that review resource should be proportionate to the degree of accuracy achieved by the technology.

Evaluating the adequacy of the fraud keyword safeguards is complicated by the latitude that the recommendations give service providers as to what kind of action to take following initial keyword detection, and possible statutory interpretation questions as to whether (and if so in what way) the illegality judgement provisions of S.179 and the swift takedown obligations of S.10(3)(b) apply. 

Ofcom's approach is summarised thus:

"... we do not consider it appropriate to recommend that services swiftly take down all content detected as a positive match by their keyword detection technology, instead we recommend (as discussed below) that the decision on whether or not the content should be taken down should be taken in accordance with their content moderation systems and processes." [Annex 15, A15.122]

This is consistent with Ofcom’s broader policy approach to content moderation:

“Given the diverse range of services in scope of the new regulations, a one-size-fits-all approach to content moderation would not be appropriate. Instead of making very specific and prescriptive proposals about content moderation, we are therefore consulting on a relatively high-level set of recommendations which would allow services considerable flexibility about how to set up their content moderation teams.” [Volume 4, p.18]

 Ofcom continues, in relation to its fraud keyword recommendations:

“Consistent with Chapter 12, we are not persuaded that it would be appropriate to specify in detail how services should configure their content moderation systems and processes to take account of content detected by the keyword detection technology (for example, that there be human moderation of all such content), or the outcomes that those systems and processes should achieve (for example, through detailed KPIs).” [Annex 15, A15.123]

It then says:

“We are proposing in that Chapter that all U2U service providers must have in place content moderation systems or processes designed to take down illegal content swiftly.” [Annex 15, A15.124]

The area in which the keyword recommendations depart most significantly from hash and URL matching is thus in the steps to be taken in respect of positive keyword matches: treating them in accordance with the service provider’s internal content moderation systems and processes.  Ofcom’s approach is not to be prescriptive but to give service providers broad latitude in what steps to take in respect of positive keyword matches.

There is, however, an underlying dilemma. There are significant costs and risks associated with being prescriptive: the interference with a platform’s own rights (e.g under ECHR Protocol 1, Article 1), the unlikelihood that a single size of straitjacket can fit all in-scope service providers, prejudicing existing services, the chilling or dampening effect on development of new services, and the greater likelihood that faced with a prescriptive requirement service providers will take an over-cautious approach to blocking and removals. 

Yet the less prescriptive the measures, the broader the range of permissible approaches, the less predictable the effect on users and the greater the likelihood of arbitrary interference with user rights. This dilemma is not of Ofcom’s making. It is hardwired into the Act, but it falls to Ofcom to resolve it. It is an unenviable task. It may be impossible.

Specifically in relation to the fraud keyword detection recommendation, Ofcom says:

"... Implementations that substantially impact on freedom of expression, including the automatic take down of detected content, could be in accordance with the measure in our Code of Practice.” [Chapter 14, para 14.283]

and:

"whether or not such content were, incorrectly, subject to takedown would depend on the approach, to content moderation adopted by the service, rather than the content's detection by the keyword detection technology in and of itself." [Chapter 14, paras 14.284, 14.302]

Ofcom acknowledges that:

“There could therefore be variation in the impact on users’ freedom of expression arising from services’ different implementations of the technology and different approaches to moderation and take down of any detected content.” [para 14.283]

Ofcom, does not, however, discuss the implications for the “capable of being adequately examined" requirement if those variations are insufficiently foreseeable.

The discussion in Annex 15 contemplates that a service provider might have “no systems and processes in place to identify false positives before content is taken down”. That, it is said, would be a factor leaning towards configuring the system to towards greater precision at the expense of recall.

Recommended safeguards for content moderation generally include setting of performance targets, as they relate to accuracy of decision-making; training and materials; and appeals. For performance targets, it is for the service provider to balance the desirability of taking illegal content down swiftly against the desirability of making accurate moderation decisions. As above, different service providers could apply that guidance yet reach significantly different conclusions.

In the context of proportionality Ofcom seeks to diminish the impact on users’ freedom of expression by exempting news publisher content from the fraud keyword matching recommendation (reflecting the Act's exclusion of such content from regulated U2U content). However, that prompts the question of how service providers are to distinguish between news publisher content and the rest, in the context of an automated system: something which raises its own safeguards issues.

Ofcom’s fraud keywords recommendation cross-refers to its Recommendation 4B for large or multi-risk services: that the provider should set and record (but need not necessarily publish) internal content policies setting out rules, standards and guidelines around what content is allowed and what is not, and how policies should be operationalised and enforced. The policies should be drafted such that illegal content (where identifiable as such) is not permitted.    

Recommendation 4A (which is stated not to apply to CSAM perceptual hash and URL matching, but does not exclude fraud keyword detection) also appears potentially relevant to the fraud keyword matching recommendation: the service provider should have systems or processes designed to swiftly take down illegal content of which it is aware (mirroring the statutory obligation in S.10(3)).

Recommendation 4A goes on that for that purpose when the provider has reason to suspect that content may be illegal content, the provider should either make an illegal content judgement in relation to the content and, if it determines that the content is illegal, swiftly take it down; or do the same where its terms of service prohibit the type of illegal content in question and the content is in breach. 

Ofcom comments in relation to Recommendation 4A that:

"The design of this option is not prescriptive as to whether services use wholly or mainly human or automated content moderation processes." [Volume 12, para 12.50]

Thus there appears to be the potential four-way interaction between internal content moderation policies, the statutory takedown obligation, the Recommendation 4A takedown recommendation, and the provider's public terms of service. 

How these might mesh with each other is not immediately clear to this reader. In part this could depend on questions of interpretation of the Act, such as whether awareness for purposes of the statutory takedown obligation requires human awareness or can be satisfied by an automated system, and if so whether awareness equates to reasonable grounds to infer under S.192.    

Overall, the scope for arbitrary interference on user rights of freedom of expression appears to be greater for fraud keyword detection than with CSAM hash and URL matching.

The question of safeguards for proactive, automated detection systems is due to raise its head again. Ofcom has said that it is planning an additional consultation later this year on how automated tools, including AI, can be used to proactively detect illegal content and content most harmful to children – including previously undetected child sexual abuse material.

30 July 2024. Correction to description of 'high hurdle'.