‘Computer says no’: how Brussels is letting algorithmic social-scoring in by the backdoor

How many ‘scores’ did you get today? You probably do not know. But a bank has certainly assessed how risky you are. A platform has ranked the content you’re going to view. A fraud system may have flagged your behaviour as unusual. A potential employer may have sorted your CV before a real human being could have a look at it. At the border, an automated system may have decided you deserve a second look before you even land. Some of these scores can, and will, cost you a job, a home, an income, and even your future. Others are almost invisible, but could be equally harmful. Just as courts and regulators are beginning to expose the consequences of these systems, Brussels, with its Digital Omnibus, is considering weakening key safeguards against automated decision-making. Red line Brussels promised not to cross During negotiations on the Artificial Intelligence (AI) Act, ‘social scoring’ became one of the clearest lines the EU promised it would not cross. The political shorthand was China, followed by grand statements about not wanting a society where people are classified as trustworthy or suspicious, with an array of penalties – and few privileges – following from that classification. The EU prohibited certain forms of social scoring. But perhaps we spent too much time looking for one giant, all-encompassing score, instead of focusing on the hundreds of smaller ones built around us. Take SCHUFA, Germany’s dominant credit agency. It holds data on around 69 million people, and its scores influence whether someone gets credit, signs a phone contract or rents a home. In 2023, the Court of Justice of the EU recognised how much power can sit inside a ‘mere’ number that is being used to inform fully automated decisions: when a score is used to decide whether you get a loan, a contract or another service, that score can have very real consequences for your life. Now SCHUFA faces another major challenge. In August, the Austrian NGO noyb sent it a cease-and-desist letter after investigations found that the company stores millions of historical records which should have been deleted, and uses them for testing purposes. But if people cannot see all the data used to judge them, how are they supposed to challenge the result? Workers know another version of that problem. Uber fined €825m for deleting drivers Last month, the Dutch Data Protection Authority fined Uber almost €825m for using fully automated decisions to block and deactivate drivers. Customer ratings and suspected fraud feed decisions that cut a driver off from the platform, and therefore from their income. According to the regulator, those decisions took place without prior human assessment, with workers exposed to lose tomorrow’s earnings because of a system detecting an alleged pattern. The Netherlands offers many a warning. SyRI, a government system designed to detect fraud involving social benefits, allowances and taxes, was supposed to identify people considered ‘higher risk.’ In 2020, The Hague District Court struck down the legislation underpinning it, finding the system insufficiently transparent and verifiable and incompatible the right to private life. A few years later, the Dutch education agency DUO gave students fraud-risk scores using their age, type of education and the distance between their home and their parents’ address. A higher score increased the chance for investigation and even a home visit. Around 21,500 students were selected for ‘fraud’ checks between 2013 and 2022 partly on the basis of the algorithm. The Dutch Data Protection Authority found the system discriminatory and unlawful: students with a non-European migration background were selected more often because they tended to score higher on those supposedly neutral criteria. There is a pattern here. Scoring systems have a particular tendency to discriminate against people who are already vulnerable. They turn inequality into a variable and then present the outcome as ‘efficiency.’ AI scanning your CV And now the European Commission itself is preparing an AI tool to help rank candidates for EU jobs. Its latest generalist competition attracted 174,922 applications. The tool under development can score and rank candidates to help recruiters deal with that volume. Allegedly, humans may still make the final decision. Yet, a system can exercise enormous power before anyone presses the final button: it can decide which CV reaches the top of the pile, which welfare claim looks suspicious, which worker deserves scrutiny, or which racialised person should receive closer attention. Much of this ‘scoring society’ is even harder to see. Advertising systems constantly assess who is likely to respond to what, or recommender systems - yes, that ‘For You’ feed - ranking what deserves your attention before you see it. The consequences are deeply unequal. Whereas a wealthy person rejected by one bank can try another, someone desperately looking for affordable housing may have no second option, or a platform worker suddenly deactivated cannot wait months for a complaint to be resolved. The people with the least power are often those about whom institutions collect the most data. Legally, these systems are not all the same. Article 22 of the GDPR gives people special protection against serious decisions driven by automated processing. And that protection does not disappear just because a human formally makes the final call. If an automated score or ranking strongly shapes the outcome, it can already amount to automated decision-making. Other systems may be harder to capture in that way because they rank, recommend or flag people without – at least not obviously - producing the final outcome, but can still shape who looks promising, suspicious or risky before a human ever gets involved. This is why the EU’s current deregulation drive matters. Brussels backtracks The Commission’s Digital Omnibus in the iteration of the so-called ‘Data Omnibus’ proposes to rewrite Article 22, one of the GDPR’s strongest safeguards against significant decisions taken solely through automated processing. Most strikingly, it would allow automated decision-making to be considered ‘necessary’ for a contract even when the same decision could be taken by a human. That is not a minor technical adjustment. It risks turning a safeguard into an exception: if a company can say that automation is useful or efficient, the question becomes not whether a human could make the decision, but whether the company has a reason to automate it. The timing is difficult to ignore, mind you. While courts and regulators are finally showing why these safeguards matter, Brussels is considering making automated decisions easier to justify in the name of ‘efficiency’. What are we willing to give up in the name of – allegedly - making regulation simpler? If a safeguard is treated as a burden because it makes automated decision-making harder, we should at least ask whose burden we are reducing and who will carry the risk instead. During the AI Act negotiations, we at EDRi kept advocating to avoid becoming an authoritarian-scoring society. Maybe we should start looking at a scoring society that is being assembled piece-by-piece.