The Banality of the Evil Algorithm, or Why we Should Teach Ethics to Programmers

Have you been following the news recently? How about the latest scandal, related to automated decision making, affecting people’s lives and futures? Depending on when you’ll read this, the story might be different. But there’s probably an AI story like this going on in your timeline, right?

A-level fail

As of writing, it’s the A-level algorithmic meltdown in the UK. What happened? In short, students in the UK could not sit exams this year, because schools were closed due to coronavirus. The government had to figure out a way to grade them, especially the A-level students. Enter the algorithm – teachers were asked to supply an estimated grade and a place within the school ranking for each of their pupils. 

This information was then crunched based on the past performance of the school to avoid too many students receiving top grades. In other words, not based on the current performance of the student, but the school’s ranking so that fewer students will be in a favourable position. What could go wrong?

Well, a lot, as it turns out. When the results were out, around 40% of the A-level students’ grades in England, Wales and Northern Ireland were lower than the teachers’ initial assessments. Some were even down two grades. And the grading was affecting state school disproportionately more than private ones. In other words, it affected poor students more than it did their more affluent peers.  

‘F*ck the algorithm’

This unfortunate example sums up all that’s wrong with automated decision making. Ultimately, it is quite likely that an algorithm would represent a bias towards a specific categorisation, which is, in turn, based on specific past experiences. Surely, a favourable bias for underperforming private school pupils, but not that much for bright students in underperforming schools.

After a mass uproar, which involved students protesting and yelling “F*ck the algorithm”, among other things, the decision was reversed, and the teachers’ initial grades were adopted. The blame for this whole commotion was effectively put on “the algorithm”.

Why blame code for this?

However, there isn’t a reason why ‘the algorithm’ is to blame for any of this. It was a centralised decision that utilised technology and then backfired tremendously. It would’ve failed with or without automation, probably just not as fast and certainly not as public. As in almost any similar scenario, the idea is to blame, not the technology that’s been used to make it happen.

But does that mean that all professionals involved in the execution, such as software engineers are blameless for all the current and future adverse impacts, related to their code? Not at all. 

The banality of evil

What people sometimes fail to grasp is that they are still responsible for the outcomes of their actions, even if an authority has ordered the actions themselves. Hannah Arendt refers to this as the banality of evil – mindless acts of unintentional evildoing that are masked as compliance with authoritative orders. 

Don’t get me wrong, I’m not comparing programmers to war criminals. I’m merely acknowledging that it’s easy for a person to forget her responsibilities of doing the right thing and focus on obediently following orders that would potentially benefit a career trajectory. 

You can have the power, but you’ll also have some responsibilities

It’s a play between power and responsibility. I can assume with a fair amount of certainty that most people who code are well aware of their power. A fair share of them might even base their decision of going into programming on some conscious or not promise of what will happen once they’ve learned this trade. It’s the promise of being able to command machines in a world that’s increasingly dependable upon machines. 

At the same time, it’s particularly easy to hide behind terms such as ‘the algorithm’ when talking about responsibility. Partly because most of us aren’t entirely sure what “the algorithm” even means in the specific case. Is it just an “if” function or will it fool me if we meet in a chatroom? No matter the level of sophistication though, its basic decision-making process can be traced back to either a specific person or to a group of people. 

Let’s teach ethics to programmers, shall we?

The social and economic power of breeding algorithms needs to be grounded by the concept of responsibility. And apparently, some computer science programmes have already acknowledged this by teaching mandatory ethics classes to their students. 

Inspired by this finding, I had a discussion both online and offline on why we should teach programmers ethics. It ended with an almost unanimous agreement that we need to teach them what good means so that they’ll be able to distinguish it from evil. Because the odds are that at some point of their future careers in computer science, they’ll have to make decisions that will have either terrific or tremendous consequences for some people or communities. 

Also, why restrict ourselves to only thinking in terms of people’s professional lives? The truth is that computer science can give you tools to be quite unethical in your personal lives as well. Think about it – the harm that can be brought upon a group of people can just as easily be experienced by a person if targeted specifically. 

The banality of responsibility 

It’s possible to teach a person what good is, and therefore provide her with the tools to evaluate where her actions lay on the spectrum. But is it possible to ever teach a person that having the power to do something doesn’t mean that this something has or needs to be done? And what does power even mean in the context of employment and subordination?

Fundamentally, I don’t believe that a person is either good or evil. Most of our actions are contextual and are caused by a complex interplay of factors that are often invisible to any observer. Which is, and I’m aware of the irony, great news for the people, trying to excuse their wrongdoings by offering subjective explanations. 

However, it is also great news for the people who are refusing to participate in something, even if they’re contractually required to do so. Recognising and doing good matters even when the context favours evil. Although far too optimistic considering the harsh business realities of our times, I do believe that acknowledging your vital part in a specific structure is a reasonably good start. Would ethics classes be enough for this? Probably not, but you wouldn’t turn down a compass when you’re lost.