Discrimination occurs when a person is denied an opportunity or a right due to a certain opinion based on groundless and inappropriate reasons. Sometimes, artificial intelligence systems are not impartial and show prejudices that result in discriminatory algorithm outputs against an individual or a group of individuals due to their race, gender, age, etc.
From a scientific perspective, this phenomenon is known as “bias”, i.e., a distortion caused by prejudice. This does certainly not concern only algorithms, but also human minds.
Humans have cognitive bias, i.e., systematic distortions of judgment that, substantially, derive from two different sources: the first one is biological, while the second one is the result of the cultural and social context in which an individual grows up and lives.
We can affirm that the first source is “infrastructural” and the second one is “informational”.
For example, the order in which options are presented conditions human decisions: the human brain “prefers” some things over others because they are presented in a certain position compared to others. An example is the way products are presented on supermarket shelves or the way movies are presented on platforms. This is “infrastructural” bias.
Typical “informational” bias is gender bias. If we are used to see men at top positions, we could be inclined to incorrectly believe that women cannot reach leadership positions.
Artificial intelligence systems are potentially subject to the same distortions or bias that condition humans.
An artificial intelligence system being used to suggest the most suitable applicant for a vacancy could provide unappropriated suggestions due to infrastructural reasons related to the algorithm, which could have some defects causing a certain characteristic to be under- or overestimated. Otherwise, it could provide a wrong or discriminatory suggestion due to informational reasons, i.e., reasons related to the data used to “train” the system.
In fact, data could present distortions just like the environment in which humans grow up: in fact, they are a product of such environment. Hence, the expression “garbage in garbage out”: if data are of poor quality, algorithm predictions will be of poor quality, too, and will generate errors or distortions.
A typical area in which bias can be found is that of equality between men and women or between people of different races or religions, because our economic and cultural system has been based for millenniums on discriminatory behaviours, which are often supported by legal systems.
If female applicants in a certain company have always received a lower remuneration than male ones, an algorithm used by an artificial intelligence system that was trained on those data will be inclined to propose women a lower average remuneration. This kind of distortion is called historical bias and, in many cases, it is the result of a social inequality in training data.
The emerging Artificial Intelligence Act (please refer to Carlo Giuseppe Saronni’s article in this TopHic issue) should also deal with this kind of bias.
In fact, art. 29 (a) of the text proposed by the Parliament provides for an impact assessment on fundamental rights, based on which the system user should make an assessment on high-risk systems aimed to identify potential breaches of fundamental rights, including the right to non-discrimination.
As said, bias can also depend on a poor representativeness of data.
In general, the more the data received by an artificial intelligence system on a certain area or matter or on a certain population, the more solid and reliable its prediction and capacity to represent it properly.
Consider, for example, racial discrimination: if few data are available for a certain race, the system prediction on that race will not be fully reliable since the system is trained on few information.
Then, if characteristics (e.g., low income) are constant for that race, the system will associate a certain race with a low income also in opposite circumstances, thus damaging persons belonging to that group.
To solve the low representativeness of data, the regulation should provide under art. 10 (3), that “data must be representative” of the population which the algorithm has effect on.
The rationale behind the regulation is commendable.
However, it is difficult to concretely comply with this provision.
In fact, data available to the algorithm will always depend on the reference population up to that moment and, therefore, it will sometimes be difficult to avoid some cases of misrepresentation.
Lastly, some kinds of bias depend on the type of data used.
There are some human characteristics that can hardly be represented.
Intelligence, moral sense, empathy, the ability to conciliate people are aspects that can hardly be represented by measurable data and, subsequently, evaluated, for example, by a personnel recruitment algorithm.
In this case, again, the output provided by the machine could be discriminatory or non-efficient for the user, since it could lead to the selection of a less deserving applicant only because fundamental qualities are not captured by data and therefore are not considered.
One last example.
Intelligence is a complex of mental faculties that distinguishes humans from animals.
IQ can be measured. However, IQ regards a very limited aspect of human intelligence, i.e., logical abilities (elementary inferences that involve short-term memory) and spatial visualization (most of all, rotation and pattern recognition).
These abilities, either innate or trained, certainly favour those professions that are more based on spatial reasoning, such as mathematicians and physicists, who have a higher preparation in geometry. These individuals will be favoured by an artificial intelligence algorithm that measures human intelligence based on IQ.
In fact, even if the machine is very sophisticated, it needs an element to be measurable, in order to give it a value.
In this sense, IQ is a measurable, established, and privileged parameter to quantify intelligence.
Nonetheless, this parameter cannot properly capture the whole dimension that it aims to measure, i.e., intelligence, since measurable data could weight more than other important – but non easily measurable – personal qualities.
Therefore, the evaluation of intelligence based on IQ represents a synthetic but not exhaustive solution to represent the characteristics of an applicant, because the system could exclude deserving individuals due to the impossibility for its impossibility to process hardly measurable aspects of intelligence.
Clearly, these are very sensitive issues.
Data scientists cannot be left alone in considering these problems but should rather be supported by jurists and philosophers who can help them identify the possible bias in the artificial intelligence system and the best technologies to mitigate them.
Luckily, large companies are recruiting these professional figures, who should help data scientists organize their work.
