Skip to main content
SearchLoginLogin or Signup

Diagnóstico y mitigación de sesgos desde América Latina | E.D.I.A

In this phase of the project, we developed a tool for bias assessment in natural language processing, more concretely, in word embeddings and large language models.

Published onMay 06, 2024
Diagnóstico y mitigación de sesgos desde América Latina | E.D.I.A
·

(Diagnosis and mitigation of biases from Latin America)

Prototype for E.D.I.A —

Team and Country: Fundación Via Libre - Universidad Nacional de Córdoba, Argentina

Research Team: Luciana Benotti, Laura Alonso Alemany, Beatriz Busaniche, Hernán Maina, Lucía González, Lautaro Martínez, Amanda Rojo, Mauro Schilmann, Matías Bordone, Alexia Halvorsen

Date of presentation to f<A+I>r: first call (end of 2021)


Synthesis

In this phase of the project, we developed a tool for bias assessment in natural language processing, more concretely, in word embeddings and large language models. This tool does not require any technical skills, thus discrimination experts can work at the core of the problem, to prevent harms before they are actually deployed as products.

The prototype is available as a graphical, stand-alone tool, in the Huggingface online platform, a standard for natural language processing tools:

https://huggingface.co/spaces/vialibre/edia

The code and documentation of the prototype are available at:

https://github.com/fvialibre/edia

We created a 3-minute video for general public, describing the problem and our approach to address it:

E.D.I.A: Estereotipos y Discriminación en Inteligencia Artificial

In the paper stage, we analyzed a preliminary approach to the problem, working together with an interdisciplinary team and a variety of users. We detected some limitations and needs from improvement, which were addressed in the development of the proof of concept of the prototype.

When the proof of concept was developed, we carried out two workshops at the Universidad Nacional de Córdoba in the city of Córdoba and the Facultad Latinoamericana de Ciencias Sociales (FLACSO) in the city of Buenos Aires. The workshops were attended by approximately 70 and 30 people respectively, and were carried out with interdisciplinary teams in varied areas of discrimination, in IT industry, academia, public institutions and civil society. After those workshops, we crystallized the insights in the functionalities and approach of a prototype. Discrimination experts at the workshops expressed keen interest in incorporating this tool to their activities. Our pilot includes working with these groups, to consolidate their adoption of the tool and methodology.


Problem

The goal of this project is to provide a tool and a methodology for discrimination experts to assess biases in key natural language processing artifacts, more concretely, word embeddings and large language models. These artifacts are at the core of many everyday applications, determining their outcome. They are inferred from big volumes of text, and thus crystallize the biases in those texts, and carry them into their behavior in downstream applications.

Many techniques have been proposed to address this problem, but most of them require skills in programming, statistics and even very specific, highly technical expertise in natural language processing techniques. The technical skills required to apply these previous approaches to bias assessment preclude that experts in discrimination are engaged in the process of auditing these systems, as they don’t usually have developed all of these skills, or not enough as to be comfortable manipulating these complex proposals.

Although it is known that discrimination experts need to be involved in the development cycle of artificial intelligence applications to prevent societal harms, the truth is that technical barriers prevent them from participating in the core of the development process, and are thus relegated to a peripheral role, for example, harm assessment, where their expertise has much lesser impact than if they could participate, for example, in risk assessment, preventing harms from actually occurring.

In this project we have developed a prototype that allows discrimination experts to assess biases in complex artifacts at the core of the language technologies process, showing that technical barriers are avoidable.


Findings

Our main findings at this stage of the project are the following:

  • It is not necessary to resort to technically complex concepts and programming skills to carry out a useful, actionable assessment of biases in word embeddings and large language models.

  • Discrimination experts can obtain very valuable insights from the exploration of biases through our prototype and with the proposed methodology, providing them with strong evidence to plan actions to mitigate those biases in downstream applications.

  • Discrimination experts find that the obtained insights are also useful to validate (or refute) their intuitions on discrimination, and argue with other actors in the discrimination scenario.

  • The funding to carry out this project has allowed us to work with our own priorities, not the usual requirements of academic publication, like standard metrics and datasets that are irrelevant to our local context. Historically, such requirements have taken most of the resources from research projects, and have prevented us from pursuing an agenda that is locally relevant, instead of aligned with the global north agenda.

We worked with users from the very first stages of development, to identify concepts and methods that could be technically complex for a diverse audience, and test alternative ways to carry out the bias assessment avoiding or circumventing such barriers. In the paper stage of the project we identified the following technical barriers:

  • A focus on metrics as the main output of the process. Metrics are opaque and counterintuitive for non-technical users (and for a good part of technical users as well). At the paper stage we hypothesized that metrics need not be the main way to assess bias, among other reasons, because they have been shown to be poor representations if the conditions for measurement change, for example, to measure different biases, with different lists of words, or different word embeddings.

As an alternative to metrics, we provided a number of visualizations to carry out a more intuitive assessment of bias. These visualizations can be integrated as a complement to metrics, but users have also found them as useful devices to carry out the exploration process and to obtain conclusions that can be effectively communicated.

  • Mathematical or statistical concepts have been used to describe the process of representation and assessment of bias. Such concepts tend to create the impression that people who have not studied them cannot understand the process.

We have devised ways to address the process of bias representation and assessment without resorting to specific mathematical, statistical or machine learning concepts, instead replacing them with more mundane, intuitive concepts. We then tested these terminologies with users and found them to be satisfactory.

  • The role of mitigation in the process has been very central in previous approaches, but we find the way mitigation is addressed in other works relegates bias assessment to a peripheral role, similar to the shallow patches that are currently being employed in productive applications. We find that assessment should instead be carried out at early stages in the development of language technologies, so that more integral, preventive measures can be taken, like using other word embeddings or language models, re-training them, or re-thinking the approach altogether.

  • Modeling words in isolation has been found very satisfactory for some parts of the process, but more context was needed in other parts of the process, so we incorporated:

    • relations of the words with other words, in a graphical way,

    • relations of words with their contexts of occurrence in texts,

    • relations of words with respect to different biases at the same time in the same graphical representation,

    • relations of words with their possible contexts of occurrence in language models, that is, their probability of occurrence in different contexts according to language models,

    • the ability to characterize multi-word expressions.

Implementation and management

It is very difficult in the current context to create and consolidate software development teams. However, we have been able to work with an interdisciplinary team, and incorporate people addressing different needs as they were arisign. For some of the members of the team, the project has been a formative experience.

Some of the priorities of the project changed as experience came. We had initially planned resources for UX - UI design and development, but the session with Sasha Costanza-Chock right before the prototype phase made us rethink this planning. We instead decided to develop a proof of concept without investing much in UX-UI, and obtain feedback from high-stakes users. This feedback was then instrumental to make the prototype more usable and a better fit to the needs and expectations of users.

Outputs and dissemination

A working prototype

The main output of the project is a graphical prototype of our tool in the Huggingface online platform, together with video tutorials that explain a recommended methodology to use it, so that users who have not attended a workshop can begin to use the tool autonomously:

https://huggingface.co/spaces/vialibre/edia

Code and documentation of the prototype

Together with the graphical interface, we have also released the code and documentation of the prototype, so that the tool can be implemented and used in any context, by people with programming skills:

https://github.com/fvialibre/edia

We have also provided a dockerized version of the tool.

Dissemination for the general public

We have produced a 3-minute video to introduce the problem and the tool to a generic audience: E.D.I.A: Estereotipos y Discriminación en Inteligencia Artificial.

We have also presented the problem and our proposal, including the tool, at a number of general public sensitization talks about artificial intelligence, media interviews and our general educational task.

Dissemination in academic venues

Part of the team has a strong academic background, wherefrom part of the motivation for this work arises. As we believe that a paradigm change is needed in our academic area, we have made a special effort to disseminate this work in academic venues:

  • we have made publicly available a paper providing an analysis of existing approaches to bias assessment, their limitations to be used by discrimination experts, and our proposal to overcome them, targeted to an academic audience, published in arXiv, a well-known publication site for the relevant academic community:

02. A methodology to characterize bias and harmful stereotypes in natural language processing in Latin America.pdf

Workshops with interdisciplinary teams

We carried out two hands-on workshops with interdisciplinary teams, totalling 70 people at the Universidad Nacional de Córdoba (September 26th 2022) and 30 people at FLACSO (Facultad Latinoamericana de Ciencias Sociales, sede Buenos Aires) (October 3rd 2022). Attendants were industry experts on NLP, from big and small industries, social scientists from different areas of expertise (for example, researchers from communication science with expertise in discourse analysis of gender violence in written media, researchers in psychology working with ageism and ableism, researchers in nutrition epidemiology working in the perception of a healthy diet as represented in social media), members from public institutions (the ombudsman’s office, people in charge of providing seminars for congresspeople), and some philosophers, students (both undergraduate and graduate), professors in data science and bias assessment in data science enthusiasts.

The workshops consisted in 1) a 30 minute introduction, 2) creation of mixed teams, with at least one data scientist and at least one discrimination expert, 3) a hands-on session with 5 to 10 facilitators, where the teams carried out the exploration, 4) sharing of the results of the exploration so far, and 5) how to keep contributing to the project and integrate these tools in their everyday practice.

The main objective of these workshops was to obtain feedback about the usability of the tool, and to evangelize on the approach and methodology, as well as to communicate the existence of the tool. In addition, as a further result of these workshops, we established contact with some teams that were very keen to carry out a systematic integration of this process in their current practice:

  • a small local industry that develops chatbots,

  • a Congress office that process Congress documents,

  • a research team in nutrition science, who work in the perception of health in dietary choices in social media,

  • a research team in communication science, who work in the study of gender violence in written media

We think it would be very beneficial both for our project and the development of the ethical practice of these teams to give continuity to this starting contact in a pilot phase.

We also provided a simplified version of this workshop, without the facilitators for the hands-on part, but with the exploration in teams and sharing of results, for a local women in technology association and in the Artificial Intelligence Day at the Universidad Nacional de Córdoba.

We are also planning to carry out a hands-on technical workshop at the Khipu school for Machine Learning, March 2023, in Montevideo, Uruguay. This school is a very important venue for machine learning students and practitioners across Latin America. We have also presented a workshop proposal for 2023 Rightscon.

Training of the research and development team

In the process of development of this project, two junior and a semi-senior computer scientists have developed their skills in natural language processing in general and in bias assessment technologies in particular, with constant interaction with seniors and consultants, and taking active part in the decision making process.

Impact

This prototype and the workshops where it has been put into practice come to show that technical barriers in this problem amount to gatekeeping practices, even if inadvertently.

Our dissemination efforts have made the problem understandable for a wider range of people in very diverse communities, from industrial to civil society to public institutions and even academia. We have conveyed the message that the problem is approachable, we have shared a basic methodology to address it, and we have carried out hands-on practices working with the tool. The tool is readily available for participants in practical sessions. In addition, the tool is accompanied by videotutorials so that people without previous experience with it can learn the rationale behind the tool and use it.

Moreover, the tool is readily available as a software package, together with documentation, for developers to implement it in their work environment.

Recommendations

The f<A+I>r call for projects has been a unique opportunity for our team to mature and develop an approach to a central problem, from a perspective that is not subject to the agenda of the global north academia and the need to publish at the few prestigious venues. Such venues impose methodological requirements, like experiments on standard datasets, with standard metrics, that are inadequate or simply irrelevant to the issues that are relevant to our Latin American context. We have experienced growth as an independent team, with an ability to set our own agenda, that we had not experienced before. In this sense, the f<A+I>r alliance has provided us with a whole new sense of direction that we are now developing in our home institutions and venues, with a renovated sense of purpose in our professional activities.

The management of the whole project from the f<A+I>r alliance has been very smooth, both in the disbursement of the funds and in the follow-up of our project, which has been very useful at all times, with invaluable feedback from organizers and other teams of the cohort. The organizers (Paola, Jaime, Mariel and Caitlin) have been ever supportive and sharp in providing insights and criticism, and addressing our doubts.

On the negative side of project management, we have suffered from a certain lack of clarity in the objectives and timeline of the project. We are currently very worried that a lack of continuity with the pilot phase will not allow us to keep our team engaged, as we will not be able to keep junior and semi-senior developers on roll and they will need to move on to other activities, and possibly we will not be able to re-engage them. We are also worried that we will not be able to keep engagement with the interdisciplinary teams that are interested to participate in the pilot phase.

Comments
0
comment
No comments here
Why not start the discussion?