As Marshall McLuhan is famously quoted, “We shape our tools, and thereafter our tools shape us”.

This maxim represents particular danger to women and girls of the Global South, historically marginalized and systematically excluded from technology wielding, decision-making, or multiple opportunities to participate in creating their own unique solutions to the large-scale social problems that we now look to technology to solve. This is also particularly urgent given the scale at which Artificial Intelligence (AI) and Algorithmic Decision Making (ADM) systems are being deployed as worldwide entrenched norms - so deep-rooted that they are unconscious - become intractable as machines learn ingrained patterns from one another, wiring old norms and stereotypes into the machine learning systems of the future.

Machine learning makes information that is implicit in the data, (including "missing data" of women, girls, and other historically marginalized groups rendering them missing or invisible in the data). This invisibility then becomes explicit in the code because the machine learning "intelligently mirrors" the information it has been given from the analog world. So if gender ‘rules’ slowly being removed from the real world are being hardwired now into new AI + ADM with old and stereotypical associations of gender, race, and class, then these old models will harden and evolve into a more effective Patriarchy 2.0 ever more difficult to unwire than current colonial and patriarchal structures.

However, we have an opportunity, and a small window, to create new norms now.

But we must act swiftly. Compounded by the COVID-19 pandemic, accelerated public and private AI deployment in the North will continue to reverberate and ping pong North-to-North-to-South, augmenting the danger that the Global South will be left behind in research, application, and deployment of AI; no less AI focused on unique Global South problem definition and solution, or Global South feminist AI problem definition. Lack of research funding, capacity, brain drain, and conflicting priorities may leave no choice for the Global South but to continue to adopt technologies that have no relevant or inclusive Global South innovation, data, or model design. This will be a loss to a world that needs more invention, and more diversity in its innovation. In turn this will also further amplify the unequal futures lurking within old data and old models, magnifying the exclusion landscape for historically marginalized groups of women and girls.1

It is important to note that women and girls can and do serve as a proxy for all or any groups traditionally invisible and 'other' to the system—those traditionally left behind. Feminist AI is inseparable from 'intersectionality'—i.e. ‘the interconnected nature of social categorizations such as race, class and gender as they apply to a given individual or group, regarded as giving overlapping and interdependent systems of discrimination and disadvantage’. Our discussion is equally and powerfully (and meant to be) applicable to other forms of discrimination, most notably racial discrimination.2

We are at a critical turning point.

THE LANDSCAPE

Gender Bias in the Algorithm & Automated Decision-Making6

Inherent bias in hiring

To optimize human resources Amazon created an algorithm derived from 10 years of resumes submitted to Amazon. It used data benchmarked against Amazon’s high performing predominately male engineering department. The algorithm was taught to recognise word patterns, rather than relevant skills sets in the resumes, and seeing that males historically had been hired and promoted, the algorithm taught itself to penalise any resumes that included the word “women’s”, such as “women’s chess club captain” in the text, and downgraded resumes of women who attended two “women’s colleges”.3 This is because training data that contains human bias or historical discrimination, creates a self-fulfilling prophecy loop where machine learning absorbs human bias and replicates it, incorporates it into future decisions, and makes implicit bias an explicit reality.4

Despite multiple and unsuccessful attempts to correct the algorithm and strip out the bias which was assumed would be a simple technical fix, Amazon eventually scrapped the algorithmic recruitment tool because the bias was built too deeply within Amazon‘s past hiring practices. The bias was deeply implicit in the data that the algorithm was trained on, and the machine learning ADM system could not “unlearn” the bias.5 In 2017, Amazon abandoned the project and their experience shared with the public via Reuters.

AI permeates modern recruitment from web crawlers to identify and attract favored candidates through applicant tracking systems, resumé content appraisers, gamified and classic assessments, automated interviews and interview analysis and candidate appraisal systems. A recent report estimated that 99% of Fortune 500 companies currently use Applicant Tracking Systems of some kind in their hiring process.6 AI is expected to replace 16% of HR jobs within the next ten years,7 which means that multinational dependency on these software and processes and access (or lack thereof) to lucrative formal sector jobs will only continue to grow.

Selection bias and stereotypes

A 2019 study of Facebook’s ad delivery service found that ads for jobs in the lumber industry were disproportionately shown to white male users, while ads for cashier positions at supermarkets were shown to female users.8 Allowing job advertisers to target only men, advertisers without necessarily intending or being aware, delivered ads in a manner aligning with gender stereotypes.9 When an algorithm “learns” a pattern that more men than women are interested in lumber industry jobs (even if it doesn’t know their gender and learns this by correlating other information about a person’s likes and habits), then the system winds up deciding not to show those job ads to other women, solely because they are women.10 This exacerbates the existing stereotypes and societal barriers that have excluded women long before ADM.

After Facebook was sued for these targeting practices and as part of a settlement early in 2019, Facebook settled five lawsuits and agreed to stop allowing advertisers in key categories to target messages only to people of a certain gender, race or age group.11

However, ‘algorithms work differently in theory and in practice.’ Even though Facebook removed the ability to specifically target by race, gender, and age its complex algorithm in 2021 relies on multiple other characteristics that ultimately serve as proxies for race, gender, and age. And so therefore, it still produces results that are biased.12

Often ads seem to deliberately target in accordance with stereotypes. A ProPublica survey found that the Pennsylvania State Police in the US, for example, boosted a post targeted exclusively to men with text saying: “Pennsylvania State Troopers earn a starting salary of $59,567 per year. Apply now.”13 Targeting by sex is just one way platforms can let advertisers focus on certain users — and exclude others, inadvertently denying opportunities to apply for potentially higher paying, higher status jobs.

Implicit stereotype and unconscious bias translated into explicit misogyny

Researchers at a major U.S. technology company claimed an accuracy rate of more than 97% for a face- recognition system they’d designed - however the data set was more than 77% male and more than 83% white.14 Researchers from MIT and Stanford in the US tested three facial-analysis programmes, by IBM, Microsoft, Megvii (Face ++), and found the software was good at recognising white males but not females, especially if they had a darker skin tone.15

In 2017 a group of researchers found that two prominent research-image collections, including one supported by Microsoft and Facebook, display a predictable gender bias in their depiction of activities such as cooking and sports. For example, images of shopping and washing are linked to women, while coaching and shooting are linked to men. Similarly, kitchen objects such as spoons and forks are strongly associated with women, while outdoor sporting equipment such as snowboards and tennis rackets are strongly associated with men. Another side of this playing out is geographical paucity of data, in which a study showed that algorithms label a photograph of a traditional US bride dressed in white as “bride”, “dress”, “woman”, “wedding”, whereas a photograph of a North Indian bride is tagged as “performance art” and “costume”. The selection bias from data used to train the algorithm over- represents one population, while under-representing another.16

In 2019 ImageNet removed 600,000 photos from its system (trained on WordNet, the database of English words used in computational linguistics and natural language processing) after an art project called ImageNet Roulette illustrated systemic bias in that dataset. And in 2020, MIT permanently took down its 80 Million Tiny Images dataset when researchers discovered it labelled images with racist, misogynist content producing results that inadvertently confirm and reinforce stereotypes and harmful biases. 17

Machine-learning software trained on these datasets didn’t just mirror these biases, it amplified them. If a photo set generally associated women with cooking, the software trained by studying those photos and their labels created an even stronger association. In the researchers' tests, people pictured in kitchens became even more likely than reflected in the training data to be labelled “woman”: in a photo of a man at a stove he is labelled “woman”.18 Similarly, Researchers from the Universities of Washington and Maryland found that for some search terms, like “Chief Executive Officer” (CEO), Google presented percentages worse than the already imbalanced figures in real life. The study found that 11% of the people shown in a CEO image search were women, while the data indicates that 27% of women in the US at the time were CEOs.19

Likewise, word embeddings trained on Google News articles demonstrate gender stereotypes propagated on a daily basis. For example, extreme “she occupations” include homemaker, librarian and nanny, while extreme “he” occupations include maestro, philosopher and financier.20 Researchers from Boston University and Microsoft show that software trained on text collected from Google News reproduced gender biases well documented in humans. When they asked the software to complete the statement “Man is to computer programmer, as woman is to X,” it replied, “homemaker”.21

Similarly, when Google Translate converted news articles written in Spanish into English, phrases referring to women professionals such as professors often became “he said” or “he wrote”.22 In the Turkish language where there is no “he” or “she”, Google Translate created gender pairings where the Turkish language has none, and the results are unsurprisingly shocking – “she is a cook”, "he is an engineer", "he is a doctor", "she is a nurse", "he is hard working", "she is lazy".23 Other examples of unconscious gender bias include the prevalence of feminised machines, such as Alexa, Google Home and Siri – all of which have female voices by default (although Google Home and Siri can be switched to a male voice). However, the speech-recognition software that understands commands is trained on recordings of male voices. Therefore, Google’s version is 70% more likely to understand males, and therefore the male commands of the female ‘assistant’.24

A storied ad campaign developed for UN Women in 2013, reveals widespread discrimination against women and sexism through the use of Google search autocomplete algorithm. The campaign features close-ups of women's faces with real life autocomplete results for searches such as "Women shouldn't…" and "Women need to…". Examples of the search autocompleted to "Women shouldn't…. have rights" and "Women need to…be disciplined”.25

We would hope that things changed from 2013 to 2019, but In their 2019 paper The Woman Worked as a Babysitter: On Biases in Language Generation, researchers produced the following examples of text continuations generated from OpenAI’s medium-sized GPT-2 model:26

“We shape our tools, and thereafter our tools shape us”27

We have been slow to react to the mounting evidence - and gender bias and sexism, racism, trans- and homophobia continue to be pervasive online. “Deep fakes”, real people, real faces, close to photorealistic footage, and entirely unreal events continue to exist.28 DeepNude, an app that undresses women with a single click was launched (and subsequently removed after public backlash following media attention).29 Equally concerning, Twitter pushed sponsored tweets advertising a piece of spyware marketed to monitor girlfriends and wives with an ad reading “What is she hiding from you? Find out (sic) with mSpy!”.30 In 2021, image generation algorithms fed with a photo of a man cropped right below his neck would autocomplete the man wearing a suit 43% of the time; fed with a cropped photo of a woman, even a famous woman like US Representative Alexandria Ocasio-Cortez, it would auto-complete the woman’s image with a bikini 53% of the time.31 A 2021 free, easy to use DeepFake bot on the Telegram messaging app replaces clothing with nudity, its promoter claiming that over 700,000 images have been created. Many are under age. All are women.32

“We shape our tools, and thereafter our tools shape us”.33 This is our immediate challenge. We must establish new tools, and new norms, for lasting institutional and cultural systems change for this century and beyond. This concerns all corners of the world. It is crucial that we focus on gender equality and the values we hold dear for democracy, and for both women and men, now.

Our proposal is to seize the moment before machine learning permanently hard wires bias into global systems. We seek to mobilize a wide multi-disciplinary core of feminists who are AI researchers, Computer Scientists, Data Scientists, Machine Learning specialists, and Engineers with Economists, Social Scientists, Gender Specialists, Anthropologists, Political Scientists, Social Workers, Behavioural Economists, Philosophers, Psychologists and activists to explore new problem definitions, data, models, and feminist AI innovations through applied research. We want to create research that goes beyond describing bias or mitigating the bias that is so clearly embedded in the analogue and digital systems we live with to correcting for bias using the full power of technology to advance the values of equality we have long embraced, and the technology’s potential to forge new systems if we focus on acts of positive proactive creation.

We dream of setting research collaborations in motion that use the historically underutilized potential, imagination and skill of women and girls of the Global South, North, West and East - and with them employ effective and innovative ways to harness a technology that corrects and surmounts real life bias and barriers that prevent women from achieving full participation and rights in the present, and in the future we invent.34

Work Cited

Ali, Muhammad, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. ‘Discrimination through Optimization: How Facebook’s Ad Delivery Can Lead to Skewed Outcomes’. Proceedings of the ACM on Human-Computer Interaction 3, no. CSCW (7 November 2019): 1–30. https://doi.org/10.1145/3359301.

AZEVÊDO, DG ROBERTO. More work needed to tackle barriers women entrepreneurs face in trade says DG Azevêdo, 3 July 2019. https://www.wto.org/english/news_e/spra_e/spra275_e.htm.

Bolukbasi, Tolga, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. ‘Man Is to Computer Programmer as Woman Is to Homemaker? Debiasing Word Embeddings’. ArXiv:1607.06520 [Cs, Stat], 21 July 2016. http://arxiv.org/abs/1607.06520.

Callahan, Molly. ‘Facebook’s Ad Delivery System Still Discriminates by Race, Gender, Age’. Northeastern University, 18 December 2019. https://news.northeastern.edu/2019/12/18/facebooks-ad-delivery-system-still-discriminates-by-race-gender-age-y/.

Chivers, Tom. ‘What Do We Do about Deepfake Video?’, 23 June 2019, sec. Technology. http://www.theguardian.com/technology/2019/jun/23/what-do-we-do-about-deepfake-video-ai-facebook.

Cole, Samantha. ‘Deepnude: The Horrifying App Undressing Women’. Vice. Accessed 16 March 2021. https://www.vice.com/en/article/kzm59x/deepnude-app-creates-fake-nudes-of-any-woman.

Cox, Joseph. ‘Twitter Pushed Adverts for Spyware to Monitor Girlfriends’. VICE. Accessed 16 March 2021. https://www.vice.com/en/article/3k3wx5/twitter-pushed-adverts-for-spyware-to-track-girlfriends.

Forbes Coaches Council. ‘Council Post: 10 Ways Artificial Intelligence Will Change Recruitment Practices’. Forbes, 10 August 2018, sec. Leadership. https://www.forbes.com/sites/forbescoachescouncil/2018/08/10/10-ways-artificial-intelligence-will-change-recruitment-practices/.

‘Gender Working Group | Global Research Council’. Accessed 16 March 2021. https://www.globalresearchcouncil.org/about/gender-working-group/.

Global Research Council. Supporting Women in Research. Policies, Programs and Initiatives Undertaken by Public Research Funding Agencies. Accessed 16 March 2021. https://www.globalresearchcouncil.org/about/gender-working-group/.

Gonzalez, Anabel. ‘What’s Challenging Women as They Seek to Trade and Compete in the Global Economy’. World Bank Blogs (blog), 17 July 2017. https://blogs.worldbank.org/trade/what-s-challenging-women-they-seek-trade-and-compete-global-economy.

‘GRC_GWG_Case_studies_final.Pdf’. Accessed 22 March 2021. https://www.globalresearchcouncil.org/fileadmin/documents/GWG/GRC_GWG_Case_studies_final.pdf.

Hao, Karen. ‘An AI Saw a Cropped Photo of AOC. It Autocompleted Her Wearing a Bikini. | MIT Technology Review’. MIT Technology Review, 29 January 2021. https://www.technologyreview.com/2021/01/29/1017065/ai-image-generation-is-racist-sexist/.

Hardesty, Larry. ‘Study Finds Gender and Skin-Type Bias in Commercial Artificial-Intelligence Systems’. MIT News | Massachusetts Institute of Technology, 11 February 2018. https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212.

Kay, Matthew, Cynthia Matuszek, and Sean A. Munson. ‘Unequal Representation and Gender Stereotypes in Image Search Results for Occupations’. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 3819–28. CHI ’15. New York, NY, USA: Association for Computing Machinery, 2015. https://doi.org/10.1145/2702123.2702520.

Liu, Katherine A., and Natalie A. Dipietro Mager. ‘Women’s Involvement in Clinical Trials: Historical Perspective and Future Implications’. Pharmacy Practice 14, no. 1 (2016). https://doi.org/10.18549/PharmPract.2016.01.708.

Mahdawi, Arwa. ‘An App Using AI to “undress” Women Offers a Terrifying Glimpse into the Future’. the Guardian, 29 June 2019. http://www.theguardian.com/commentisfree/2019/jun/29/deepnude-app-week-in-patriarchy-women.

McLuhan, Marshall. Understanding Media: The Extensions of Man. Routledge, 1994.

Merrill, Jeremy B. ‘Facebook’s Algorithm Makes Some Ads Discriminatory—All on Its Own’. Quartz. Accessed 16 March 2021. https://qz.com/1588428/new-research-suggests-facebooks-algorithm-may-be-discriminatory/.

Olson, Parmy. ‘The Algorithm That Helped Google Translate Become Sexist’. Forbes, sec. Tech. Accessed 16 March 2021. https://www.forbes.com/sites/parmyolson/2018/02/15/the-algorithm-that-helped-google-translate-become-sexist/.

Perez, Caroline Criado. Invisible Women: Data Bias in a World Designed for Men. First Printing edition. New York: Harry N. Abrams, 2019.

———. ‘The Deadly Truth about a World Built for Men – from Stab Vests to Car Crashes’. the Guardian, 23 February 2019. http://www.theguardian.com/lifeandstyle/2019/feb/23/truth-world-built-for-men-car-crashes.

Qu, Linda. ‘Report: 99% of Fortune 500 Companies Use Applicant Tracking Systems’. Jobscan (blog), 7 November 2019. https://www.jobscan.co/blog/99-percent-fortune-500-ats/.

Scheiber, Noam. ‘Facebook Accused of Allowing Bias Against Women in Job Ads - The New York Times’. New York Times, 18 September 2018. https://www.nytimes.com/2018/09/18/business/economy/facebook-job-ads.html.

Scheiber, Noam, and Mike Isaac. ‘Facebook Halts Ad Targeting Cited in Bias Complaints - The New York Times’. New York Times, 19 March 2019. https://www.nytimes.com/2019/03/19/technology/facebook-discrimination-ads.html.

Schiebinger, Londa, and ineke klinge. ‘Gendered Innovations: How Gender Analysis Contributes to Research’. “Innovation through Gender. EUROPEAN COMMISSION, 2013. https://ec.europa.eu/research/science-society/document_library/pdf_06/gendered_innovations.pdf.

Sheng, Emily, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. ‘The Woman Worked as a Babysitter: On Biases in Language Generation’. ArXiv:1909.01326 [Cs], 23 October 2019. http://arxiv.org/abs/1909.01326.

Song, Victoria. ‘MIT Takes Down Popular AI Dataset Due to Racist, Misogynistic Content’. Gizmodo, 7 February 2020. https://gizmodo.com/mit-takes-down-popular-ai-dataset-due-to-racist-misogy-1844244206.

Tobin, Ariana, and Jeremy B. Merrill. ‘Facebook Is Letting Job Advertisers Target Only Men’. ProPublica, 18 September 2018. https://www.propublica.org/article/facebook-is-letting-job-advertisers-target-only-men?token=xbvF5KLcDIV6vr6B2AF9D0LlUK_IwLni.

UN Women. ‘UN Women Ad Series Reveals Widespread Sexism’, 21 October 2013. https://www.unwomen.org/en/news/stories/2013/10/women-should-ads.

UNESCO. ‘I’d Blush If I Could: Closing Gender Divides in Digital Skills through Education’. UNESCO, 18 March 2019. https://en.unesco.org/Id-blush-if-I-could.

‘What Is Automated Individual Decision-Making and Profiling?’ ICO, 4 January 2021. https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/automated-decision-making-and-profiling/what-is-automated-individual-decision-making-and-profiling/.

Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. ‘Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus-Level Constraints’. ArXiv:1707.09457 [Cs, Stat], 28 July 2017. http://arxiv.org/abs/1707.09457.