IGF 2025 - Day 1 - Workshop Room 3 - WS 219 Generative AI and LLMs in Content Moderation Rights and Risks

>> MARLENA WISNIAK: Hello everybody and welcome to our session on generative AI and LLMs in content moderation. Great to see you all here this person and welcome to the folks joining us online.

My name is Marlena Wisniak.

I'm really thrilled to be here today with my esteemed panelists. From right to left. David Sullivan.

>> Digital trust and safety partnership.

>> MARLENA WISNIAK: And David can share 30 seconds of what it is later.

(?) and to my left. (?).

Today we'll talk about a topic really emerging and lot of acronyms. Apologies in advance. GenAI. LLMs. Being based in San Francisco it is something people talk about daily and it seems farfetched but we already see that in the world today. I'll share a few key takeaways from our emerging research. And then we'll hear from folks on the panel about different use cases. Different risks. We'll hear a regional perspective from Madwa. And really try to bridge the technical and societal aspects of LLMs and placing it into everything happens in the world including geopolitical developments.

So this topic is really relevant. We've been working on content moderation the past five, six years and LLMs have become really an interesting development. And interesting from human rights perspective means both potential good as well as alarming use days cases. I'd say one of the biggest issues for LLMs which is a subset of generative AI trade on vast data. Is promise of efficiency and adaptability they also pose serious risks. Automated content moderation as many folks in the room probably know pose already a lot of human rights risks and cause violations and LLMs have become, at least in silicon valley and increasingly presented as a silver bullet for solving these issues.

However, research has shone that they can reinforce existing systemic discrimination, censorship and surveillance. And one of the most pressing issues that we've found and ECNL has conducted research on this topic the past year. Working with hundreds of different focus across civil society, academia, industry. On mapping impacts for on content models. One of the most pressing issues is concentration of power. So the way then LLM used for content moderation works today is there is a handful of companies that develop foundation model, or LLMs. The one focus probably most aware is ChatGPT. There is also Claude, Gemini, lama and others. So there is at the AI developer level.

Then the glory level which are often smaller social media platforms. Discord, Reddit, Slack. They often will not have their own LLMs. But they will use other LLMs like the ones I mentioned and fine tune them for their own purposes.

So what does this mean? Any kind of decision made at the foundation level let's say defining Palestine content as terrorist content will trickle down. Unless it is fine tuned.

What this means globally is content moderation defined at the foundation level will also be replicated on the deployment one and really even more homogeneity of speech as before.

However, alternative approaches are emerging and we've seen throughout our research that there are community‑led initiatives, especially in the global majority that focus on public interests AI that is culturally informed and really decentralized.

These models, though smaller in scale demonstrate comparable performance with LLMs and highlight the potential for more space moderation.

Our report, which our friends at IGF can maybe post online and I encourage you to cheque out. Look at the various impacts from privacy, freedom of expression, information, assembly association. No non discrimination, participation and remedy.

We'll talk about some of these later: But I encourage you to read it. Thorough analysis of each of the rights and last part is recommendations which will also dive into this session. And just to note that we'll have ample time for questions. I really want to hear from folks in the room as well as online. I error see a few experts on this topic. And also encourage everyone to participate, even though it seems like it is brand new. A lot of the questions around large language models have been around for a long time. But even offline or human‑led moderation. These questions are often the same and they are just exacerbated and accelerated due to the scale and speed of AI.

With anyway said, I wanted to turn it to David to share a few use cases of LLMs for content moderation and the TSBBSR, business for social responsibility has led research on the topic. So David if you could share findings and introduce your work.

>> Thanks Marlena. Take these off for a moment. It is great to be here with everyone. I'm David Sullivan. I lead the digital trust and safety partnership which brings together companies providing all different types of digital products and services. Including those of Google or Meta as well as smaller players such as Discord and Reddit. And our companies come together around a framework of best practises for trust and safety. A framework that is content and technology agnostic. And the idea is basically that companies can come together around the practises they use to develop their products, develop the governance and rules for product, enforce the rules, improve over time and be transparent with their users and with the public.

So it is about the practises of safety as opposed to agreeing on what type of content should be favored or disfavored on these kinds of digital services.

So last year in 2024 we brought together a working group of partner companies on best practises for AI and automation and trust and safety. So that looked at the full range of technologies from, you know, even the most basic kind of rule‑based systems that have been used as part of trust and safety going back 20 years dealing with things like spam. To, you know, possibilities for use of generative AI as part of trust and safety. Of which content moderation is kind of one component.

And so we spent the better parking lot of a year looking at what companies were doing and trying to identify some best practises. As well as what we called generative AI possibilities. Ways companies might be experimenting and beginning to use this technology as part of trust and safety. As well as what the limitations and challenges and what ways to overcome challenges. I'd encourage people to go to DTSpartnership.org and that is right on the front page. We worked with a team at BSR helped us do that research. Hanna from that team is here and also a expert in the space.

So I want to just briefly mention a few things. First, as I think I already said. I think that use of trust and safety ‑‑ use of AI and automation in trust and safety has always been a blended process. Of human and technology. That's always been the case. And it continues to be the case. Even as what that blend looks like may change quite substantially as LLMs and generative AI technologies get incorporated into trust and safety.

Second is that perfection when it comes to content moderation is nearly an impossibility. And so we're always thinking about potential for overaction or underaction. When it comes to how companies are enforcing their policies. And we can talk a little bit more about some of the tradeoffs there. And I'm sure we'll talk a lot about that with this group here.

So with that in mind, I wanted to just use our framework of these five overarching commitments to talk about five examples of kind of possibilities for the use of generative AI as part of trust and safety. And hopefully those will help kick off some discussion. So the first of our commitments all offer company members make is around product development.

So one example of a generative AI pocket in product development is use of generative AI to enhance and inform the kinds of risk assessments that companies do when they are developing and rolling out new products or new features within products. Generative AI could help identify emerging patterns. Could help identify edge cases that would then become more mainstream.

You connect data points between different risk factors and could potentially be use as part of red teaming exercises by trust and safety teams. Brainstorming attack scenarios and things like that. That is product development.

The second commitment all our companies make is to product governance. So as part of that commitment, some of the best practises we've identified are around external consultation. Incorporating users perspectives into company policies. And consulting with the civil society organisations and other external groups as part of developing and iterating those policies. So there again, I think LLMs could potentially be leveraged to gather many you have more data.

You have companies currently using kind of surveys and focus groups and getting information from outside experts and all of that may not always be coherently brought together as I could be.

And one other things LLMs help to create the kind of feedback loop where those organisations that spend a lot of time telling companies what they should and shouldn't be doing with policies would be able to hear this is how your input was used in the development of 234 new content policy around whatever issue.

On enforcement. So there I think one of the things that where I am cautiously optimistic is about the potential for generative AI to augment human review, as opposed to replace it. We hear a lot these days about AI replacing humans when it comes to content review. But I think the area where there is most potential, both in terms of shielding humans from having to review the worst of the worst type of content. But also being able to help provide context to human reviewers that maybe will help them with their decision making and also being able to route the things that are easily determined to be content violating away from humans to then make their own work more efficient. And question talk of course about a lot of the challenge there is as well.

Just quickly on improvement. I think one thing GenAI can do is sort of enhance the automated evaluation of context around violations. So it can be hard for companies to be able to... let's see...

Basically the idea is being able to have more information at your disposal in order to figure out how your policies are actually being implemented in practise. And to incorporate that context into automated actions, as well as that sort of information for human reviewers.

And then lastly, on transparency. There I think there is also potential for generative AI to improve the explainability of the decisions that companies are taking. So I think maybe all of us at one point or another have had an experience of having something you have posted, finding it violates some services guidelines one way or another. And when you appeal those things, you get very little information in return.

So there is potential for these types of technologies to be able to provide a little bit more information. So the example, for example, would be, you know, if you have posted a video that's an hour long, it could tell you here is the two minutes that we found to be violative. And you can have a chance to correct that.

So those are I think some possibilities, some positive use cases. We're going talk a lot more about the limitations and challenges. I just wanted to mention just a couple of them. And this is where I think the stakes gets very high. Is that we don't know what kind of tomorrow's content crises are going to look like. And we know these models are not good when they are dealing with novel challenges that are not adequately represented in their training data. That is when they really go off the deep end.

So we need to be aware of that. The second thing is that for all companies that have trust and safety operations that have their content policies, they need to exist in three different forms. There has to be a public‑facing version for users to understand what is allowed and not. There has to be the internal detailed "these are the specification of how we enforce this policy," which you don't want to make completely public. Because bad actors can use that to kind of gain the system.

And then you need a version that is machine ratable, can be used by LLMs. So that is a complicated sort of balancing act and one that also complicates these challenges.

Lastly, I think there are just tradeoffs when it comes to the metrics that companies use here. So you can, you know, optimise for precision, which is really the meth Rick about how correct your decisions are. Or you can optimise for recall, which is about as much ‑‑ covering as much content as possible.

These are things that have consequences about digital services and impact. Sour constantly having to balance. We need to make sure we get as much of the really harmful content as possible, so maybe you are okay with false negatives in that situation. Where as in other situations you want to worry about false positives. So nose are real tradeoffs you. Can't just wish them away.

So I think hopefully that maybe helps kick things off. And I'll stop there. To give others time.

Thanks.

>> MARLENA WISNIAK: Thanks David. And can everybody hear us?

Because the mic situation a little off. But you have to wear the earphones to hear.

Next we'll hear from don Raj about multilingual models in particular. Lot of this research is often on English content. And some colonial languages. But hopefully and surprisingly to focus in this room that is not the case across languages. There is a lot of inequities. And CDT really over the past few years has done groundbreaking research on this topic. It has informed our own research as well so I'm thrilled to have you don Raj here and also shout out to Aliya Bhatia who is not here with us today but has done some great research.

>> Thank you. And thanks for the invitation to join the conversation. Yeah. I'm Don Raj Takur, research ‑‑ (?) based Washington, D.C. and Brussels. We're non profit. Tech policy advocacy group.

Focussed on range of issues. One of which is our own content moderation.

Great, so yeah, just a follow‑up on then what David discussed on like the application of large language model, content analysis and trust and safety systems, I can talk a bit more specifically about how those are applied, how those systems and technologies are applied in will further discuss and explain is low resource languages.

And this is based on research that as Marlena mentioned, (?) has done ‑‑ (?). And report on case studies. Looking at different (?) languages.

86 when we talk about multilingual large language models, what we're essentially focussed large language models trained on text data from several different languages at once. And logic or claim researchers make with these kinds of models is that they can extend the various multi faceted capabilities and benefits, for example, that David highlight, to languaging (?) English. And even to languages which there is little or no text data available. So you can also then see how you can apply some of these kind of benefits to many of the content, user‑generated content from various kind of languages around the world.

That said, there are several issues and challenges that come up. And that is what I'll spend a bit more time talking about. Skipping over the potential benefits relating David's covered quite well.

So lot of sources also show that multilingual language models also struggle to deal with what is called there is this wide disparity between languaging and how much text data is available. Researchers describe or categories of high resource and low resource languages.

English has by multiple magnitude much more text data than any other language. And there is lot of reasons behind that. You can think of the legacy of British colonialism, American neocolonialism and subsequent (?) regional and indigenous language, mostly the companies we're discussing now when we talk about frontier model companies are based in US as well. As well as social media companies. So English becomes dominant language there.

So what we call high resource language effectively refer to languages where significant amounts of text data available. Such as English and many other European and other languages. Chinese, Arabic, for example.

On the other hand, other side of the spectrum, low resource languages, very little textual data available. But these can still be major languages in terms of number of speakers. Can include, for example Swahili, Tamang or Arabics. Also like Indonesian, literally hundreds of millions of speakers.

What this leads to then is disparity or inequity in potentially of these technologies to social media but other use cases. (?) as applies these kind of languages. But (?) another type.

One questions comes up in lot of our work. Not so much the technical capability of these two technologies but also how they are incorporated in trust and safety systems. And so here we come across several different kinds of problems. So low resource language, for example, we have this problem of lack of training data. But often that lack of training data is not just for content in general but can be for specific domains. In our research, for example, we spoke to many, we spoke to LLM and LLP researchers. Highlighted it wasn't just a problem of having enough data and catch available generally for developing. But speech such as hate speech. If hate speech a concern, for example, for particular application, then you need particular data on that as well. And that is also part of low resource problems so to speak.

Many users. And this is think well known fact. People online when they come to users content engage in what is called code switching. So alternate between two languages, for example, because many people employ multiple language in daily conversation as well.

There are other challenges such as the agglutinative nature. Some languages will build words on lexicon roots and add suffixes to create complex meanings that in other languages require entire sentences or multiple sentences to convey the same thing.

How LLMs handle the process is quite different. Adds challenges to analyst text or content in those languages.

There is also the issue that glass area. Linguistic context. A relationship between use of two languages.

If I use the example of Quechua. And relation to Spanish which is colonial languages, one would represent power status and issues of importance and how people use the language and same sentence or paragraph with catch what. Where as Quechua would be used more mundane function. Certain language to represent more power and other use the represent less power.

What example would building around this dynamic replicate or exacerbate this power dynamics wean the two languages. Combined with problem you see a lot. For example, in indigenous languages where it is history of.

In our research what we found is I'll highlight this, but these are like direct impacts on people social media users who post in these languages. For example, what people observed is often we take longer time for social media companies to moderate content in these languages versus content that was uploaded in high resource or colonial languages. And because of challenges around developing models, lack of (?) can handle these languages, could lead to ‑‑ led to longer time to moderate content.

There is often, often in addition reports of content removal, (?). And so on.

People also highlighted different ways of recognising problems. Tactics we call resistance. For example, I mention code switching. There is algo speak. Using random letters in a word. Or different emojis. Watermelon referring to Palestine, for example.

So using various tactic to get around.

I can stop there for now.

>> MARLENA WISNIAK: Thanks so much and this is a perfect segue to talk about some real world regional harms going to Middle East and north Africa region. Issues around content moderation and censorship more broadly.

>> Thank you. Unfortunately the region quital (?) in examples. I want to first thank my co‑panelists for laying the ground pretty well to provide specific examples.

I want to make my comments around three issues. First of which already been alluded to the question of where do you invest in those systems. And in which languages. Arabic, I don't to call it minority language. Millions speak it. Official UN language, yet unfortunately AI systems used by tech companies and social media platforms more specifically tend to be poorly trained and I'll mention some specific examples there.

But the issue here is also in cases where there is a minority language, companies think it is sometimes not, you know, market... it is not ‑‑ there is no incentive for them to prioritise that language. Even though, for example, in the context of Palestine Israel, in 2021 when this was a surge of violence in the ground and also a surge of online content, protesting or documenting abuses. We noticed that there was an overmoderation of Arabic language, undermoderation of (?) language. And platforms more specifically. And conducts human rights due diligence into matters content moderation of that period. One of the reasons behind such dynamic was the fact that there were no Hebrew classifiers to moderate hate speech in Hebrew language but there was for Arabic.

One would despite the context was very clear. High incitement. So high volume of content inciting to genocide, inciting to violence. Pretty much direct hate speech. But nevertheless the company did not think it was a priority at the time to roll out clarify classifiers that would able to detect and remove such content.

When we push back of course and after the due diligence findings ware out, now Meta has classifiers. But we found out in new round of violence, unfortunately, I mean after October 7th, that those classifiers was not even well trained to capture more, larger volume of hate speech and incitement to violence and genocide or dehumanisation or dehumanising rhetoric.

Which Halloween leads me to the second issue. Under and overenforcement. Let's now zoom into the concrete issues of how these systems are not ‑‑ they are far from perfect. But the risks and the direct impact of which can be very, very harmful. One of the examples here in terms of overenforcement. If you talk to, for example, Syrians. Syria has been one of the most sanctioned countries on earth planet. Thankfully many of the sanctions are being removed. But the result of that, you know, for example, on counterterrorism legislation or law. We've seen aggressive moderation using these AI tools or algorithmic or alternative decision making to remove content coming from Syria or hosted by Syrians. Same can be said about Palestinians in which a lot of content is erroneously removed. So accounts are being shutdown, content removed. The word shadow banning was mentioned. This is also another issue we come across quite often. Where people are not only their content has been removed but they feel pretty much quite caged, where they content doesn't have the outreach that they are, or the level of engagement that they are used to.

There were also couple of years ago back in 2021 all right while ago with the Facebook papers, for instance. One of the shocking revelations to me was that Meta's automated decision making systems for removing, detecting and removing terrorist content in Arabic got it wrongly 70% of the time. That is quite huge.

And when we talk again about region that is approach at the receiving end of in aggressive counterterrorism measures, the result is this mass scale censorship of activist of human rights defenders, of journalists and particularly around peaks and violence of escalations where people do come to online platforms to share their document realities and (?) what's happening on the ground.

There are other examples I could mention where AI got things terribly wrong. At extremely sensitive and critical moments. One example I can think of in 2021 when Instagram falsely flagged third holist mosque as terrorist organisation and as a result all hashtags. And that time is quite interesting. When Israeli Army stormed al‑Aqsa mosque and ‑‑ this is when Instagram decided oh now is the time to mislabel this as terrorist organisation. And as a result all the content been ad and falsely removed.

This during the unfolding genocide in Gaza, also examples where one famous example was of a person whose Instagram bio was mistranslated. He said, you know, praise to be God I'm Palestine. But the system translated praised to be God Palestine terrorists are fighting for their freedom.

Many years ago also a case of a Palestine construction worker who was working in Jerusalem arrested by the Israeli police because he was flagged to them that he's about to conduct a terrorist attack. And they relied on Facebook's translation. Automated translation. Which falsely or mistakenly translated the man saying good morning. Posting a picture of himself, smoking a cigarette, and leaning on a caterpillar, to good morning, I'm going attack them. And the man was detained for a few hours and interrogated. And then he was released after the Israeli police realised Facebook made a mistake in translation.

The man had shut down his accounts I do remember. Meaning those types of actions and their consequences can be quite detrimental for people's ability not only exercise themselves but can constitute or instil a sense of fear they might be subject to similar detrimental consequences.

David mentioned an interesting point. Which I would like to elaborate on. The tension between precision versus recall. For my observation having worked on multiple crises in the MENA region over the past fee years is companies tend to overrely on automation around times of crises. And particularly when they are attacks or, you know, feel like under pressure that they need to remove as fast as possible large amount of the content in order to avoid being, you know, liable.

One example I can think of is of course October 7th attack. Immediately after 10s of thousands, hundreds in fact, hundreds of thousands of content was just largely removed using automation. And there that balancing act that is not possible to even achieve means that companies are willing to sacrifice or, you know, to say okay it is fine if we erroneously get content decisions or content moderation decisions wrong, as long as we try to catch as large of content as possible.

And there one specific example I can mention here is Meta's decision to lower the threshold for hate speech classifiers. Directly in the aftermath of October 7th attack. To remove ‑‑ for these classifiers to detect comments in Arabic language, specifically those coming from Palestine. So lowering contents thresholds from I think it was around like 85 or so all the way down to 25%.

Meaning the classifiers, you know, at that very low level could remove and hide people's comments because again the emphasis here on removal versus precision or accuracy in the decisions.

Now what does that mean for the users? For people? And their ability to use those platforms freely and safely to express themselves?

We've had situations where people were banned from commenting for days. We've had people who had really extremely innocuous, I mean, just Palestine flags or the watermelon emojis being removed. We've had even people receiving warnings before following particular accounts. For instance, if you were a journalist known for covering the events in Palestine or in Gaza more specifically. And you would get a notification saying are you sure you want to follow this person because they are known for spreading disinformation. I'm talking about credible journalists, as you know professional journalists, not influencers or content creators.

So we've had many examples again where hundreds if not thousands of people who had their content removed as a result of these types of ‑‑ that tension to which companies tend to tilt towards again overmoderation or aggressive moderation rather than accuracy.

Lastly, what I want to say is that, okay, I'm not an expert on LLMs. But I am what concerns me the most is that we are at the cusp yet of another era of new technologies or iteration of technologies which there is a lot of promise. But there are yet to be proper human rights impact assessments. And that is something that you excellently catch in your report. That we still don't have access to these systems. It is hard to independently audit them. And therefore to understand and also work the companies what are the risks and how can be mitigated before her already rolled out at a scale and then as civil society find ourselves in the position of having to document the harm, try to connect the dots and understand okay why is it that the certain population at certain time being subject to censorship and what would be the catalyst reasons behind it and then provide that as evidence platforms for them to correct course and adjust the systems and policies behind them. And I'll stop here.

>> MARLENA WISNIAK: Thanks to much. Before I open it up to the floor I just wanted to highlight a few of the key risks that we found. Just following up on the speakers points. And I really do encourage you to read our report. We distilled it down to 70 pages, catalyze still quite long.

We read over 200 science papers and really brought a human rights legal analysis to it. And try to make it more digestible by having different chapters are. So every right is its own chapter. And also one technical primer. Some of the concepts David shared which are very common in the AICS technical world. Less in policy and vice versa. And we need more human rights. BSR was brought up several times. Handful of orgs and people doing that. But it is really concerning you have this many human rights assessment for this big of an impact.

Some key LLM impacts we found. One of the benefit side because there can be some potential use cases. LLMs typically better at assessing context. So if we are going to use automated content moderation, they typically perform better accuracy level is higher than traditional machine learning. And they can be also better for personalized content moderation.

We talk a lot about user empowerment and agency and if folks want to adjust their own moderation settings. If someone is comfortable with sensitive content, for example or gore, or nudity. They can choose that. Versus others can filter that out.

And also, LLMs can be better at informing users in realtime why their content was removed, for example. What steps they can take to remedy it. And there is such a big gap today with explaining to users why the content was removed and what they can do to appeal that.

That said, few key risks specific to LLMs. One because there is so much content. And I often say I've been working AI for a long time. For those who know me. And AI is neither artificial nor intelligent. It uses a lot of infrastructure. Lot of hardware. And it is mostly guesstimates.

So LLMs, like I should have begun with that. Is large language models. It basically statistics on steroids. It is not divine intelligence. It is just a lot of data. With a lot of compute power, which is also one of the reasons why it is so concentrated.

What happens when systems have so much data often from web scraping is that they can infer sensitive attributes. Much more than traditional ML systems can. And when we think about relationships between governments and companies today, that really puts minorities at risk of being targeted and increasingly surveilled.

Marwa and folks here already talked about over and underenforcement. Unfortunately marginalized groups are both impacted by false positives and false negatives. Like from Marwa's example. Palestine content is both overly censored and at the same time genocidal hateful content is not removed from the platform.

Hallucinations is a very, like typical GenAI LLM example. So when you ‑‑ when companies rely on LLM driven content moderation to moderate misinformation, for example, T LLMs can put out really confident‑sounding statements that are just wrong. So using that to inform human content moderators or automated removal often leads to just errors and inaccuracy.

One last thing. I will mention. Is that our organisation work a lot on protests, civic space, assembly and association. Often actions and content that are by default contrarian. Minority. Anti‑power. Protests usually protest something. You protest a powerful institution.

And if you think about AI both traditional machine learning and LLMs, they are statistical bell curves. And minority content falls outside this dataset. And Marwa and David I think hinted at that. Thinking about crises, and quote/unquote exceptional content.

So even the best intentioned platforms will make errors. Just because this is content that falls outside of datasets and the bell curve. It is by definition compensational or contrarian.

So that is really something to consider thinking about assembly and protests. And I've leave it at that. I encourage you all to cheque that out and reach out. And I would love to open it up now. One of the things we've been thinking a lot about ECNL after doing this human rights impact assessment is now one, what kind of recommendations can we make to AI developers and employers? What are we still missing in academic and civil society? Community, what gaps are there?

So if anybody has thoughts on that I would love to hear from you. Otherwise any other questions. And folks online, please either write your question in the chat and I'll bring it to the floor or you can raise your hands.

Yeah? I'll take a couple questions. Just because we have limited time.

Please raise your hand if you have a question.

Please. And if you can introduce yourself, name and affiliation. That would be great.

The mic is over there. I'm sorry. Please line up and go to the mic.

>> Just want to make sure I'm audible. Baldazar from university (?) and I'm wondering if there are news for other actors sorry more this case government and civil society to introduce the technical design for content maturation within digital platform. Or is it largely proprietary by this social media companies and there is no way to influence the technical lifestyle so to speak. And we sort of rely on the tech platform to make decision on what is the next iteration of the LLM going to be? Or is there really ‑‑ like I have a new ‑‑ some kind of external human in the mechanism so civil society or government can influence more technical sense to complement legal intervention and (?)

Thank you.

>> MARLENA WISNIAK: Thank you so much. I have a few thoughts but I'll handed Toyota panel first.

>> One of the kind of feedback in our research is to engage greater community leadership and participation in the building of LLM. So, for example I can come in the form of the building of datasets, ownership of datasets. Datasets for speck kinds of content. And building LLMs, many examples around the world this happening with local researches and communities coming together to build LLMs. Language technologies for specific purposes outside social media. Problem came up lot and I don't know how if others have thoughts on this was that social media companies once we engage is often not aware of these communities of local LLM developers and LP researchers or these kind of developers which they could benefit a lot from but there is a gap there, disconnect there and I think that is one area as well.

Also specific elements for governments and industry to invest these kind of partnerships and support efforts.

>> DAVID SULLIVAN: Building on that, I do think there is an opportunity coming up where there is a lot of enthusiasm and interest within the trust and safety community for open source. In particular there is a new project cause roost, robust, open, online safety tooling. Which is where a lot of companies are coming together to open source some of the technologies and tools in the space. That's been a sticky area when it comes to really challenging online safety issues. But I think there is an opportunity there and there are ways for people to get involved. I think that is one positive to look at.

>> MARWA: Plus one to involving communities from the get go from the start. And yes, and I also do, yeah, do confirm that I don't think that social media companies are connected to local developers or local LLM experts. I certainly don't see that happening in the region.

I would also say in addition to these voluntary multi stakeholder mechanisms or fora, maybe there should be a space for mandatory human rights impact assessments. Also from throughout the cycle of development, starting from the very beginning. Of course and during the ‑‑ after ‑‑ during the launch of those systems and their enrolment. And of course following any adjustments or modifications of such systems and their use.

>> MARLENA WISNIAK: And I'll just add briefly on the AI life cycle. ECNL has been working with Discord on piloting what do we call our framework for meaningful engagement. From the first stage of life cycle and AI design they are designing interventions to moderate content on line. So partnering with them and stakeholders around the world. Only of you in the room. On helping them do that. So it is a very specific case study. And more about that.

Another example where I think folks can be involved is after the deployment stage. So way that LLMs work is they are trained and then there is the whole validation/evaluation section. There are often done through reinforcement learning by human feedback. I won't go into details but basically requires people too go through the outputs and retrain them.

One thing we've been advocating for is involve communities at that stage as well. Typically during these reinforcement learning, it is mostly silicon valley folks or experts like probably us in the room who would do that but not the communities affected. And it is very, very homogenic. So people from elitist academic institutions and high name NGOs and those based in silicon valley. And that is problem because it is supposed to fix or improve but ends up perpetrating even more bias. So many ways to involve folks. And that is definitely step 1 I think to actually make these system better.

>> Part of the (?). Kind of understood LLMs always make mistakes. I think that seems obvious. But do you think there is an area to evolve LLMs to do a good job here? Or do you think always need for other mechanisms to like have humans involved in the thing? Or do you think LLMs not the right technology at all and we need other ways to empower users and give users decision about which contents they want to engage with and they want to see? So what is a way forward?

>> I can take a quick. Dimensions to. This how do companies address content moderation and there are actually many different models. Keep in mind. And they could use: You could imagine a subrated moderator thinking of ways for their specific case and could be helpful for community building that sense. So keep in mind there is this range of options available.

But I think when we think of larger scale, opposition always human moderators should be part of the consideration, calculus how you address content. I think others like David and Marwa mentioned this was well.

What's important is flexibility around this. For some other particularly low resource languages where there is, should be heavier emphasis on human moderation. And that could evolve over time. But through always some kind of combination between the two.

>> DAVID SULLIVAN: I would add there is an excellent research paper Google folks put at last year in 2024 how LLMs could be leveraged to support human raters of content. Goes into a level of detail beyond me but might be helpful. I think one of the opportunity here's which is cognizant of all of the risks when it comes to how AI can be misused when it comes to content moderation.

When we think about AI as this technology that is overhyped and lacking business application, I do think that content moderation and trust and safety is a concrete business application for AI. And one where the developers and deployers are often the same company. And so I think there is ‑‑ there are some opportunity there is. But it comes with all of the risks we've talked about.

>> Very quick follow up. So you said if the humans are part of the chain, I think that the challenges of scaling up and also timely reactions right. Can you comment on that.

>> MARLENA WISNIAK: Excuse me, those people behind. I urge you to read the reports and we have also large section on recommendations so that selects forward. We only have two minutes so briefly please.

>> Professor Julia queen Mary university London academic and I'm a lawyer. And hence my question. Obviously for lawyers it would take extremely long qualification for a judge to adjudicate content, right? For lawyers takes a very are, very complex decisions. Where as I understand LLMs and artificial intelligence is based on obviously complex processes of labelling and other processes. I was wondering in addition to LLMs by definition will always have these problems which you so fantastically described. Isn't amplification fact that lot of content is targeted particular user groups.

Isn't that actually the more promising way and putting slightly provocatively to deal with harm say on social media.

Thank you.

Of.

>> MARWA: If I understood the question right, you are saying that instead of this binary remover leave the content up then, down ranking or remote... yeah, promoting and demoting content.

Well the thing is that even then we do have some what I would say discriminatory application of what kind of content gets promoted and demoted.

Which brings us back to the question of design. And also the question of equitable investment in resources. And transparency.

>> MARLENA WISNIAK: And I'll just add briefly there is a whole other range of LLMs that are other explored. We're very concerned about the use of AI for content removal or ranking. There are other ways like informing users how to appeal content. Informing them about content policies, or less stereotypical content moderation but how to enforce it better and exercise our rights better, where there is a much higher risk and more promise. So we encourage platforms to think beyond content removal.

Last question, briefly.

>> Yes. High. Sorry I'm take this off. My name is Carla, from association for pervasive communications. And my question is well more relative to how can we relate this conversation to the building of autonomous infrastructure? We work with community networks which are networks that are designed, created and developed by the communities, for example indigenous communities that are not connected to the internet. And you get like this prior conversation before connecting on how you want to connect. What do you want to do with your data? And in those terms we use open source. And in that sense, you know, that makes the conversation completely different on how you connect and how you even feed this model. So how can we connect these two movements? Let's say. Like, where is your work at on taking one step back and looking at the infrastructure at the connection, at the different servers where all this data is collected? And how the data is collected. I would like to know more about that from your side.

>> MARLENA WISNIAK: 30 seconds.

>> Sure. So some of the exams came up research relate to as I mention are like community participation ownership of data. So there are models where engage communities about what kinds of data. How you classify. What categories important. And who ultimately owns and becomes steward of that.

That kind of emphasis is very different from the current models. And having LLM develop as partner with these kind of communities and context is one approach. Example of community‑based internet networks you are talking about and also introduces different kinds of business models as well.

>> MARLENA WISNIAK: Thanks so much. And unfortunately this session already wrapping up. There is so much to be said about this topic. I hope one takeaway you have is that it is an emerging field. There are still too little transparency. And if question urge platforms to share more data, including how and when LLMs are used. We often don't even know that. That is one thing. One of the things that we try to do at ECNL is really document the human rights harms as opposed to AI hype. There is a lot hype in this space as you probably all know.

And at the same time excitement around community‑driven models like don Raj talked about.

So there is like not everything is doom and gloom. There is hope for some community driven, public interest, fit for purpose models. I think we can explore. And really find a way that produce, develop AI then respects labour rights including human content readers, and engaging stakeholders and going forward team to work closely with our partners. Many of you in the room and implements recommendations with the platform, test them as well. Very much ongoing. So you will see that in our report on the last section.

Please reach out if you quantity to get involved. This conversation is only starting. You know, who knows if LLMs with even be deployed. They are very expensive to run to begin with. Something we didn't really talking about.

But in any case, hearing your voice and concerns is really important. So thank you so much for being here. And happy IGF.