Sign up for The Media Today, CJRâs daily newsletter.
âNothing quite like casually reviewing horrific anti-Black hate speech to jolt you awake before 9:30 a.m. on a Monday,â I tweeted from shaky fingers.
Back then, in 2021, Twitter was still Twitter, I was still on Twitter, and I had spent years working inside Trust & Safety (T&S) departments at Twitter and Twitch as a senior policy expert in free expression.Â
That role put me in a position of immense power because inside technology companies, T&S departments have quietly acted as the governing bodies for our daily digital interactions. These departments are charged with keeping internet platforms and services as usable and safe as possible. Depending on the company, this usually means developing and enforcing rules or community guidelines (content moderation); engineering certain new products and trying to convince other departments not to engineer other products; conducting research; monitoring user accounts for coordinated, manipulated, or inauthentic behaviors; and dealing with various takedown requests.Â
In recent years, journalists and academics have sought to bring more transparency to the often opaque field of internet governance. While the resulting literature has offered useful insights, insufficient attention has been paid to the unique experiences of Black people working in T&S. It is in this context that I dedicated my time as a CCSRE Technology & Racial Equity Practitioner Fellow at Stanford University to conducting research to start addressing this gap.Â
My fellowship and research for this article ended up being anything but ordinary. I knew that I would be asking individuals to take enormous personal risks simply by speaking with me about their everyday experiences. I also knew that I would be asking them about some of the most traumatic experiences of their working lives.Â
These concerns were not taken lightly. Drawing on my own experience of fielding inappropriate and/or inconsiderate interview requests from academics and journalists, I was conscious that I would need to take a different approach. The Black T&S workers I wanted to speak with expressed concerns about remaining safe and protecting their identities. I ensured them I would treat their stories with the utmost level of respect and confidentiality.Â
I understood my sourcesâ concerns about keeping their secrets more than they knew. During the time I was conducting interviews, I had a big secret of my own. I had been subpoenaed by Congress and was blowing the whistle on Twitter to the United States House Select Committee on the January 6 attack.
After my testimony was first used pseudonymously in a Congressional hearing, and then I publicly revealed my identity, I was thrust into a new spotlight. With this came attention from the right-wing corners of the internet. Soon my name, picture, and old internal Slack messages from my work at Twitter were misconstrued and published to the world as part of what became known as the Twitter Files.Â
The Twitter Files created an unimaginable new vector of abuse for people who had already seen the worst sides of the internet and were keenly aware of online harassment. While T&S workers had feared targeting by an embittered colleague or an external hack, no one had considered that their employers would go to those lengths. Yet because of the Twitter Files, T&S workers faced death threats, rape threats, and doxxing that required them to flee their homes.Â
This all also happened at a time of industry-wide layoffs and market uncertainty. Elon Musk paved the way for T&S roles to be eliminated en masse by firing 80 percent of the company formerly known as Twitter. Other tech companies like Meta, Amazon, and Alphabet followed suit and decimated their T&S teams.Â
Thus, as I did this research, the risk calculation changed for T&S workers across the industry. Along with this, people I had previously interviewed changed their minds about being included in this story.Â
This article is based on interviews with six individuals who identify as Black Americans and who worked within T&S departments of tech companies. Their roles varied from building training curriculum to leading policy teams. They collectively worked at 12 technology companies ranging from legacy organizations like Google, Twitter, and Meta to financial and education technology start-ups. Iâve given them the pseudonyms Avery, Carter, Edison, Frankie, Kai, and Parker, and used gender-neutral pronouns to refer to them all.Â
What I found from my interviews is evidence that people are experiencing a wild combination of circumstances by simply existing as a Black person within a T&S department. They feel what itâs like to be othered within a supposedly progressive environment. They encounter aggressions and slurs at work. They feel the weight of specific labor necessary to fight for the digital rights of individuals around the globe who look like them. And all of this takes a heavy personal toll.
This has serious implications for humanity. While it may seem like ghost work, the work from these seemingly invisible hands forms the bedrock of how our internet was built and is regulated. These individualsâ brains have created our modern society and developed the architecture for our digital human rights. As one of the T&S workers I spoke with told me, âBy the end of my career, I think I will have had a profound effect on the way that the internet works, especially social media platforms.â
However, almost every interviewee would become cautious. Speaking in hushed tones, they would describe a harrowing work experience on a T&S team that they had never previously shared. After this happened a couple times, I decided to ask why they were sharing these secrets with me.Â
âNo one has ever asked,â I was told.Â
The six stories I highlight within this article are not meant to be a representative sample to generalize the experiences of all Black T&S workers or all T&S departments. Rather, these interviews serve as the first inquiry and foundational exploration into an area rich for future in-depth research. Each theme and finding drawn from this article deserves to be delved into with dedicated research using quantitative and qualitative methods. These areas of further study include discrimination in pay and leveling in T&S departments, a lack of diversity within senior T&S leadership, inadequate warnings about the content workers will encounter, a very specific type of labor extracted from marginalized employees that Iâm calling Compelled Identity Labor, and insufficient institutional support for the lasting impact of their work.Â
Recording the experiences highlighted within this article and from additional future research is essential. We are in a moment of critical reflection about technology, and with external regulation for internet governance on the horizon, itâs vital that we amplify these untold stories. As we envision new ways to create safety in our future digital lives, itâs important to listen to and honor these human experiences. By drawing upon the experiences shared with me, I also provide four recommendations for how to begin to create more equitable regulatory environments and develop a healthier digital experience for all.Â
Diversity, Inequity, and ExclusionÂ
Whistleblowers Ifeoma Ozoma at Pinterest and Charlotte Newman at Amazon gave the public the first real view into the modern environment for Black employees inside technology companies. These Black women, who both worked on policy teams (which are typically housed inside of T&S departments), reported in 2021 that their companies and departments were rife with discrimination in pay and leveling based on race.
Ozoma and Newmanâs experiences resonated with the Black T&S workers I interviewed for this article. Most said the first obstacles in their careers presented themselves during the recruiting and hiring process.Â
Kai bluntly stated that on their first job in T&S, âit was very clear there was a huge disparity in my level of education and what I was being paid.â
âIt feels pretty systemic that [Black] people coming in are downleveled, compared to their experience and their education,â Parker said.
In addition to stories about Black employees being overqualified and underleveled, interviewees said they were often unable to negotiate starting salaries like their non-Black colleagues and were often paid less.Â
Avery said that when they were hired, they were offered a certain salary that seemed low to them. âI counteroffered,â they said. âThat final number was the first number somebody else [who wasnât Black] got. They didnât have to negotiate for the same amount.â
Parker added, âI thought Iâd done a good job negotiating, but more people came on after and ended up with way more money.â
Every interviewee said that the companies for which theyâve worked, like most in the technology industry, presented an outwardly progressive stance toward diversity, equity, and inclusion (DEI). Notably, though, all said that their onboarding experiences portrayed a very different reality.Â
Avery compared tech companies, often heralded for their innovation, to âtraditional work environmentsâ like law firms, noting that while tech industry jobs are marketed to be different, âthere are still things to watch out for.â
Kai said tech companies âdo a good job of marketing the legal floor as a philosophical or moral stance of the company: âWe do not believe in discrimination.â Well, thatâs also illegal.â They continued, âHowever, they made this seem like itâs a moral sort of imperative.â
Avery also felt tech companiesâ PR around DEI rang hollow. âThe sentiment is that itâs not a serious attempt at creating a more equitable environment,â they said. âOr that itâs just something to appease people, or something to write down on paper.â
Frankie outlined a detail about where diversity occurred that was a repeated theme across my interviews. Companies âsay they are pro-diversity. But companies show who they care about. Who is at higher levels?â Frankie asked. âWho are the vice presidents [of T&S]?âÂ
Carter added, âThere is a problem with T&S as far as Black leadership goes.â
Parker summed it up this way: âI lost hope of tech companies being this environment where thereâs sincere intentions about diversity.â
Exposure Warning and Macro Aggressions
The unique function of self-regulating the content posted to technology platforms and services falls to the employees of T&S departments. For my interviewees, exposure to hate speech was omnipresent in these jobs, regardless of whether they were directly responsible for moderating hate speech content. This included graphic examples of anti-Black hate speech, gore, and violence.Â
I asked the interviewees whether they felt properly prepared for â or at least warned about â the type of content to which they would routinely be exposed. All said that while there were warnings that they would likely encounter sensitive content, most were not made aware of the extremity of the material or the frequency with which it would be seen.
Avery said that prior to taking their job in T&S, they were not told about the types of imagery they would be exposed to. âThere was no specific, âWeâre going to show you examples,ââ they said.Â
âI wasnât aware,â Kai said. âI know I was told we would see some pretty âgnarlyâ stuff, but it wasnât particularly explicit on what âgnarlyâ would be.â
âItâs traumatizing,â Avery said. âYou are going to see charred human bodies.â
Kai said that the warnings they received were inadequate. âWhich is not to say I wouldnât have taken the job,â they added. âI just should have been allowed to consent to a larger degree than I was actually allowed to.â
Once Black employees were hired and working inside T&S departments, their daily experience and environment could be challenging. As well as being exposed to hate speech on the platforms they worked for, they described casual usage of anti-Black slurs within their own workplaces, including being on the receiving end of such slurs from colleagues.
âSometimes the micro feels macro,â Frankie said. âT&S talks about sensitive things bluntly. But people still need a certain level of decorum.âÂ
âThere has been a large usage of slurs unnecessarily because of the workplace I am in,â Kai reported. âPeople think they need to use the actual slur instead of some censored version of the slur.âÂ
Frankie added, âIâve asked non-Black colleagues to not say the N-word and not tag Black colleagues when using it.âÂ
Kai said that âthe N-word with the hard ârâ is often usedâ in T&S: âYou do feel it in your body, like physically, when someone is using that word, even if someone is repeating it because itâs words on a page,â they said.
The experiences are often worse. Frankie described an incident when they joined a new company and a colleague posted a photo of the new team on the service they worked for.Â
Frankie said, âOne of the comments directed at me said, âWhoâs that N-word in the pic?ââÂ
When Frankie brought it up with the colleague who posted the image, their colleague replied, âit could be worse.âÂ
Compelled Identity Labor
In Sarah T. Robertsâ seminal book, Behind the Screen: Content Moderation in the Shadows of Social Media, she describes a conversation she had with a commercial content moderator contracted by a third party. The moderator discussed how the technology companyâs content moderation rules seemed to vary across geographical regions, and surmised that âone person on the [internal T&S] team is just passionate about the issues in the Middle East.â
This is not a subject I had previously seen discussed. When I read it, I immediately thought of the numerous Palestinian people I worked with on T&S teams who had been thrust into determining exceedingly complex and personal geopolitical situations.Â
The experience is something that also resonated with my interviewees. This is a unique type of work that I am calling Compelled Identity Labor. It refers to situations where T&S departments default to treating employees with marginalized identities as the voices and/or representatives of entire online communities who share their identities. This can force Black T&S workers into a situation where they are tasked with being the sole voice advocating on behalf of the human rights of all those people who happen to look like them.Â
This forced labor creates a sort of cognitive dissonance where people have to divorce themselves from their identity in order to meaningfully advocate on behalf of themselves. Kai conjured the philosophies of Frantz Fanon to describe the experience. âThereâs very different masks you have to put on,â they said.
âIt gets very different when youâre the face of the conversation piece and you also have to intellectualize it,â Kai continued. âI had to do a lot more emotional labor to get through those moments.â
None of my interviewees consider themselves experts on race. All, however, described instances where non-Black colleagues had treated them as experts on anti-Black hate speech or slurs. Multiple interviewees described how higher-ups within their T&S departments had specifically asked for their input on whether the company should decide that the N-word ending with an âaâ should be treated the same as the word ending in âer.â
âHaving these conversations makes you look at people funny,â Frankie said. âI donât want to always have to be that person.â
My interviewees said that this Compelled Identity Labor often multiplied during times of high-profile civil unrest. Many noted that George Floydâs death in 2020 had brought a particularly challenging spike in the volume of this type of labor within T&S teams, on top of their personal processing of the events.
For Frankie, those months of social upheaval felt exhausting and relentless both professionally and personally. After working for hours during the day moderating the anti-Black hate speech content on the service they worked for, they would close their laptop at night and then go out to join a protest. They said they felt like they couldnât take a break from being surrounded by hatred.Â
Kai also recalled challenging experiences in their T&S department following Floydâs death. They said their department struggled to establish a policy for handling the influx of hateful content posted to the platform. In addition, the company was deciding how to publicly position itself in relation to the growing conversations about racial justice in the wake of the #BlackLivesMatter movement. Kai was asked to help.
Kai said the âwell-meaning white people I worked withâ had a âreliance on me to cosign or help them get to a decision, or help them through the decision-making process so that we as a platform could navigate the space, and they could navigate the field.â Kai added that their T&S leadership âdidnât want to make a decision that felt like they would offend me.â
Kai also described a conversation that happened around the same time when they were placed in the position of making an argument for the Confederate flag to be designated a hate symbol. A white colleague made a counterargument that the flag was reclaimed language and not hateful. Â
âThat is the kind of stuff where youâre like, âThis is a joke, right?ââ Kai said. âBut theyâre not kidding, theyâre very serious, because they fundamentally believe in this idea of âfreedom of expressionâ â and what they donât see is the harm that some expression can cause people actual physical harm.â
While all the interviewees spoke of the responsibility they felt in carrying their entire community on their backs through their daily work, they also said that despite their discomfort in these situations, they felt it was important for them to be in these rooms.
âWhen the culture becomes a part of the conversation, itâs important to have the right people in those conversations,â Edison said.Â
Lasting Impact
The experience of working in these environments, performing this Compelled Identity Labor, and being exposed to hate speech content was profound for the people I spoke with. Given what these humans described as seeing, doing, and experiencing every day, I wanted to know what kind of impact they believed the work of T&S was having on them.Â
What I heard from my interviewees aligned with reports from nearly a decade ago about secondary and vicarious trauma experienced by moderators and journalists whose work requires them to view graphic content on the internet. It also mirrored the âpsychological strainâ described by whistleblower JoĂ«l Carter, who came forward in September 2023 with claims of discrimination and retaliation occurring in the T&S department of TikTokâs parent company ByteDance.Â
Kai expressed that the trauma and strain is almost inescapable if your job requires viewing hateful or graphic content online. Itâs ânot quite a questionâ of whether you feel it, they said, but rather the extent to which you feel it.
âYou learn very early on that you donât need to be married to the argument in an emotional way, but what you do need is to make the argument,â Kai continued. âIn the moment, it feels fine. But afterward, it is awful. Itâs like I canât believe I had to do all of that because people donât believe that my humanity or identity should be disallowing assembly.â
Avery said, âBalance is doing what Iâm supposed to, in terms of enforcing accurately the community standards and finishing that work. But after, I have to take time to process what Iâve seen.â
âWeâre professionals, but weâre also individuals. There are moments where I donât want to take part in this at all,â Parker said. âThat has been really hard for me.â Â
Kai said it was difficult when colleagues âdid not think about me or people who look like me to want to save them from potential violence.â They added, âThatâs a lot to carry with you.â
Parker said that the way the T&S working environment affected people was âa very individual thing.â They explained, âYou have to hold a lot of things, and itâs just tiring, honestly.âÂ
Frankie described the physical and psychological impacts the job has had on them as âjarring.âÂ
âSometimes Iâm not able to sleep so well. I wake up seeing images, and I react to loud noises,â they said. âI have less patience, I cry a lot more, and I have gotten a darker sense of humor.âÂ
And, Frankie added, they lost 30 pounds due to the stress of their work.
Carter said that over time, the work has chipped away at their confidence.
For Avery, it has made them more wary of the world and caused them to experience hopelessness. They said, âI just have to walk around knowing ⊠people who are plotting to harm folks on the basis of skin color ⊠are really common enough for me to be worried that they are closer to me in a social setting than I would like them to be.â They added, âTheyâre your middlemen; theyâre your police officers; theyâre your people next door.â
The impact of T&S work has been so severe on the people I spoke with that they feared it had caused permanent psychological damage. It has also made them reevaluate their careers.Â
âMy work has a more profound impact on me,â Kai said. âIâll never be the same, and that is disturbing to me â even saying that is highly disturbing to me. I will never be the same, and I canât unsee the things that I have seen.âÂ
They continued, âI have had to get out of the direct adjudication of content that involves large amounts of hate speech. I can no longer make those arguments; I canât divorce myself from making those arguments. So Iâve had to change my career trajectory.â
Glimpses of Hope
After listening to each individualâs stories and hearing the impact they have experienced, I ended all of my interviews by asking what advice they would give their former selves or any other Black person considering working in T&S.
The responses shocked me. Despite the challenging circumstances they had all outlined, none of my interviewees said they wouldnât take the job again; nor would they discourage Black folks from taking T&S jobs. Instead, they offered measured and cautious advice.Â
Given the nature of the labor, Parker stressed the importance of creating a separation between personal life and work life. They also noted the importance of not attempting to do this work alone. âIt took me a long time to understand that I needed support communities,â Parker said. âItâs super important to have that group of people who create a safe space for you. Learn how to do that.â
Kai echoed the importance of support and community. They said it was necessary to have these systems firmly in place while doing T&S work: âNot feeling great and having to do this job on top of that is not the move.â
My interviewees also cautioned that while the job may feel like the weight of the world at times, itâs important to remember that they have options.Â
Carter encouraged Black people interested in the tech industry to jump and take the leap: âFigure out what about the work drives you, and then just go for it.â They added, âBut donât stay around too long.â
Edison agreed. âDouble down on what you can bring to the table and nurture whatever that is. And if it doesnât go right, move on,â they said. âThese are all just jobs, in the end.â
Kai said the most important thing for Black people in T&S roles is to manage their expectations, and âhave an understanding of what you can and cannot do.âÂ
Then Kai shared a piece of advice they said had taken them a while to realize. It was something I wish someone had told me a decade ago when I started on my career path.Â
âYou are not going to fix racism by doing policy. So that will give you a leg up.â
Findings and Recommendations
The experiences of the six individuals I spoke with paint a picture of internet governance that is not only fragile but places a heavy human toll on Black T&S employees. While these stories are not a representative sample meant to generalize the full experiences of Black T&S employees, the findings from these interviews highlight the first inquiry into numerous areas that are rich for future research.Â
Findings from these interviews with Black T&S employees illuminate a uniquely challenging â and at times hostile â working environment. Among the many things contributing to this situation are discrimination in pay and leveling that begin with recruitment and hiring; a lack of diversity in senior leadership roles within T&S departments; an environment reverberating with racial slurs; inadequate preparation for exposure to traumatizing content â sometimes even in roles where it is not a job requirement â and insufficient support for coping with exposure to such material; Compelled Identity Labor that requires employees from marginalized identities to speak on behalf of their identities; and deep psychological impacts from the entire experience of such work.Â
Without change and further research to find solutions, the tech industry risks failing to attract and retain Black T&S workers it so urgently needs. In this absence, a gap will be felt in both the online and the real world protection of marginalized communities. Based on my interview findings and personal experiences, I have four recommendations to tackle the litany of challenges to creating environments where Black T&S employees are not only treated equitably, but are also able to thrive.Â
First, T&S departments need to eliminate disparities in pay and leveling. As Kai pointed out, discrimination based on protected characteristics is illegal. Employees who see it happening should discuss it with their companyâs internal employment lawyers, who are typically housed in employee relations departments.Â
To proactively ensure equity, T&S heads should also bring in external auditors to conduct assessments of leveling and pay across their departments. If there are discrepancies, leadership should immediately rectify any inconsistencies, realign current compensation, and offer affected individuals back pay and requisite stock options.
Second, T&S departments need to develop more robust warning and boundary systems for employees who will be exposed to harmful content. Before hiring, companies should provide more thorough waivers that detail the types of content employees are expected to assess. As Frankie told me, âCompanies need to make sure they are specific about what âsensitive contentâ means.âÂ
Companies should also acknowledge that T&S employees are humans, and humans have limitations. This means providing and maintaining a list of âno-goâ boundaries that employees can set for content they prefer not to review. T&S heads also should enforce standards for the workplace that include encouragement against the uncensored usage of racial slurs.
Third, T&S needs to hire experts who specifically focus on race, ethnicity, and national origin. These are complex concepts that vary across the globe, and companies must end their reliance on single employees or groups who hold marginalized identities to speak on behalf of their own identities. Compelled Identity Labor is not a healthy or sustainable solution.Â
Once hired, these new experts, in addition to their Black T&S colleagues, should not have to rely on their own wellness practices to manage the impacts of their work. Companies should provide expanded resources for T&S workers to support their mental resilience and psychological well-being.Â
Fourth, tech companies should continue to invest in developing and supporting diverse T&S teams and leadership. This includes the incentive of tying executive compensation to diversity goals. In a time in the United States when diversity, equity, and inclusion programs are under attack, this could be difficult.Â
However, given the unique function of T&S in providing safety for humans, Edison explained to me why it is vital that these teams reflect the diversity of humanity. âWe need the awareness shown by those who lived it. It keeps us honest and open and realistic about what weâre doing, the impact that we have, and makes sure that we donât harm anybody.â They added, âWe can take the bias out of certain behaviors now.âÂ
Despite the challenges they face at work, my interviewees understood this deeply. They remained adamant that their experiences â and this research â should not deter people from entering the field of T&S and doing the vital work of internet governance. Rather, they wanted Black folks thinking about joining the industry to know that they are essential to a functioning, healthy internet.Â
Frankie said, âYour thoughts, your impact, and your empathy are needed.â
âYou have more to give than you think,â Edison reiterated. âItâs important to realize youâre there for a reason.â
Anika Collier Navaroli is currently a Senior Fellow at the Tow Center for Digital Journalism at Columbia University and a Public Voices Fellow on Technology in the Public Interest with The OpEd Project. She was previously a CCSRE Technology & Racial Equity Practitioner Fellow at Stanford University. Anika also held senior policy official positions within the T&S departments at Twitter and Twitch. In 2022, she blew the whistle about her warnings to Twitter that went unheeded leading to the January 6th attack on the Capitol and the platformâs ultimate decision to suspend former President Donald Trump.