Join us
Illustration by Evan Grothjan

The Future of Visual Investigations, If We Can’t Trust Our Eyes

Journalists and other experts tell us how, in the age of AI, technology is transforming the work of visual forensics. “We might have to be a little bit more imaginative,” Emily Tripp, of Airwars, says.

Sign up for the daily CJR newsletter.

On the evening of October 17, 2023, an explosion struck Gaza City’s Al-Ahli Arab Hospital complex. Shaky videos, filmed by bystanders, captured footage of the grounds partially consumed by flames. Each clip posted online revealed more horror: blankets used as stretchers for carrying maimed bodies, a cratered parking lot with charred cars. Hundreds were reported dead. But what these videos couldn’t reveal was who was responsible. 

In the initial fog of war, several breaking-news teams, including the New York Times and CNN, cited a statement from Gaza’s health ministry blaming an Israeli strike. Visual-investigation teams got to work analyzing footage of the explosion, with the aim of bringing objective facts to light. Instead, visual investigations emerged from major newsrooms and independent organizations that, in some cases, contradicted one another’s work. In several cases they reported inaccurate information or published inconclusive findings. Amendments, editors’ notes, and mea culpas followed. 

The episode exposed the limitations of visual investigations, cutting to the core of the discipline’s credibility. If readers were led to believe that visual investigations produced incontrovertible results, how were they to make sense of uncertainty from experts and legacy media?

Visual investigators’ tools and techniques—including scraping data and satellite imagery, user-generated content verification, archiving social media, and geolocation—were pioneered by independent organizations including Storyful, SITU Research, Forensic Architecture, Bellingcat, Airwars, and Syrian Archive. They gained international prominence around the time of the Arab Spring, in 2011, and during the Syrian civil war, when in-person reporting became too dangerous.

Teams later formed at major media outlets, including the Times, the Washington Post, the Wall Street Journal, Le Monde, CNN, BBC, and the Financial Times. They were honored with Pulitzers, Peabodys, and Emmys. Today, visual investigators are known by many different names: open-source reporters, open-source intelligence (or OSINT) practitioners, forensic architects, even digital Sherlocks. 

But in recent years their work has been complicated by myriad concerns, chief among them the changing nature of conflict and the evolution of technology—vulnerabilities that have only grown more apparent since that night in Gaza. Artificial intelligence has made it possible to fake evidence, causing widespread distrust in visual media. And then there are the actions of tech companies themselves. In April, Planet Labs, the satellite imaging company, stopped releasing daily imagery of the Middle East. In July, Google added sophisticated image generation capabilities into Google Earth, allowing people to create phony satellite imagery. (The outcry from journalists and others spurred Google to quickly reverse course.)

“I’m less concerned about having to prove to people that something is synthetic. What I’m more concerned about is how we prove to people that something is real,” Imogen Piper, a visual-forensics reporter for the Washington Post, told us. Alexa Koenig, the director and cofounder of the Investigations Lab at the University of California Berkeley Human Rights Center, said that the standard for earning readers’ trust has changed: “We’ve got to figure out how we pivot in this moment, because what we used to use to signal legitimacy is not gonna work, if it is still working at all.”

Visual investigators will have to find ways to overcome these mounting challenges. We spoke with experts—journalists as well as OSINT specialists and nonprofit leaders who influence the field’s development—to understand both the recent past and where the field goes next. Their comments have been edited and condensed for clarity.

Verification isn’t enough
James Harkin
Director, Centre for Investigative Journalism

Verification tells you whether an image has been faked. But the real problem within the shaky terrain of journalism on which we stand is not the fakery of images but the propagation of half-truths.

If we’re not careful, given the speed at which conflict now happens, we might not properly do the homework in terms of not only our understanding of fakery, but also our understanding of half-truths. The danger is that the next world war might begin with a sort of grainy, contested image, which is thrown up online from some distant and inaccessible outpost right into the pages of a newspaper that has already sacked all its journalists.

Caution: Image candy ahead
Krishna Karra
Remote sensing journalist, Bloomberg News

The Iran conflict is an example of people not trusting. MizarVision, a Chinese company, was publishing all these images in the beginning. This was before all the satellite imagery restrictions. People figured out that a lot of the images that they were publishing were kind of AI-upscaled or color-shifted versions of Maxar or Vantor satellite imagery.

I think the challenge with OSINT has always been lack of synthesis and more “Here, look at the thing,” the image candy—almost blurring the lines between “Look what this revealed” as opposed to “What does this mean?” It’s easier than ever to access all the freely available imagery that is real. Then it’s easier than ever to create fake versions of that imagery or generate new images from nothing.

Where I’m trying to focus my efforts, and where I think some reporting is going, is in treating satellite imagery as data or a time series of data—the way that data journalists might treat a large corpus of data, where you’re applying statistical methods or machine learning to large stacks of data—to say something that is not visually evident in a single image, but is true. Then we’ve shown something that, I think, is more rigorous than “Here’s a before-and-after image.”

A dramatic drop in available material
Alison Killing
Senior reporter for visual investigations, Financial Times

Often, in the past, we would have been able to get quite a large volume of material covering specific incidents. We saw this in Syria—there was a huge amount of material covering that war—similarly in Ukraine. Then with Iran, and in Gaza as well, suddenly the amount of information that was available to us just dropped dramatically. So that opportunity to cross-reference material has also gone away—which makes our job much more difficult—and it’s partly because people have realized the power of open-source investigations. They’ve realized the danger that having large amounts of video poses to somebody who wants to commit human rights violations.

‘Now part of every reporter’s beat’
Malachy Browne
Enterprise director of the New York Times’ Visual Investigations team

Illustration by Evan Grothjan.
 

We have access in ways that could only have been dreamed of back in the eighties, nineties, and aughts. It’s a bit like social journalism was, where some people were in the corner twelve, fifteen years ago, and are now part of every reporter’s beat. I think OSINT is going to become general reporting and investigative reporting, and other teams are going to start just hiring OSINT people. It’s just going to bleed into an expectation for all reporters and journalists.

I think the strongest use of OSINT is in places that you can’t reach, and particularly useful in conflict—like Sudan, parts of Ukraine, Iran at the moment, and Gaza, obviously.

Al-Ahli Hospital was a—I’m going to use the word “disappointing”—episode for me in OSINT journalism because you have different newsrooms analyzing the same material and arriving at different conclusions. There was intense interest in covering it, and also what we could say about it. We came out with our analysis that we were pretty confident in from an early stage, but we went through repeated editorial checks and balances, source-checking, to be sure because it was a contentious story. There were challenges that required really specialized attention and also the original footage. (If you don’t have the original footage, the frame rates are being changed, and you can miss an explosion, a detonation of one of these missiles.)

One last thing about Al-Ahli Hospital: you can use the sort of fancy graphics, video editing, motion graphics, and all of the open-source material to describe a story, but you need to have the reporting right as well at the same time.

Resource and funding needs expand
Hadi Al Khatib
Founder of the Syrian Archive and managing director of Mnemonic, an NGO that collects, archives and verifies open-source materials

We are in a situation where first we disbelieve what’s happening online until we are able to collect information from different sources. That wasn’t the case in 2011 when we started in Syria, because we didn’t have that amount of disinformation, fake imagery, or AI-generated content. Now there’s much more information. What we see in Palestine now is a completely different scale—a very small geographical area, so much content published about documentation of war crimes there, very strong censorship, surveillance, and content moderation—that makes it really hard for us to understand when multiple incidents happen.

Another difficulty that we are facing is conflicts in different regions right now that are scaling up. New weapons are being used. It’s really important for us to identify certain weapons—expertise is needed. For that, resources and funding are needed, to be able to specifically track who uses those weapons. For us, this is really essential: to understand the potential perpetrators, the suspects, and so on.

Monstrous hybrids
Henk van Ess
Verification tools builder, open-source investigations expert and trainer, author of The Researcher’s Guide to Practical AI

The biggest problem for people like me is the hybrid AI, which is the hardest to spot. Basically, you have a photo that is 99 percent real, but then you change just 1 percent—which is very easy with the current tools. So you have an atrocity, a soldier, a uniform, and some kind of badge on the uniform—and you just change that badge.

If you want to fight it, detectors alone are not enough. You also need a detective. I scrutinize a lot of photos, and I make tools for journalists that have to do it automatically in big systems like Mimir. I found out that if you balance the detectors with a detective, then you slowly can make sense of stuff. If you only trust a detective who did old-fashioned work, you will probably fail, and if you only use a detector, you probably also fail.

Preempting lies
Eliot Higgins
Founder and creative director, Bellingcat

With the Alex Pretti and Renee Good shootings, we made a decision. Rather than waiting twenty-four hours to get an article out, we decided to publish our findings as we were making them because there was an ongoing discourse, and we knew there was going to be information from the US government that was highly likely to be completely untrue.

So we wanted to preempt those lies by publishing what we were able to verify through our social media channels. And also the way information is distributed means that when you publish a video like we published, showing the syncing videos of Alex Pretti being disarmed and shot, it became almost like a meme-able object within these kinds of arguments that are happening online between different parties about what happened. But it also informs the journalists who are then asking questions of the White House. I think we’ve got a better understanding of how, when we’re publishing, it actually affects different types of audiences and in what context that’s happening.

OSINT theater and grifters
Manisha Ganguly
Investigations correspondent and visual forensics lead for The Guardian

Illustration by Evan Grothjan.
 

OSINT has become expensive. When I started doing OSINT, it was something that you could do with a laptop, an iPhone, and maybe a camera. Now you need very expensive software suites. We’re delving into what I call OSINT theater, where we’re less focused on the actual outcome of the investigation. We’re so concerned with showing off how smart we are, and what these new skills are, that we’re losing sight of why we do these investigations. We do these investigations for accountability, and that is completely missing from the entire OSINT community, as I see it.

At the same time, you’ve got the emergence of OSINT grifters, these military or tech bros who are coming into this space who don’t exactly have any specific understanding of the history or the geopolitics of the conflicts that they’re claiming to cover. And what they’re effectively doing is amplifying random breaking news and unverified videos and contributing to the sort of tsunami of misinformation and disinformation we’re dealing with.

A holy shit, we have a problem here moment
Craig Silverman
Cofounder of Indicator, a publication and platform focused on digital investigations

In 2014, I did a research project with the Tow Center at Columbia, where I was looking for examples of unverified claims that were being reported, typically by news websites, but also, to a certain extent, spreading on social media.

I noticed 100 percent hoaxes and hoax news stories that were getting insane engagement on Facebook. That was a holy shit, we have a problem here moment.

Over the past twelve to eighteen months, the manipulation tactics have become really mainstream; there’s now an Andreessen Horowitz–funded TikTok bot farm. There are still policies preventing this, but they’re not enforcing policies around disclosure of advertising; they’re not going after people who are out there hanging up a shingle, renting accounts, renting ad accounts. So we have a massive volume of AI-generated slop and deceptive content.

Satellite imagery meets shoe leather
Imogen Piper
Visual forensics reporter, Washington Post

The problem with synthetic media is that it’s eroding people’s trust in the image. I’m less concerned about having to prove to people that something is synthetic. What I’m more concerned about is how we prove to people that something is real, because that becomes trickier when people’s trust in evidence is eroded, and our job is to point to evidence.

Combating misinformation and fact-checking things is going to come into our jobs more and more. However, I think that open-source investigations shouldn’t get sidelined by this. We shouldn’t be debunking every viral AI video that’s created.

We are in a place where I can collaborate with reporters in the field and corroborate information. And that just becomes so much more valuable—and also it means our stories center the human side of things.

Moving closer
Robert Trafford
Deputy director of Forensis e.V and assistant director of Forensic Architecture, interdisciplinary research agencies operating across human rights, journalism, architecture, art, academia, and the law

Even in a geopolitically significant context, like the work we’ve been doing the past two and a half, three years in Gaza, most of the material is coming through not-open social media, like Telegram, group-based social media, and social media channels where our access to certain groups is also the result of the trust that people already have in us.

We have tended toward an effort to become closer to our subjects and the subjects of our stories, and also closer to our audiences. We use a lot more witness interviewing, in kind of a process of collaborative reconstruction, which is then always anchored in the traditional tools of open-source investigation.

‘The Elon Musk of it all’
Jack Sapoch
Open-source investigator, Lighthouse Reports

I think this moment crescendos with the Elon Musk of it all—that Twitter, especially, was such a home for the open-source community, something really beautiful: horizontal, collaborative conversations. By the time he came around, there were completely different things being incentivized. You can see a number of open-source digital investigations being published, and stuff will go out and people just disagree with it. It’s really tough because there’s never a clear conclusion, or very rarely there is. The outcome is this slow muddying, or watering-down, of how external people might perceive this field, going from this implicit, more neutral-feeling language of truth to something that feels partisan, or something that somebody can look at from different angles and arrive at different conclusions.

A discipline gets its legs
Brad Samuels
Founding partner of SITU Research, a visual investigations practice

Illustration by Evan Grothjan.
 

Six or seven years ago it started getting its own traction and legs as a discipline—just in time for evidence to have a crisis in terms of authenticity and meaning. It’s a weird thing to emerge and get footing as a disciplinary identity and then, “Now what are we actually doing?” because there’s been this paradigm shift in information in the middle of it. I don’t know where it’s going, but AI will certainly be part of it since it’s such a massive epistemic shift.

Yeah, we are building new tools. We are thinking about what the security protocol should be for something that uses Gemini versus an offline LLM. But things that are fundamentally changing are so much bigger than that. Taking stock has to happen every three or six months.

Acknowledge biases
Alexa Koenig
Director and cofounder, Investigations Lab, University of California Berkeley Human Rights Center

We’ve got to figure out how we pivot in this moment, because what we used to use to signal legitimacy is not going to work, if it is still working at all.  I think psychology is going to be really important to the next wave—all the studies that are being done right now on video evidence and the biases that we bring as humans to our perception. How we can begin to counter some of those biases is going to involve things that journalists and others are going to need to know, or at least have in the back of their mind, or be exposed to in order to avoid stepping into mistakes in their analysis of visual imagery.

We’ve always had to engage with messy information
Emily Tripp
Executive director, Airwars, a nonprofit watchdog that documents war using open-source techniques

The thing that’s always struck me about the field is the promise of democratization, that this could be something that you didn’t have to have paid a load of money to do, to go to this school, to be part of that network, or have these sources in order to do this work. Instead, it’s something that anybody can do. It’s a really, really nice promise. But now the New York Times’ Visual Investigations team will invest a huge amount of money and resources and will have access to the highest-quality satellite imagery, and will be put on equal footing with those investigations as, say, nonprofits. And so then suddenly we’re competing in a visual language using essentially the same tools, but it’s not democratic because it isn’t something that everybody can do.

I’m not looking into the future going, “Oh my gosh, this is getting harder and harder.” I’m going into the future thinking, “Okay, the skills that we need, the people we need to do this work, we might have to be a little bit more imaginative about that.”

Has America ever needed a media defender more than now? Help us by joining CJR today.

Emily Goligoski is a New York City–based researcher and writer, and previously led audience research at The Atlantic. She teaches about the business of media at Columbia and CUNY.
Marisa Mazria Katz is a journalist and radio reporter who founded the Eyebeam Center for the Future of Journalism. She teaches in the Rhode Island School of Design’s Computation, Technology, and Culture department.
Jake Charles Rees is a curator and head of the Logan Programme at the Centre for Investigative Journalism. He teaches visual investigations at the Royal Academy of Art, The Hague, and University of the Arts, London.

More from CJR