Perhaps it was the early hour—maybe the panelists hadn’t had their morning coffee yet—but the mood seemed subdued at the “Quality, Quantity, and Mass Content” session at Thursday’s paidContent conference—and some of its participants seemed defensive.

In the day’s agenda, the panel promised to address the question, “When a business can be damaged by a flick of the Google algorithm and when everything is SEOd to the max, what’s the advantage of mass content?” ZDNet editor in chief Larry Dignan, moderator of the discussion, started out with the recent announcement by Google that its search engine algorithm had been tweaked to suppress site-scraping and low-quality content.

Dignan said that he had not seen much change in his sites’ traffic, and the rest of the panel said much of the same. Lewis Dvorkin, CPO at Forbes, said that it was too early to tell how it would affect traffic, but that he didn’t really waste too much time thinking about it. Reuters president Chris Ahearn said that he was still waiting to hear how their client sites were faring after the change. Jason Rapp, president of Mahalo, which produces educational videos for the web, expressed long-term optimism but conceded that the company did make some immediate layoffs in response to the hit their posts took in traffic.

The head of Yahoo’s Contributor Network, a.k.a. Associated Content, Luke Beatty, said that he’s seen a third of the content in the network pushed up by the algorithm switch, and two-thirds of it pushed down. Since that content is contributed by—and promoted online by—freelancers, the fate of each post rests on the authors. “It’s really about how the contributors are managing their library, for them it’s very significant,” said Beatty. Translation: if the individual writers aren’t putting the effort into bringing readers to their posts—through SEO, through Facebook, through Twitter—then they’re not going to get the clicks.

A blog post by Google’s Amit Singhal and Matt Cutts last week announcing the algorithm fix was titled “Finding more high-quality sites in search.” An excerpt:

Google depends on the high-quality content created by wonderful websites around the world, and we do have a responsibility to encourage a healthy web ecosystem. Therefore, it is important for high-quality sites to be rewarded, and that’s exactly what this change does.

High-quality. But what does that mean? For that matter, is the label “content farm” fair? Is SEO an underhanded enterprise, or does it help readers find exactly what they’re looking for? If we search on Google for “What time does the Super Bowl start?”, will we actually be disappointed when we are directed to a post on HuffPo that answers our question?

So after the panel’s slightly somber start, Dignan asked the group to help define “quality content,” and to opine on “whether we’re comfortable outsourcing the judging of it to Google.” What’s good, and who gets to decide? Can an algorithm be the arbiter of taste?

Yahoo’s Beatty answered first: “I think quality has to rest in the hands of the consumer, and in the editor that’s reviewing the piece,” he said. “It’s on a content by content basis.” He added that the definition of quality for a piece of news will be very different for the definition of quality for an evergreen piece, like a recipe, for instance. Consumers will decide whether it’s useful to them, and editors will decide whether the contributions that come in meet their standards.

Mahalo’s Rapp said that he saw quality differently; being an educational company, he thought of a piece as high-quality if a viewer had learned something by watching or reading it—how to play a song on the guitar, or how to solve a math problem, for instance. For him, “quality” is synonymous with “expertise.”

News, of course, does not have a long shelf life, said Ahearn. So at Reuters, quality is defined by whether a reader can act on a piece of news: whether they trust its accuracy and its ethics. Ahearn added that he is thinking about ways to include more freelancers and contributors to be part of their network, to expand their reach.

“I’ve been looking at the Demand [Media] model…the Associated Content model, and asking, ‘What does this mean for news?’” said Ahearn. “I think for a journalistic organization to adapt to the changes that are before us, we need to be more flexible.”

Forbes’s Dvorkin said that he has two different definitions of quality: one for print, and one for digital. For the web, what’s most important is “timeliness, constant updating, relevance to the audience for the topic you’re trying to build, aggregating, your expert voice, and…getting contributors and staffers engaged in the conversation. They need to react and talk to their audience.” For print, “a finite product,” the focus is on accuracy, because there are no updates after the fact.

Dvorkin then shifted the conversation, attempting to underplay Google’s significance in the equation. “You said something earlier about Google deciding quality—I don’t think of it that way,” he said. “Especially moving forward, I look at it as your friends and your colleagues deciding quality. I spend a lot more time focused on social, and where our strategy is coming from there, than I do on worrying about the things that Google is doing. Because you can have an impact on traffic coming via Twitter or Facebook or Foursquare or whatever…. Not that we’re not thinking about Google and doing what we need to do, but to me, that’s yesterday’s story, not tomorrow’s story.”

Ahearn agreed. “Google is an elephant in the room; it can distort traffic,” he said. “It becomes a game…but social is going to be far more interesting for us. That’s a lot more nuanced.”

ZDNet’s Dignan did not seem convinced. “Don’t you still get most of your traffic from Google?” he asked. “We put a lot of effort into social, cultivating our Facebook page, but as far as payoff at the end of the day, when you look at referrals, it’s still dominated by Google.”

Dvorkin and Ahearn shook their heads. “But I think you’re defining social wrong,” said Ahearn. “If it’s about cultivating your Facebook page: So? It’s about cultivating your audience. How are they taking your content, how are they carrying your message?”

One thing that the panel seemed to agree on was that the number of page views a post got was not the measure of its quality. “I just think the page view obsession is nonsense,” said Ahearn. “We have to do it right now. But it’s not a measure of value, at the end of the day…. I think that we continue—collectively, as an industry—to make the mistake of hard-wiring everything into page views and advertising.”

Dignan challenged that: “You can’t tell me Forbes doesn’t care about page views,” he said to Dvorkin.

“Well, I don’t obsess over them,” Dvorkin answered. “I lived at AOL, I obsessed over page views at AOL. But [at Forbes] we’re all about building an audience.”

But someone in the audience had a good question: If the revenue model throughout the online content industry—and the ad agencies that service it—still all generally rely on page views and clicks-per-million as metrics of success, then how do we transition out of that? How do we actually “measure engagement,” when it comes to dollars and cents?

“I don’t know,” Ahearn admitted. “But it’s about changing the conversation…. We get off the ‘juice’ as the easiest thing to measure. It’ll be a collective movement over time. But I think if we don’t get honest and talk about it, and say it’s nonsense, then it won’t change.”

If you'd like to help CJR and win a chance at one of 10 free print subscriptions, take a brief survey for us here.

Lauren Kirchner is a freelance writer covering digital security for CJR. Find her on Twitter at @lkirchner