When the New Orleans Times-Picayune announced it would cut back its printed editions to three days a week and shift to a digitally centered model based on free content, the response was immediate, loud, and, with few exceptions, hostile.

Some complained the paper and its parent, Advance Publications, was abandoning the large segment of its readership—mostly poorer and less well-educated—that relied on the printed product. Others lamented the staff cuts that accompanied the changes and the ham-fisted manner in which they were carried out. (For all the background, read Ryan Chittum’s deep dive from last spring.) But a unifying concern, underlying all the others, was for the quality of the news itself. The “digital-first” model—dependent on a high volume of new posts to drive ad-generating Internet traffic—would, many believed, result in a shoddier product.

Since the 2012 changeover, discussion about the Times-Picayune, now known as Nola.com, has been one long argument. Is the news content now worse: softer, more sensational, more trivial, less well-sourced? Or is it, as defenders argue, just different: faster, more immediate, livelier, more conversational, more engaging? Or what?

Now a class of undergraduates led by a Tulane communications professor has weighed in with study that seeks to settle the debate. Professor Vicki Mayer and the students of her Media Analysis class last fall completed what’s known in academic circles as a content analysis designed to compare the quality of the printed Times-Picayune of 2011 with the newspaper of 2013 as well as the allied digital platforms—website, tablet, and smartphone—now given primary emphasis and designed ultimately to supplant the paper altogether.

The results, while not the last word, are nonetheless troubling. The study provides solid support for the idea that digital platforms—even from the same news organization—require, or at least incentivize, softer, less-well-sourced, poorer quality news.

Nola, for its part, says the study’s methodology is hopelessly flawed and in no way accepts its conclusions, as we’ll see.

The class studied a month’s worth of news coverage three days a week—Sunday, Wednesday, and Friday—for four consecutive weeks in October 2011 and October 2013, about a year before and after the digital overhaul. The study catalogued staff-bylined stories appearing on A1 and B1 in print in both years and took screenshots of banner stories and those in the curated, upper part of the homepage of the Website (which can contain three to 10 stories, the study says), as well as those on the homepage of the mobile apps, tablet and phone. The screenshots were taken between midnight and 10am.

The two main categories for stories were by news beats (courts, cops, politics, sports, etc.) and by whether they represented “hard” vs. “soft” news. The latter is defined as:

…stories that were trivial, generally focused on entertainment and sports. In addition, we looked at the emergence of opinion columns and reader-response stories as two genres that have arisen on the homepages of digital formats. These pieces could be considered soft news, following James Fallows’ critique of opinion-based stories as easier and more speculative, rather than reporting-driven stories.

Importantly, the study did not track the digital platforms over a 24-hour period, which, as we’ll see, Nola believes to be a fatal flaw. I don’t agree.

I should disclose that I consulted in the design of the project. Mayer approached me (and others) last summer asking my thoughts on the idea of such an analysis, and we talked a couple of times and exchanged emails over what categories of stories the analysis should be looking for. Mayer sent me some preliminary data in October, but I didn’t have any further participation until I got the final results last month. This is the first time the results are being published.

Since this isn’t my data, I tried to vet it, including by sending it (with Mayer’s concurrence) to Jim Amoss, Nola’s top editor, who, to his credit, offered a detailed criticism of the study’s methodology, which I’ll quote at length. I say “to his credit” because I’ve made no secret of my opposition, both in principle and for its practical results, of the free-digital model, which Advance has been rolling out across the country, to devastating effect, in my opinion.

Still, there’s my opinion, and then there’s the Tulane data, and I’ll do my best to prevent the one from coloring my view of the other. I’ll also post the whole study so everyone can decide for themselves.

Here are the main findings:

—Overall, the 2013 news operation—predictably—produces a much higher quantity of content than did its 2011 predecessor, which had a much bigger staff.

—Importantly, the 2013 version of the printed Times-Picayune is not terribly different from its predecessor in terms of the type of stories covered.

Dean Starkman Dean Starkman runs The Audit, CJR's business section, and is the author of The Watchdog That Didn't Bark: The Financial Crisis and the Disappearance of Investigative Journalism (Columbia University Press, January 2014). Follow Dean on Twitter: @deanstarkman.