“Most companies would rather not be graded at all,” she says. “If you start talking to them too early, they’ll say, ‘Oh, please don’t do it. We’d rather you went and wrote another book.’ Or, ‘We’d rather you’d rank governments—they’re much worse.’”

That’s part of the reason it took as long as it did to release the first set of draft criteria: This one is coherent enough, MacKinnon says, to share with the world—and with the companies who will serve as the project’s case studies, to test out and polish the questions that will be used to create the actual rankings. She’s not ready to say exactly which companies will be included in the first big report yet—there’s a draft list that keeps changing, but “if you’re thinking of the most powerful Internet companies that are used by the greatest number of people, you can guess the obvious ones,” she says. She and her collaborators are also looking at including newer companies, less obvious ones, cloud computing leaders, widely used domain hosts, and companies that are household names in other countries, but not in the US.

At the same time, her strategy for keeping herself accountable is to keep talking to everyone who has a stake in the project’s outcome and to make the ranking project itself as transparent as possible.

“I was advised early on that we should put as much in the public as early as possible and not being all secretive about what we’re working on,” she says. “The companies are not going to be able to say they didn’t know it was coming.”

Disclosure: CJR has received funding from the Motion Picture Association of America (MPAA) to cover intellectual-property issues, but the organization has no influence on the content.

If you'd like to get email from CJR writers and editors, add your email address to our newsletter roll and we'll be in touch.

Sarah Laskow is a writer and editor in New York City. Her work has appeared in print and online in Grist, Good, The American Prospect, Salon, The New Republic, and other publications.