Advocates for open and transparent government are quick to note that no American presidential administration has, in practice, been enthusiastic about reducing secrecy in the executive branch—for some obvious and sometimes quite legitimate reasons. There are secrets that almost everyone agrees should remain secret. But secrecy must be balanced with the citizens’ right to examine the operations of their government—to learn, to improve, to enforce, and sometimes to shame. That’s especially true when there are political or bureaucratic incentives for secrecy that deserve far less respect than true matters of national security. And despite the bipartisan resistance from those in power, the arc of history has trended, if unevenly, toward openness. Claims of excessive secrecy have become a tried and true political battering ram, easily wielded by the party in opposition. Technological evolution has not only made the dissemination of information easier and faster, but also has heightened our appetite for disclosure. The trend isn’t confined to the political sphere. Betty Ford’s frank discussion of her struggles with cancer and alcoholism in the 1970s marked a new era of openness in our personal medical lives, and the invention of the personal video camera spawned a cottage industry around moments—gaffes, goofs, tragedies—that were once private.

Against that backdrop, there is wide agreement among journalists and openness advocates that the administration of George W. Bush was an aberration, at least in the modern era. Bush and his advisers came into office with a broad vision for a more powerful, less accountable executive branch—a vision that has long been popular in conservative legal circles. Presidential power ebbed after Watergate, when some of the strongest laws promoting transparency were adopted by Congress, reducing the executive branch’s ability to do its work in secret. Even before the terrorist attacks of September 11, 2001, the Bush administration—and especially Dick Cheney, who assumed unprecedented power as vice president—enacted policies and waged court battles to roll back what they saw as unjustified infringements on presidential power, and to reduce the oversight and transparency that had been forced upon the presidency.

Then, just eight months into Bush’s first term, September 11 gave the administration what became its defining rationale for a draconian clampdown on the free flow of government information to the public. Presidents traditionally act with the freest hand in matters of national security and, following the attacks, secrecy became both a means to an end and a goal in itself. Information on transportation and energy infrastructure, once easily accessible on government Web sites, was removed. The Justice Department invoked a state-secrets privilege in an extraordinarily wide range of cases. The administration and its conservative allies waged a rhetorical war on journalists who worked to learn and disclose the government’s secrets. Legal justifications for the administration’s detainee and warrantless wiretapping polices remain shrouded in secrecy today.

Legally, some of the administration’s greatest incursions against transparency were made with remarkable ease, only requiring executive orders or directional memos from senior White House staff—a fact that should hearten open-government advocates who are optimistic about the potential of Barack Obama’s administration to redress these grievances. To take but one example, though it is one especially dear to journalists, much of the damage done to the Freedom of Information Act under Bush could be undone with the stroke of a pen, and Obama, in the campaign and the transition, has suggested he’ll do just that. Other changes abetting excessive secrecy that resulted from court rulings or emerged from bureaucratic traditions are far more entrenched, and will not afford easy or quick fixes. Indeed, the struggle between openness and secrecy will continue in the coming years.

One ruling that will be hard to reverse has roots reaching back before 9/11. In February 2001, shortly after Bush’s inauguration, Vice President Cheney formed a task force to help develop a new energy policy. Even before the policy was announced that May, environmental groups worried that the policy proposals would tilt heavily toward industry concerns. At the request of two Democratic members of Congress, what was then known as the General Accounting Office (now the Government Accountability Office), a nonpartisan investigative arm of Congress, asked Cheney for basic information on the task force—who it met with, where and when, and for minutes and other records of its work. Cheney, through his counsel David Addington, denied the request, claiming that the GAO did not have the authority to request the documents, despite a history of previous administrations responding to similar requests.

That year, Judicial Watch, the Natural Resources Defense Council, and the Sierra Club filed two suits seeking the same records, one under the Freedom of Information Act and another under the Federal Advisory Committee Act (FACA), a 1972 law requiring that materials generated by public-private advisory committees be available for general inspection. After a nearly four-year-long court battle that included a stop at the U.S. Supreme Court, federal courts upheld Cheney’s right to withhold the records and also denied the groups’ challenge under the Advisory Committee law.

The environmental groups ultimately secured most of the documents by a circuitous route—from the Energy Department through Freedom of Information Act requests. But the case was considered a major blow to the Advisory Committee Act, which had already had its reach trimmed by a series of rulings that culminated in the Cheney decision. “It’s just been shredded,” says David Vladeck, a veteran public-interest lawyer who teaches at Georgetown. “Read the statute and you’d think one thing; and you read the way the courts have interpreted it, and it’s no longer an effective safeguard.”

In hindsight, it’s easy to see how the task-force battle was a deliberate attempt to expand the executive branch’s ability to operate in secret, rather than a simple political tussle over energy policy in the wake of Enron’s collapse. As Bruce Montgomery, an archivist at the University of Colorado at Boulder who has written widely on presidential materials, wrote, “The case marks one of the Bush administration’s most significant victories not only in reasserting the prerogatives of the presidency, but also in cloaking the executive in greater secrecy.”

President Obama, in the course of his election campaign, promised that as many meetings “as possible” between federal agencies and lobbyists would be available as Web videos. But he’s had little specific to say on FACA, or how his administration might treat advisory commissions with private members who don’t happen to be federally registered lobbyists.

Another major setback for the interests of openness, and again one with roots that clearly reach to a time before 9/11, was Bush’s executive order concerning the Presidential Records Act, passed in 1978 to ensure public control over the records presidents create while in office. (So enshrined was the principle of private ownership before the Records Act was adopted, that past presidents sometimes destroyed their papers or willed them to their heirs, who were free to sell them for a profit.) Presidents from both parties have been, to varying degrees, unenthusiastic about the law, and it has suffered from a string of court challenges and executive orders restricting its authority. The Clinton administration, for example, continued a legal effort initiated by George H. W. Bush to limit the disclosure of Bush’s records. It lost that battle in 1995, in a decision that seemed to clearly state that ex-presidents surrendered control of their administration’s records twelve years after they left office.

When George W. Bush took office twelve years after the end of Reagan’s presidency, however, White House Counsel Alberto Gonzales sent the national archivist a note instructing him that the new president wished to delay the release of any Reagan files. The closing line? “This directive applies as well to the Vice Presidential records of former Vice President George H. W. Bush.” Then, in November 2001, the administration issued a new executive order that declared that records from prior administrations would not be released unless the sitting president expressly approved it. But that order went even further: endowing past presidents with the power to keep their documents from being released even after the twelve-year threshold. Vice presidents, too—a category that included Bush’s father at the date of the order and now includes Dick Cheney—were given the authority to hold their records. As if that weren’t hubristic enough, Bush’s order allowed past presidents and vice presidents—or their heirs—to pass on their withholding privileges to representatives in perpetuity. “It was essentially overturning the Presidential Records Act,” says Thomas S. Blanton, the director of the National Security Archive, an independent research institute at George Washington University. Obama has promised to nullify the Bush order, which he could do with a simple executive order.

The National Security Archive and a coalition of sixty organizations that work on transparency and openness issues have proposed another executive order that Obama could issue early in his term—one that would create a task force to determine how to rein in the worst excesses in the information-classification system, the federal government’s primary tool for official secrecy. Over many years, a perennial series of blue-ribbon commissions has suggested that over-classification is a serious problem in the federal government, not only for the public’s historical interest but also for data-sharing among agencies. For one, the 9/11 Commission warned that such “secrecy stifles oversight, accountability, and information-sharing.” Former congressman Lee Hamilton, who was vice-chairman of the commission, estimated that about 70 percent of the information he viewed was “needlessly classified”—a shockingly high portion given the sorts of records the commission needed to do its work.

Officially, federal law only describes three levels of classification: “top secret,” “secret,” and “confidential.” The process is statutorily overseen by the Information Security Oversight Office at the National Archives, which collects data on the quantity of classified information, monitors agencies’ compliance with classification rules, and handles appeals of classification decisions. A limited number of agencies and individuals are authorized to classify material, and while there are widely used provisions for exemptions and extensions, classified information is supposed to be automatically declassified after ten years.

The amount of classified information, and the number of people authorized to deem it classified, have been expanding since 2000. The departments of Agriculture, Health and Human Services, and the Environmental Protection Agency, for example, were authorized to classify for the first time early in the Bush administration. In 2007, the most recent year for which records are available, a report by the Information Security Oversight Office (ISOO) catalogued more than twenty-three million classification actions government-wide; in 2000, the number was just over eleven million, although some of this increase can surely be attributed to the growth of digital communication.

In 2003, classification’s durability and reach were extended by another Bush executive order that mandated a three-year moratorium on many automatic declassifications, allowed the CIA director to overrule declassification decisions made by the Oversight Office, and expanded classification for information provided by foreign governments—a category that could include such historical treasure troves as diplomatic cables and the like. It also made it easier for agencies to reclassify information that they had previously declassified, as long as there was some plausible way to retrieve the documents from the public realm.

In one incident that illustrates the reach of this order, federal agents removed documents from the personal papers of the late Senator Scoop Jackson, housed at the University of Washington. Far more troubling was the revelation in 2006 that more than twenty-five thousand documents had been pulled from the stacks at the National Archives and Records Administration, which are open to the public. Matthew Aid, a part-time historian who first noted the disappeared documents, catalogued a bizarre list of what had gone missing: cables documenting a widely known intelligence failure from the Korean War; a 1948 message chastising the State Department for not predicting riots in Bogotá; talking points on how to handle questions about Japanese peace offers before the end of World War II; etcetera. Archive officials conducted an audit and determined that a third of the documents that had been pulled were not eligible to be reclassified, even under the new Bush standards.

Alongside the official classification system exists a murky system sometimes called “pseudo-secrecy.” More formally known as “sensitive but unclassified” or “controlled unclassified” information, it functions with little regulation, monitoring, or clear force of law. Attempts to fully measure the use of this category are frustrated by the fact that there is no single definition for what qualifies as sensitive-but-unclassified (SBU) information.

Concerns about sensitive-but-unclassified information date back at least to 1972, when a House committee held hearings deploring the ways that similar labels were being used to keep information from coming to light under the Freedom of Information Act, which carries an exemption for properly classified documents. Explicit presidential support for such pseudo-secret labeling dates to a 1977 presidential directive on telecommunications technology by Jimmy Carter, and it has been used in every subsequent administration.

In 2007, a Defense Department official charged with improving interagency information sharing estimated that there were 107 different labels in the category—from “Official Use Only” to “Sensitive Internal Use”—and none is monitored by the Oversight Office, which means that there are no official numbers to describe the trend. But openness advocates and some journalists have suggested that the Bush administration has significantly expanded both the number of such labels and the volume of documents being labeled. For example, in April 2002, officials at the Department of the Interior instructed their employees that “all unclassified DOI systems” should be “considered SBU.” The 2002 law authorizing the creation of the Department of Homeland Security specifically said that all scientific research produced by the department, wherever possible, should be unclassified, but President Bush used a signing statement, which spells out how the executive branch will interpret and implement a law, to make clear that he intended to mark much of that information as sensitive. In 2006, when the National Security Archive conducted a study, via FOIA, of the use of sensitive-but-unclassified labels at thirty-seven major government agencies, it found that roughly two-thirds of the SBU programs were operating without any statutory justification. Only one program had an automatic procedure for removing the designation, as is required by law with classified information. According to Barton Gellman, a reporter with The Washington Post, Vice President Cheney’s office routinely took to stamping papers with “Treated As: Top Secret/SCI,” a designation some classification experts believe Cheney invented. “At least with classification, you have ISOO overseeing what gets classified and what doesn’t get classified, keeping track of how much is classified,” says Rebecca Carr, who covered access and First Amendment issues for Cox Newspapers from January 2005 to May 2008. “The pseudo-classification category is like the Wild West. There’s nobody watching the store.”

The administration’s secrecy-related policy that most alarmed the journalism community was a memo issued by Attorney General John Ashcroft a month after the 9/11 attacks. The memo suggested that information officers at executive-branch agencies could deny FOIA requests as long as there was a “sound legal basis” for doing so. This represented a fairly regressive shift. Under the Clinton administration, requests were only to be denied if there was “foreseeable harm” in releasing the documents. Ashcroft’s memo sent a clear signal that information officers should feel free to expansively deploy FOIA’s nine exemptions to deny whatever requests they could. And the letter assured agencies that they would have the full backing of the Justice Department in fighting any lawsuits that resulted from such denials.

In 2002, the National Security Archive sought to clarify what effect the Ashcroft memo had on FOIA compliance and sent FOIA requests to thirty-five major agencies asking for documents that would illuminate how they’d handled Ashcroft’s new directive. The responses confirmed that most branches of the armed forces and the Nuclear Regulatory Commission had taken strong steps to expand the use of exemptions; that about a quarter of the agencies surveyed had taken some steps to implement the changes; and that just over half had disseminated the policy to their employees. A GAO study that interviewed 183 FOIA officers at twenty-five agencies found that 31 percent said their agencies were less likely to release documents sought by FOIA requests following the memo.

Whatever the practical effects of the Ashcroft memo in the early days of the administration, to the open-government community the memo had great symbolic importance, especially when it was combined with a separate memo issued in the spring of 2002 by Andrew Card, Bush’s chief of staff. The Card memo requested that agencies take, for obvious reasons, special care in disclosing any information related to weapons of mass destruction, and encouraged information officers to use the nooks and crannies of existing law and executive orders to limit the release of classified information that was scheduled to be declassified due to its age.

But Card’s memo did not stop at classified information. It also invoked sensitive-but-unclassified information, which has no specific legal recognition under FOIA’s exemptions. The memo suggested that information officers assess any requests for such information in light of the earlier Ashcroft memo. In other words, if information officers could shoehorn sensitive-but-unclassified information under one of the existing exemptions in the FOIA law, they could deny its release. Together, the two memos provided a legal framework for denying access to sensitive-but-unclassified information of any kind. “It really turned the concept of open government on its head,” says Andy Alexander, who was the Cox Newspapers Washington bureau chief for ten years and serves as co-chair of the American Society of Newspaper Editors’ Freedom of Information committee.

The Freedom of Information Act, adopted in 1966, is in many ways the lodestar of the open-government movement. Its premise is simple enough: that the work product of government is the property of its citizens. The history of the act’s implementation, however, shows how law in practice can be quite different from law as written. Anyone who has used the FOIA law knows that the mandated twenty-day window for an official response is a cruel joke. The National Security Archive periodically conducts audits—via FOIA—asking agencies for their oldest outstanding FOIA requests. In 2007, it found a handful of requests across five agencies that were more than fifteen years old. One had been outstanding for twenty years, or roughly half the life of the law itself. These superannuated requests are the exception, but that doesn’t mean the norm is compliance.

These problems predate the Bush administration, but there’s little doubt that they’ve grown over the last eight years. A 2007 study of the FOIA records of twenty-five agencies by the Coalition of Journalists for Open Government discovered a number of disheartening truths. The data suggested that many agencies took longer to respond to requests in 2006 than they had in 2000, and that at least half of the requests at fourteen agencies received a response after the twenty-day window the law requires. In 2006, only 60 percent of FOIA requests processed netted a full or partial response; only 36 percent netted a full response. Both were the lowest numbers since 1998, when the relevant data first were collected. Given that trend, it’s not surprising that the overall number of FOIA requests at these agencies also fell by 25 percent since 1998. “A lot of young reporters, especially, know that there’s something called the Freedom of Information Act, but that’s pretty much the extent of their knowledge,” says James McLaughlin, an associate counsel at The Washington Post. “Often, reporters file FOIA requests and if they get anything, then the feeling is ‘Wow, I got something; that’s really cool.’ And if they get it seven months later, they’re still happy. The expectation is not that this is your statutory right. The expectation is that it’s a bonus.”

Despite this woeful picture, FOIA remains one of investigative journalism’s most-favored tools, especially when it can be paired with lawsuits to shake loose information. Journalism’s emotional attachment to FOIA might have something to do with the industry’s intimate history with the passage of the act. In the 1950s and early 1960s, a coalition of journalists that included Washington Post executive editor J. Russell Wiggins and Clark Mollenhoff, a Pulitzer Prize-winning investigative reporter for The Des Moines Register, worked closely with the act’s congressional sponsor, rallying public support, writing editorials, and even lobbying public officials to ensure first its passage, and then that Lyndon Johnson signed it.

Journalistic norms of objectivity usually forbid advocacy. But freedom of information is an issue for which exceptions have been made. So when some forty years after its passage FOIA seemed threatened, journalists again devised ways to systematically address the relevant issues of openness and transparency in the federal government. Sunshine Week, launched in 2005 by the American Society of Newspaper Editors, was the first major effort by the journalism community to counter the encroachments on openness in the early days of the Bush administration. The goal was, and remains, to spur a national conversation about access issues via a week of news coverage; in 2008, Sunshine Week had about a thousand participants, including radio, television, and Internet outlets.

But there still was no coordinated effort by the various strands of the journalism industry to address openness issues. Protest petitions circulated frequently, and some press associations mixed transparency advocacy with more parochial concerns like postage rates. “It wasn’t sufficient,” says Rick Blum, a former director of OpenTheGovernment.org, a coalition of groups interested in governmental transparency. “There was a moment when the press-freedom groups and the freedom-of-information groups said, ‘We’re going to get clobbered if we don’t start cooperating.’ ”

In June 2003, media and advocacy groups met in Washington. Pete Weitzel, who as managing editor of The Miami Herald had organized Florida newspapers to lobby the state legislature on access issues and defend access rights won in court, proposed an analogous national organization. With that goal in mind, and a grant from the Knight Foundation, the Coalition of Journalists for Open Government was founded. Made up of about thirty press associations, large and small, the coalition sponsored sign-on letters protesting encroachments against openness and commissioned several detailed research projects to examine the damage done to transparency under Bush.

The coalition spurred the major industry associations—representing newspaper editors, publishers, alt-weeklies, and broadcasters—to come together with The Associated Press in 2005 to form their own outfit, the Sunshine in Government Initiative. As the coalition’s grant winds down, the Sunshine Initiative has emerged as the preeminent press voice in Washington on open-government issues, complete with its own lobbyist and an active and regular presence on Capitol Hill.

The timing of the Sunshine Initiative was fortuitous, as over the course of the next year the press would face some of its greatest challenges of the Bush era. Several high-profile stories—especially The Washington Post’s piece on secret CIA prisons, The New York Times’s exposé of the National Security Agency’s warrantless wiretapping, and a series of reports in The New York Times, the Los Angeles Times, and The Wall Street Journal on government monitoring of international financial transactions—raised hackles in the Bush White House. “Those stories really fueled congressional criticism of the press and reporting of unauthorized disclosures of classified information,” says Blum, now the director of the Sunshine Initiative. “And we were spending a lot of time defensively trying to ensure that Congress did not pass new, overly broad revisions to the antispying-espionage laws.”

It was a trying time. In May 2006, Attorney General Alberto Gonzales hinted to ABC News that prosecutions were possible for journalists who disclosed classified information. That summer, seventy-one congressional representatives asked the Speaker of the House to strip The New York Times of its press credentials in retaliation for reporting on surveillance programs. Others wanted to go beyond the symbolic and pass new laws restricting publication or making prosecution easier. SGI’s members took to Capitol Hill, meeting with legislators and offering testimony. They credit this educational effort with helping to forestall legislation that would have upset the delicate legal balance that has governed reporting on national security for decades and made it harder for the press to report and publish important stories.

A lot of the Sunshine Initiative’s battles have, in fact, been defensive, but the group has been able to play offense on FOIA, as well. It worked with senators John Cornyn and Patrick Leahy, as well as many others in Congress, to help pass a bill creating an independent ombudsman for the act that could help referee long-delayed requests and improve the monitoring of FOIA compliance government-wide. Bush signed it in on New Year’s Eve 2007, but in the following year’s budget, he inserted language in a section dealing with Commerce Department appropriations that would move the FOIA ombudsman’s office from the National Archives, where it was authorized by the bill’s plain language, to the Justice Department. Since that’s the very department charged with defending the government’s FOIA decisions, it seemed designed to neuter the ombudsman. The Sunshine Initiative, and many others, have called on Barack Obama to move the ombudsman back to the National Archives; it is part of a four-point agenda on transparency that the Initiative put together for the new president—whether he wants it or not. Similar white papers are circulating in Washington from the broader transparency community.

There are signs that Obama could be an ally. He hit most of the right notes during his campaign, and his transition Web site suggests that there will be an executive order to roll back Bush’s changes to the Presidential Records Act. It also promises greater disclosure of the sort of public-private communications that Cheney fought so hard to keep secret. The mood is cautiously optimistic, but many of the veterans of the transparency movement aren’t about to let their guard down. “As we look ahead to the Obama administration,” says the ASNE’s Andy Alexander. “I’ve been in a few meetings where people almost say, ‘Happy days are here again.’ And I have to caution them and say, ‘You know, let’s wait and see.’  No one likes being overseen.”

Thanks to the Fund for Investigative Journalism for its support for this article.

If you'd like to help CJR and win a chance at one of 10 free print subscriptions, take a brief survey for us here.

Clint Hendler is the managing editor of Mother Jones, and a former deputy editor of CJR.