Yesterday’s New York Times featured a great article by Patrick McGeehan, headlined “Tallying Stimulus Jobs Not Easy in New York.”

That wasn’t so hard to acknowledge, was it? Sometimes it’s important for the press to write about the limits of its knowledge, and this is one of those times.

“Hardly a day has passed in the six months since President Obama signed the $787 billion stimulus package without an elected official announcing how a chunk of the $5 billion or so that is coming to New York City will be spent,” McGeehan writes in the lede of his story-behind-the-story.

And with stimulus funds being tracked all over the country (see here, here, and here) hardly a day goes by without local reporters wrestling with how to avoid taking those elected officials’ job creation estimates at face value.

New York City comptroller William C. Thompson, who is running against Michael Bloomberg for mayor, says in the article that the the mayor’s office has resorted to “puffery” in projecting how many jobs the stimulus will create or save in the city.

McGeehan goes on to explain that different metrics are being used to measure the impact of stimulus dollars in different projects. For example, New York City is attributing nearly 5,000 jobs to a construction project costing $175 million to replace ramps that lead to the main ferry terminal at Staten Island. Meanwhile, a planned railroad tunnel under the Hudson River, which will cost about fifty times as much, is expected to produce only 6,000 construction jobs.

On the ferry terminal project, the city cited a United State Department of Transportation standard of about twenty-eight jobs created, directly and indirectly, for every $1 million spent. But a more conservative estimate recommended by the president’s Council of Economic Advisers projects a much smaller, and some say more accurate, ratio of eleven jobs per $1 million—less than half the DOT’s estimate.

To some degree, you can measure the direct creation of a job. (Measuring the indirect creation of jobs is another thing entirely; we’ll get to that.)

But not all jobs were created equal. So how do you define “job”? Is it a job that didn’t exist before the stimulus money landed on the doorstep, or is it an existing job that can continue to exist thanks to the windfall? Since the stimulus is a one-time infusion, how long do these jobs last? A month, a season, a year? Are the jobs full-time or part-time? What salary do they pay? Who is being put to work—a teenager who wouldn’t otherwise have a summer job, or a head of household who would otherwise not be able to put food on the table? These details matter, and they should used to qualify the word “job.”

When a July release from Congress’s Transportation and Infrastructure Committee credited transportation-related stimulus projects with creating 49,377 jobs, ProPublica called out the fuzzy math involved in counting stimulus jobs, asking what constitutes a “job” and whether simple head counts are accurate:

Besides failing to distinguish between part-time and full-time jobs, head counts are subject to another error: Construction workers often rotate among projects, which can lead to double- or triple-counting when project totals are added together. State workers who oversee multiple projects also can be counted more than once.

In many ways, the exact impact of stimulus money is unknowable. Sure, maybe the indirect, trickle-down, butterfly effect created by stimulus funded jobs means a construction team buys its lunches at a local deli or supplies at a local merchant, keeping a clerk there on the payroll, which means that clerk can pay for his kid to keep taking karate lessons, which means that his kid’s dojo stays in business, which means the sensei keeps ordering cement cinderblocks for his pupils to smash, which means a cement company stays in the black, and so on and so forth. But until the government starts paying cash for stimulus projects in marked bills that can literally be tracked from pocket to pocket the way scientists tag monarch butterflies to measure migration patterns, that sort of thing is impossible to know. Not that we shouldn’t try to get close.

Alexandra Fenwick is an assistant editor at CJR.