Proud and strange and so hopelessly hopeful.

There’s a fabulous story making the rounds of the Internerd about the Gulf disaster. This one is positing a “world-killing” event that might be brewing, as a result of the release of billions of gallons of oil from the BP well over a period of months, oil that took millions of years to originally deposit. The article lays out how the earth is fracturing around the well, and the author’s belief that there may be a catastrophic release of methane that will cause untold deaths and damage.

Given what’s happening, it seems merely common sense that a sudden release after millions of years will have some repercussions. I mean, we’ve all eaten too many beans and gotten gas – we can all imagine that, times several thousand. So. It can’t be good, right? The article makes a very compelling case.

However, I say the story is fabulous, I mean just that: it is a fabulous story. It’s fabulous as a piece of writing; it’s shitty journalism.

It is written by somebody truly in charge of the structure of article-writing. The author guides you cleanly through from a very catchy headline (“world-killing”) to a chain of reasoning supported by footnotes and “experts” in various fields. It brings in a “lone scientist” who has a groundbreaking theory that might be proven right (literally), it gives a timeline that lends pacing to the article, it quotes from witnesses who claim to have seen similar signs of disaster (earth bulging at the site, by, like, a lot!), it brings in a conspiracy theory (government lockdown, which is in fact true) and it presents other possibly similar situations that may have happened in the past. It walks you through them step by step.

It’s extremely well-written.

But from what I can tell, it’s complete and utter hooey. Mostly. How do I know? Well, I have some experience in journalism, and in science.

But how can YOU know?

We have a sort of “cult of expertise” in our society that is very dangerous, but also very useful. We trust experts – we have to: they’re experts! We must rely on them, because they know things and do work that we just can’t, don’t, or won’t do ourselves. They’re good at the things that they do. Experts? They’re essential. But there’s a real trust there, that experts are honest and professional. That trust can easily be abused by people claiming to be experts but who aren’t. But how do you judge an expert on their own terms? How do you tell what’s horseshit and what’s not, if you don’t know the field yourself?

It gets worse. We rely on journalists to translate expert-speak into ordinary-people-speak for us. There are certain shorthands that we’ve developed that let us know when something is credible. It’s useful, because journalists can act as go-betweens and help experts talk to us, and help us understand the work that experts do. Unfortunately, journalists must be invested with some trust too. And the same problem rears its head: who do you trust to be honest and credible?

For example, the author of this article has hit all of the “hallmarks” of a credible article: he talked to eyewitnesses. There are footnotes to scientific papers. Interviews. Theories! And numbers! Especially numbers.

Unfortunately, there’s a lot in the article that is, well, problematic.

Unless you know what you’re looking for, it’s easy to be swept away by the chain of reasoning. Unless you actually go and read the papers, check out the background of the experts yourself, and have some basic understanding of what’s involved, it’s easy to form an opinion based more on how certain the writer is of the article’s conclusion than how certain the actual scientists quoted within it are.

That is where trust breaks down.

So. My public service for the evening is a brief primer on spotting logic or credibility problems in things that claim to be scientific journalism. Here are some things to keep an eye out for:

1. Past doesn’t always dictate future: Events are complex and multi-causal. For example, in this article, “experts may agree” that methane releases in the past were the cause of extinction events (although linking to a BBC TV series exploring the issue does not actually prove this to be the case – television is not even a tertiary source, credibility-wise), but even so, journal articles about methane releases at different points in time correlated with extinction events prove only that, at some point in the past, there were methane releases correlated with extinction events. But it doesn’t say anything about the circumstances that led to the releases, whether methane was a primary cause (for instance, there may have been very different environmental factors: like during the Cretaceous, there was a 40C oceanic dead-zone around the equator for several hundred thousand years — things were wildly, weirdly different). It might. But it might not. Hard to say. Where are the journal articles actually linking similar events in the past to similar circumstances now? Not there.

This doesn’t mean there might not be a link. It just means that the journalist is telling us a story right now that is purely circumstantial. Keep that in mind.

PROTIP: To judge a theory’s credibility, you need to hear what the actual author of the papers about the past events thinks, not what a reporter thinks based on some other expert’s interpretation of the possible applications of the theory. Conclusions must be based on data, not merely the question “could this happen?” Sure it could! The real question has to be “Given the current data that you’ve seen from this specific situation, what is the probability of what happened in this other situation happening in these circumstances?” You’ll likely get a different answer, unless the causes and effects are pretty well known. In the case of the BP spill, we’re not presented with any sort of history that tells us that this is so. That’s because the situation is unprecedented. We’re in completely foreign territory here.

2. Numbers: Using numbers makes something seem credible, but you have to make sure the number actually matters or is relevant to the problem at hand: for example, the article says that oil is gushing at 40,000 psi. THAT IS A LOT. But wait. That’s relevant to the problem of trying to apply a cap, not to fissures appearing elsewhere in the ocean’s crust. Or is it? Maybe. But the journalist doesn’t talk to any scientists about this event. Reports that they’re happening are eye-witness, but what does that mean? How does seeing a bulge translate into bulges-per-psi? And, now what? What we need is a comparison: the ocean floor can support XX,XXX psi in this area, but not XX,XXX + YYYY psi. Remember, the oil was always down there, at pressure. Now the pressure is venting in a specific area, and that’s the problem. So maybe 40,000 psi causes cracks, maybe not, but you need to actually hear an expert in the field use the numbers to make a point. Just quoting numbers alone? Not at all useful.

Now, we know that cracks are appearing (NOAA says that they are natural, but could be getting larger now as a result of the spill). The author is pointing out a serious problem. But there’s no actual information about the extent of the problem given in hard numbers. Part of that is because there’s a lockdown and BP seems to own the EPA. So again, we have something circumstantial. But it’s still not science yet; it’s speculation. An essential first step, but…

3. Interview subjects & eyewitnesses. Eyewitnesses tell us that something might be happening, but they don’t give us a measurement baseline. They’re like coughing and sneezing: they tell us there’s something wrong with the lungs, but not necessarily what’s causing the illness. But journalists love eyewitnesses. Eyewitnesses are quotable. They provide “human interest.” They tell us a story. But they’re not science.

PROTIP: Beware stories peppered with eyewitnesses claiming things are happening because they’ve seen certain phenomena – smoke doesn’t necessarily equal fire. It often does, of course! Bless our eyewitnesses: they’re often the only way the public ever becomes aware of serious problems. But they’re not definitive. They’re not data. They’re anecdote.

4. A compelling story: I hate it but it’s true: most scientific journalists just aren’t that good at writing. They’re hampered by the fact that real science is very messy and often lacks a through-line; common sense isn’t common — things are often counter-intuitive — and a ripping good yarn is hard to achieve when you actual fact-check and source properly and use the actual quotes from the boring guys instead of the interesting ones that professional PR people create.

PROTIP: If your “holy-shit-this-is-incredible-o-meter” is going off, try Googling some names and theories for yourself… just in case it really is too good (or bad) to be true. There actually are some talented science journalists. Like, three. But they’re the exception.

5. Conflating sources: Note that actual journalistic claims in the article aren’t sourced: “frantic efforts to quell the methane”. By who? How does the reporter know this? Who told him? “Some environmental experts…” “Experts all agree…” The only things he actually cites as facts – like the 40,000psi – aren’t followed by an actual expert tying those facts together. He does that, then moves on to the next “fact” / “scary quote” / paragraph or two of exposition. BAD NEWS.

6. False credibility: And sometimes, experts just aren’t. The article in question made it onto a talk show yesterday, where “another guest expert, R.C. Hoagland, stated…” yadda yadda earth explosions horror tsunamis! But Hoagland, a cursory Internet search reveals, is a crusty old crank known for his contributions towards the moon landing hoax theory and for supporting several interesting doomsday ideas not unlike this one. Even if he wasn’t known as a crank, an interview with a scientist is not the same as a journal article, so “an expert” commenting on a story is not a good source of information per se, unless they’re talking about the scientific debate or interpreting pre-existing data in laymen’s terms. Experts who extrapolate too freely? Beware of those guys: they’re often for hire on a talk-show circuit. They’re not there to inform; they’re there to look interesting on TV. And TV is, well, TV.

But regardless of the medium, interviews by their nature are informational, not scientific (Mark Twain had a lot of zingers about how useful interviews are; many are still apropos); they are not designed for real debate, especially when we’re discussing a “edge theory” that only a few groups of people are really equipped to talk about at all, and then in very highly specialized terms that couldn’t possibly make good TV.

If experts are used in interviews, it’s often simply to lend a veneer of credibility to the TV show. Because it is just that: a show.

PROTIP: An expert is only ever an expert in their specific field. Review their contributions to that field. How many papers have they published? Are they interpreting or blue-skying? Are they genuinely involved in the scientific work and taking an active part in scholarly debate, or are they gadflies? Using Google Scholar and searching for their known publications is a good place to start.

Anyway.

There’s a heck of a lot wrong with this article. I mean, a lot. It’s really well-written, it’s very compelling, and it ties into a deep, completely reasonable fears about the breadth, depth and scope of the BP disaster, the true extent of which is being hidden by the US government at the urging of their corporate masters. But as a piece of journalism is truly hideous. It uses every rhetorical trick in the book to lead the reader to its conclusion, but I’ll bet if anybody contacted Ryskin, the scientist responsible for the original methane theory, he’d have harsh words about his work being used in this way. I suspect.

So, science journalists, you have a job to do: get this story right. It’s very important. It might even be true. Who can tell? As written, it’s worse than useless: it’s fearmongering, anti-scientific garbage.

UPDATE: Found some of the blog sources (and comments from people who seem to know the field) that the article didn’t credit directly. Some worrying stuff, but nothing definitive from the oceanographer or geologist interviewed directly. The methane-explosion-release hypothesis originally came from discussions a few weeks ago of nuking the seabed to stop the leak.

EVEN UPDATEY-ER: A real journalist/scientist has a go at this. And indeed, it’s utter bullshit.