Brent Nyitray
Fairlington Blade
Let the mutherfuker burn!
jnc4p
Kevin S. Willis
Michigoose
lmsinca
markinaustin
Arcanaut
novahockey
quarterback
ScottC
yellojktThere is a very interesting article today (unfortunately behind a firewall) in the WSJ regarding the difficulty pharmaceutical companies are having reproducing results from studies published by academics in “peer reviewed” journals.
This is one of medicine’s dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can’t be reproduced.
“It’s a very serious and disturbing issue because it obviously misleads people” who implicitly trust findings published in a respected peer-reviewed journal, says Bruce Alberts, editor of Science. On Friday, the U.S. journal is devoting a large chunk of its Dec. 2 issue to the problem of scientific replication.
Reproducibility is the foundation of all modern research, the standard by which scientific claims are evaluated. In the U.S. alone, biomedical research is a $100-billion-year enterprise. So when published medical findings can’t be validated by others, there are major consequences.
Although focused on biomedical/pharmaceutical research, I wonder how much of this problem exists in other areas, particualrly in light of this (my emphasis):
There is also a more insidious and pervasive problem: a preference for positive results.
Unlike pharmaceutical companies, academic researchers rarely conduct experiments in a “blinded” manner. This makes it easier to cherry-pick statistical findings that support a positive result. In the quest for jobs and funding, especially in an era of economic malaise, the growing army of scientists need more successful experiments to their name, not failed ones. An explosion of scientific and academic journals has added to the pressure.
Also, there was this:
According to a report published by the U.K.’s Royal Society, there were 7.1 million researchers working globally across all scientific fields—academic and corporate—in 2007, a 25% increase from five years earlier.
“Among the more obvious yet unquantifiable reasons, there is immense competition among laboratories and a pressure to publish,” wrote Dr. Asadullah and others from Bayer, in their September paper. “There is also a bias toward publishing positive results, as it is easier to get positive results accepted in good journals.”
Science publications are under pressure, too. The number of research journals has jumped 23% between 2001 and 2010, according to Elsevier, which has analyzed the data. Their proliferation has ratcheted up competitive pressure on even elite journals, which can generate buzz by publishing splashy papers, typically containing positive findings, to meet the demands of a 24-hour news cycle.
Dr. Alberts of Science acknowledges that journals increasingly have to strike a balance between publishing studies “with broad appeal,” while making sure they aren’t hyped.
I’m guessing that balance is not always well struck, nor is the problem limited to biomedical science. I’m sure our local scientists can weigh in on this.
Filed under: Uncategorized |
All Things in Moderation 
Scott beat me to it. This was making the rounds in the health policy trade press this morning. The first thing that came to my mind is the push for comparative effectiveness research. Docs already don't want too much direction from central command. I'd imagine that this helps their cause.
LikeLike
Interesting. In Pharma, once your past Phase II trials, you only want to see good data, as that drug is gonna be marketed. Prior to and up to Phase II, Pharma wants absolutely honest data to determine if they should pull the plug.
LikeLike
Anybody done peer reviewing of a scientific study? I'd be interested in knowing what gets reviewed and what that process is like. Are there published and used standards that the experiment is compared against? Are there consistently certain things that reviewers look for? How do reviewers know if what is presented is on the up and up? Or is it a 'looks good to me' cursory review? I am guessing that all these answers are based on the particular publication that the study is sent to.The big problem as I see it is that after a study is reviewed with the thumbs up, it's conclusion and data are used as as gospel in other studies.
LikeLike
Dave:A while ago we had a long discussion about how political the process is. Mich and Mike gav eus their insights, as they have both been involved. I'll try to find the thread and link to it for you.
LikeLike
Dave:The thread is here. The larger thread was about Climategate I, but it delved into the peer review process with Mich's post at 8:30.
LikeLike
As much as I'd like to give this post serious consideration, a drive-by is the best I can do for now.Bruce Alberts is a very smart man and excellent scientist. Science (the journal that he edits) has been stung a couple of times recently with high profile papers that had questionable data in them, so Alberts has issued "editorial expressions of concern." One of the papers was eventually retracted, I think.A couple other things. In my experience, it is much easier to get a manuscript published if it describes positive results. Negative results papers either don't get accepted or get buried somewhere. Which leads to Dave!'s point — if it is out there, then it isn't necessarily gospel, but it is accepted as the current theory so you have to go through significant hoops to disprove it.Another thing is that there are a variety of categories of scientist, ranging from the unimpeachable to the sloppy to the frauds. Sloppy is sometime worse than the frauds because they end up leading the whole field down the wrong path. The frauds usually are ignored, but sometimes you can't really tell until it is too late.Peer review is difficult. You can only review what you are given, so you have no idea if the experiments were done as described, if the data are real or faked or Photoshop-enhanced. You do the best you can and make sure that the conclusions are justified by the data. Even if you don't think the paper should be published, sometimes the editor will overrule you, especially if the other reviewer(s) think the paper should be published. Grant reviews are even more time-consuming and tedious. It takes me at least a couple of days to review one grant properly because, while I'm an expert in the field as a whole, I don't know the nitty-gritty about each virus system, etc. The other problem is that funding levels are so low that you know you only get the chance to advocate for one application out of your stack of 10 to 12. It's hard choosing 1 among 2 – 3 outstanding grants.We do the best we can and the vast majority of us are not trying to make stuff up. But we can see what the competition for funding does — look at the CERN neutrino guys. They are "publishing" by bypassing peer review and going straight to the newspapers. I probably won't have time to add to this anytime soon — I'm out for the next week. But we can revisit it later if anyone wants.
LikeLike
My two cents' worth on the fly: not having read the article, I don't know what the author means by academic researchers not doing "blinded" studies. It all depends on the experimental design and what you're researching. His statement is too broad to be interpreted.As far as peer-reviewing, Dave!, it's definitely not a cursory process. There are standard controls, both positive and negative, which are expected to be reported along with experimental results, and new studies are judged against previously published ones. Which is why it's such a big deal if something can't be replicated.I've been on both ends of the stick–not being able to replicate somebody else's results, and not having another lab able to replicate mine. It's not an unknown occurrence, and can usually be reconciled with appropriate troubleshooting.it is easier to get positive results accepted in good journals.Unfortunately very true, even though some of the best stories are from negative results.
LikeLike
I'm working on a longer post about the publishing process, but have a few thoughts. 1 – On reproducibility. I was working with a colleague this week who makes quantum dots. If you reduce the dimensions of a semiconductor to the point where quantum effects operate, you get interesting changes. He was trying to make some quantum dots that emitted infrared light. He followed the published procedure of several papers. I was measuring luminescence from the dots and it turned out at about half the wavelength that we expected. Some corroborating measurements (dynamics light scattering) showed that there were two types of dots. The smaller ones were responsible for the PL that we saw. He was a bit frustrated.I have a little experience in these things. People often don't publish all the details. Some time ago, we were trying to make some organic solar cells. The process involves depositing multiple layers of material–one of the most important was buckyballs. One needs to have a thin layer between the buckyballs and the electrode or the device won't work. We used the standard material and followed the standard procedure. What we didn't know (because no one published it) is that the material used for this thin layer easily crystallizes and if it does, you're screwed. I accidentally solved the problem when I was short on time one day and didn't have time to deposit all the layers. We normally deposited all the organics on Day 1 and then the metals on Day 2. For this run of devices, I didn't have time to put down all the organics, including that thin layer, in one day so I held off to the next day. If you put the metals down immediately after the thin layer, it's blocked from crystallizing and the devices are stable for months. It would have been nice if someone had published that little tidbit!2 – On blind studies. For most scientific work, that is quite frankly unworkable. Double blind studies are hellishly expensive. That's a big factor in the cost of bringing a drug to market. If you want blind studies, you'll need to double or triply research funding. Which leads me to…3 – How can you believe what you read? More important to me, how can I believe what I read? There are many reports of efficient organic solar cells that are complete BS. Unless it's from a group I trust (none of which are in China), I don't believe an efficiency measurement unless it's been certified. Ultimately, it comes down to integrity. You have it or you don't. I have gone down the path of an exciting measurement. The most dangerous ones are something that looks like what you're trying to find. I've had a couple of cases that turned out to be from an experimental artifact. In one case, an optical filter allowed through a little infrared light at a wavelength where I expected to see phosphorescence. In another case, the decay time of a signal looked like what we wanted. Turns out, the temporal response of our preamp just happened to mimic the signal we wanted to see. I didn't publish either of those results as I agree with Ron Reagan. Trust, but verify. I trust my integrity and the results that I see, but work to verify exciting results. Nothing is more stomach churning for me than running the confirmation measurement. Or just checking one last time. I've got a great result in the can right now that I think could make it into Nature (probably Nature Materials, but I'd live to hit the big time). We double-checked the measurement and I dreaded going into the laboratory. I'm pleased to report that the result was confirmed.BB
LikeLike
Mike, Michi, and BB, thanks for the insights.
LikeLike