Breaking Out of the Science Ghetto
Dean
Reader Robert Bell send along this this excellent piece on the changing world of peer review in science. It's well worth reading.
As someone with an avid interest in science--real science, not most of the junk that appears in popular news accounts--I've often been frustrated beyond belief by how hard it is to get at information in peer-reviewed papers... and how often important work that I happen to know about languishes in obscure journals that are hard to reference.
To give a very simple point, one that may seem obscure but actually matters: a widespread fallacy about low-carbohydrate diets is that they tend to reduce brain function because the brain can only use glucose as fuel. This belief is false; in the 1970s one researcher demonstrated definitively and irrefutably that some brain tissue actually utilizes ketones preferentially as a fuel source. Other studies have demonstrated that during the first few days of a very-low-carb diet there is very moderate impairment of brain function, but after that the problem goes away completely and some people actually show cognitive improvement.
Years ago I found the actual peer-reviewed papers to demonstrate all this. It's all on a hard drive for a Mac that broke down on me years ago, and it would take me quite some time to put it all back together. What I should be able to do is hunt through an online archive and simply provide hyperlinks to them so anyone who wanted could go read them, and anyone who wanted to refute them would at least have them in front of them so they'd know what they were refuting.
As The Economist notes, the vast majority of scientific papers wind up copyrighted by the publisher of the peer-reviewed journal--not the authors, not the university that funded the study, but the journals! These journals generally expectd you to pay to read their stuff, often quite exhorbitant amounts. And they frequently don't bother making their older archives available, under the assumption that any research more than a few years old isn't of interest. Yet while that's probably often true, quite often a great paper that's decades old still has much of value--and researchers today might even sometimes find themselves unknowingly re-inventing the wheel as something some scientist they never heard of already having done much of their work for them!
Although research institutions tend to be quite conservative about these things, I think it would be a great thing if more researchers would be open to experimenting with blogging technology to release their work straight onto the internet. This would be a big risk in some areas, because people would have a hard time telling crackpots from real researchers. But here's one way it could be done:
Let's say you're a molecular biologist doing some innovative work on cancer. What you'd normally do is run some experiments, write up a paper, and submit it to a peer-reviewed journal. The scientific editors of that journal would look the paper and, if they thought it had value, would forward copies of the paper to anywhere from a handful to a fairly large population of other scientists with backgrounds in the same field. Then based on their comments, the scientific editor would either reject the paper completely, send it back to you for revision, or go ahead and publish it.
What would expoloiting blog technology do to change that?
Well first off you don't want just any idiot leaving comments and trackbacks. But what would be wrong with this?
Take a preliminary copy of your paper. Mark it "DRAFT DRAFT DRAFT" prominently in several spots, then publish it on a blog for public comment.
Now, you do not simply allow anyone in the general public to comment. But what you do is you (or an assistant) allows REGISTERED USERS to comment. Who is allowed to be a registered user? Any fully credentialed scientist (or if you're a bit more liberal, approved grad students or people with degrees in related fields). Those people get comment accounts, IF THEY WANT ONE, and may comment to their heart's content.
Then later, publish a revised paper on the blog. Then finally, publish a final draft.
This sort of thing would allow for extraordinary transparency in research. Yes, researchers would have to lose a little ego, but think of the amazing synergy it could produce. You'd suddenly hear from researchers on the other side of the world you'd never heard from before. Some grad student in math in another state might chime in and show you where you got a calculation wrong. Somebody with a PhD in a separate field might out of nowhere show you a magical reference you'd never known about that changes everything.
No more waiting weeks or months to publish. No more rejections for obscure or even political reasons.
Why not?
Consider this especially: why shouldn't most research that's paid for by government grant be more freely available this way?
Blogging has so much potential for researchers. I wish more of them would think about using it.
You would have to have the journals agree that publication on the Internet won't preclude publication in the journal, for one thing.
The exhorbitance of the access rights is ridiculous. I've seen prices as high as $24 for a PDF of an article in a journal that the library doesn't have a subscription to.
I actually prefer a central site as opposed to blogs for doing what you suggest so you don't have to be on top of publications from unknown authors. Although a RSSS aggregator could also be used (just submit the blog feed).
There are some economic implications as well - getting something published in a peer reviewed journal also means one can put that on your resume and that *does* matter for your career. The readers of peer reviewed journals count on the journals to do a filtering process to save them time.
That filtering and referreeing function also seems to be potentially better done by technology as well. Not that I know from structured blogging, but essentially all you need is a template for a peer reviewed article, and some sort of aggregator. The reviewers subscribe via RSS feeds and post their comments electronically. You can keep a version history and see the evolution of ideas etc.
There is also a substantive issue regarding disclosure of material non-public information in medical research. Apparently the annual meeting of the American Society of Clinical Oncologists is a zoo of security analysts looking for the next hot biotech pick. However, disclosure can be done better electronically by a company to the stock exchange directly. It is therefore guaranteed to be accessible at the same time by all investors (assuming no insider leaking and shenanigans). The National Stock Exchange of India has this capability already http://www.nse-india.com/ - I can't link it directly - look at the "Us" menu, click on "Milestones", then look at the launch in 2004 of an electronic disclosure system.
Back to the economics implications for authors though, presumably a paper's influence can be measured by the number of citations, so in fact you can do a *better* job of sorting out the merits of someone's academic work because you can compute the number of links. (Subject to manipulation I know but ...) The distribution of links will likely follow the power law described in Albert Laszlo-Barabasi's "Linked", so if ego stroking is important, this will only enhance it because the distribution of links is highly skewed.
Finally, a sad story of the realities of taking on the publishers here.
The comment on exhorbitant prices is spot on. I have gone out of my way, on occassion, to find a library with a journal rather than to pay for the PDF download. It took more time, but it was cheaper. Yet, I can't help but wonder what the cost of an issue of Popular Mechanics would be if they, too, shunned most commercial advertising.
However, I disagree with the comment about assumed interest, as it seems to ignore the primary reason that older articles aren't available online. It's easy to print a current file to PDF and add it to an archive. It takes more time to convert a physically typset proof to PDF, especially if no digital representation of the article exists.
I have watched over the last 5 years as journal article archives have slowly expanded to include 1999, then 1998, then 1997, then 1996, then.... Well, you get the picture. I have even found a few archives that go back to the 1980s - granted, very few go back that far yet.
In some cases, the PDF version of the article is clearly a scanned image of the original printing, i.e. a GIF or JPEG and not text. That process takes time and personnel. And that takes money, which they attempt to get by charging me $24 for an 18 page PDF file.
Certianly, not every publisher is doing this. However, many (most, in my experience) of them are. But it takes time.
It also often takes a university to get to the arhcives. I get access through the university libraries to all sorts of archives that I would not be able to access otherwise. As a student, I can even access most of those resources from off campus by logging in through my university ID.
If you're not a student, however, you'd have to actually go to the library and use the in-house computers. This would give you access to the same archives of articles but, unfortunately, you might not be able to take them with you. For a variety of reasons, floppy drives and empty USB ports for flash drives are not available on the library computers. But that leads into a couple of different rants for another day.
Dean also wrote:
Perhaps not in the way you suggest, but certainly in some way. I have never completely agreed with explanations of why reports of research funded by Federal money are not somehow covered by freedom of information rules. Even the reports to the funding agency are often tightly controlled.
Now, you do not simply allow anyone in the general public to comment. But what you do is you (or an assistant) allows REGISTERED USERS to comment. Who is allowed to be a registered user? Any fully credentialed scientist (or if you're a bit more liberal, approved grad students or people with degrees in related fields). Those people get comment accounts, IF THEY WANT ONE, and may comment to their heart's content.
You get a grad student or lab assistant whose job is to weed through applications. YOU as the person in charge decide who you'll let in and what credentials you're willing to accept. And if someone turns out to be an idiot, you suspend their account.
This isn't hard man.
1. As caltechgirl asks, what about the cranks? I'm not convinced that a registration process weeds out the cranks, the idiots, and the agenda driven. When a journal editor selects reviewers, he is applying his editorial judgment to select qualified, objective reviewers. In theory, the prestige of the journal rests in part on his skill in selecting reviewers. Self-selected reviewers could easily be a lot lower quality.
2. Reviewer registration raises a problem of its own. Somebody is going to have to go through all of the applications and verify the credentials of each one. This isn't insurmountable by any means, but it's a workload you'll have to budget for,
3. The would-be reviewers who get rejected will raise a stink and claim that "dissenting views aren't tolerated." Charges of bias and favoritism will be highly likely.
4. What about cranks of the other sort? What about authors whose works should rightly never see the light of day? While I'm sure there are journals that quash deserving papers, I'm far more sure that journals provide a useful filtering service.
5. How do reviewers find research blogs that deserve their attention? Right now, journals and journal editors serve as a sort of "gathering point", bringing authors and reviewers together.
6. How do interested parties learn of the research that has been reviewed and released? Again, a journal is a known point.
I think all of these concerns can be addressed; but as you resolve them, I think that your solution will come to look more and more like, yes, a journal (with perhaps some echoes of a scientific conference as well).
The journal system has flaws. I think you've only begun to identify them. And the lack of accessibility and transparency is going to be more and more of a problem over time. But journals also serve many useful purposes: mediation, filtering, connection, publication, and dissemination. I believe that blogs can serve all of those roles as well as journals do, and often better; but I think that the underlying journal structure has a lot to learn from.
And oh goodness gracious golly me, what happens if a crank gets through and posts an idiotic comment? However will you recover? You might have to... to.... delete the comment and delete the account? MY GOD MAN, HOW COULD SCIENCE SURVIVE THAT?!?!?!?!?!
Sorry guys, but seriously, get over it. %-)
As for the problem of people finding the research blog--an entirely separate question. There are several strategies that can be persued. Two that spring instantly to mind are (1) having your university or research facility support and promote it, and (2) having the existing journals dedicated to these subjects set up the blogs for researchers within their field.
All these objections are based on top-down rather than bottom-up thinking. It is the exact same paradigm as closed-source vs. open-source software. Won't open-source software be more buggy? Nope. Won't open-source software be more virus-ridden? Nope. Won't open-source software be less polished? Yep, in some ways. And this is a tragedy, or merely proof that nothing is perfect?
There's no more need to abolish peer-reviewed journals than there is a need to abolish traditional journalism. That's not even the point.
The cost of journals isn't a problem for scientists. They have taxpayers to pay for them!
I'm sure your discovery that thanks to technology, taxpater-funded conferences at resort hotels are unnecessary will be welcomed with joy by researchers everywhere!
More to the point, I don't WANT to comment on most articles I come across, and I suspect that few other researchers do. It takes too long just to read them.
So you've got researcher apathy up against journals who will fight this by not accepting articles which have appeared on blogs. I see the advantages of this idea, but I'm not sure it's practical.
Seems to me they allow just about any comments and I for one find the criticism much more enlightening than the original paper. I love perusing the BMJ precisely because of the commentary.
Several on-line journals that I've read allow for commentary. Even the print journals post letters to the editor (although they quite frequently reject dissenting ones.)
I fail to see why everyone seems to be rejecting this idea, although I don't see it improving research much. The vast majority of the research I read lately is just fancy data massage and speculation.
Personally, I think researchers should be banned from publishing their conclusions. Rarely are they supported by the body of the paper. Perhaps they should be independently determined by committee?
The researchers are going to imply to the committee what the conclusions are, and the committe will say "sure, what the heck". In other words, it will be just like peer review today.
The bigger issue is not comment, but access. I agree with you that research is hard to find unless you have a magic key (university ID number)
In the vast realm of things grad students are asked to do, this ranks considerably below "grade these papers" as an onerous burden.
Even the PH.D. Granting process doesn't weed out all those. ;-D
dean, grad students don't get paid to sort out cranks. They get paid (precious little) to do research. Most labs can't afford supplies or personnel to do research, and yet you would have them hire someone to run a web page?
that's not nearly always the case. I know several grad students who are paid to do the "grunt work" of a peer-reviewed journal (i.e., send out review copies, sort submissions, type responses, edit spelling and grammar, respond to e-mails, etc.). Not to mention grad students who are paid to proctor huge introductory classes, grade tests, and generally file papers for professors.
If you try to investigate some astrophysical phenomena, you quickly end up in fully-online and free astrophysical journals. Where every reference is a live hotlink, and printing is DIY.
In chemical engineering, where applying the research is somewhat more immediate, there's more emphasis on using something like Chem Abstracts or Sci Finder Scholar... $10k for a site license, last I checked.