In this post I'm going to elaborate on a suggestion I saw a few days ago, in an article discussing the role of post-publication commentary in science. (And yes, I'm searching for the source of this idea - if any reader remembers whose it was, please point me to it.)
In modern but pre-internet days, researchers did the research, wrote the paper, submitted it to peer review, made changes, and published it. Other researchers them evaluated this information, using to guide their own work, and discussed its strengths and weaknesses when they cited it in their own papers.
Published papers were also discussed less formally with colleagues, both before and after publication, face-to-face and by mail and phone, and in journal-club presentations and seminars. The ideas from these discussions were incorporated into the formal papers drawing on this work, but they weren't available to anyone but the direct participants.
Now that we're all on line, published papers are also being discussed more publicly, in blogs and other places. Such discussions are extraordinarily valuable for the progress of science - they're written public evaluations, drawn from a wide range of expertise, and usually greatly enriched by comments from and links other researchers. But these pages are all over the place, and finding them requires a lot of active searching.
The Research Blogging site is helping with this problem, by aggregating blog posts that discuss individual research papers. But they can only link to posts that actively insert their code, and so miss quite a lot of the public commentary. So far the journals don't link their papers to this site, so readers who go looking for the paper don't usually think to also check Research Blogging.
A few forward-thinking online journals (PLoS and BMC groups, I'm talking about you) provide their own Comments thread for each paper, so other researchers can provide informal but public feedback . But the researchers don't use these, saying that they don't feel comfortable doing this publicly, or that they don't like the bother of having to register and log on. I know that's true for me, thought I don't know why - I'll happily blog about a paper I've read, but I almost never post comments on its official Comments page.
Sites like The Third Reviewer have tried to solve this by providing journal-independent sites where researchers can post comments about papers. But we won't use these either -the massive wave of discussion about the Wolfe-Simon paper on arsenic bacteria let to exactly zero comments on The Third Reviewer.
Most journals already provide, with each paper they've published, a list of links to the more recent papers that cite it. The suggestion I really liked was that the journals should also aggregate the informal commentary, by providing a separate list of links to ALL the web pages that have link to the paper. Journals could then stop fighting our unwillingness to post comments centrally, and just use our distributed posts to add value to the papers they publish.
I don't think this would be very difficult; lots of sites already have a 'Who links here?' feature, and I think both Google and Yahoo searches can be restricted to sites containing a specific url. The journals could include an explicit disclaimer that the journal in no way vouches for the value or creditability of the information in these links. A few bloggers might have to become a bit more circumspect (probably not a bad thing), and any blogger who didn't want their post linked to the paper could provide a citation to the paper but not link to it.
Jon Eisen, do you think this could work? If PLoS leads, I bet the others will follow.
Field of Science
-
-
From Valley Forge to the Lab: Parallels between Washington's Maneuvers and Drug Development4 weeks ago in The Curious Wavefunction
-
Political pollsters are pretending they know what's happening. They don't.4 weeks ago in Genomics, Medicine, and Pseudoscience
-
-
Course Corrections5 months ago in Angry by Choice
17 comments:
Markup Key:
- <b>bold</b> = bold
- <i>italic</i> = italic
- <a href="http://www.fieldofscience.com/">FoS</a> = FoS
Subscribe to:
Post Comments (Atom)
PLoS does this to some degree. Under their "Related content" and "Metrics" tabs, there is a "Related Blog Posts" feature that allows you to search for posts about an article quickly. Much more could be done in this regard, though.
ReplyDeleteThere will also be a huge unevenness in the value of such a service. For a few papers, this will be amazingly valuable, but most papers don't get blogged about.
I'm astonished that The Third Reviewer saw no comments about the arsenic bacteria paper.
One commentary calling for something like 'open peer review' via - educated - blogs was this one in the Guardian. Actually the idea of putting unrefereed papers out in the open, soliciting responses and then judging whether they deserve permanent publication in a prestigious place is pretty old: I first heard that suggestion at an astronomy conference in the mid-1990s, I think. But didn't one famous journal try open review a few years ago - and failed because few fellow scientists cared to even look at the papers. Perhaps open post-publication review worked so well in the 'arsenic' case only because the paper had been published and hyped already?
ReplyDelete@Zen: OK, i've been poking around on PLoS Biology, looking for this feature. I found their description of it (by clicking on the tiny "i" beside "Metrics"), and I can see where the link should be, but none of the papers I looked at had a "Related Blog Posts" heading or links in the side panels. That's the case even for papers that I had first found a blog post about, with a direct link to the PLoS Biology page.
ReplyDeleteThe problem might be that this feature only finds links that have already been aggregated by Bloglines (don't know what that is, and I can't easily tell from their site), Postgenomic (now defunct) or Nature Blogs (which, astonishingly, has nothing on arsenic).
@Daniel Fischer: Martin Robbins' Guardian article was good but it didn't make the suggestion I'm looking for.
Oops, I'm wrong about no 'arsenic' on Nature Blogs. But their Search function must not find content, just blog titles.
ReplyDeletePeter Binfield of PLOS has described gathering the comments made on blogs - as Zen said. It seems more attractive to have the distributed commentary rather than a central site, but there is the need to aggregate in some useful way.
ReplyDeleteThis also goes with the recent discussion of altmetrics - alternatives to the impact factor including mentions on blogs and twitter.
I think it's a great idea. I much prefer to post something on my blog than contribute to some journal's limited ecosystem which makes you feel "locked-in" to their systems, formatting (many are plain-text only), moderation, quality of argument etc. Plus the feeling that no-one is actually listening. At least with my blog I know my Twitter followers and RSS subscribers will see what I write.
ReplyDeleteA point I'd like to raise is that even a paper like this which demands by its extraordinary claims serious attention actually attracted a fairly small number of robust reviews from the community, certainly no more than 10. Plenty of people covered the issue, but only a few people had sufficient knowledge to advance the debate (such as yourself).
On this basis how do other papers which are less sexy, less "impactful", with consequently much less PR expect to generate significant conversation and debate in the blogosphere? The Third Reviewer is a great idea but is let down by the fact no-one uses it. I suspect only a tiny percentage of the literature is covered by ResearchBlogging, and that a tiny percentage of papers get covered multiple times (does anyone care to do these calculations?).
I suspect people feel they just don't have time for any of this engagement, such is the pressure to publish their own research. Certainly many of the PIs at work are scornful of blogging as a concept. We need to reward the peer review process much better in terms of career progression.
Life science is nowhere close to adopting a preprint culture, and I think this is detrimental to the field as a whole.
ReplyDeleteOpen discussion of this paper on various blogs seems to have generated far more useful criticism than peer review did, but it also left a trail of people without any knowledge of the field spouting off in comment sections about the political implications of the paper.
Aggregating post-publication discussion would certainly help, but wouldn't we wind up with better papers if these concerns could be hashed out before what is ostensibly the "final" version is published?
Is this just a case of doing-things-the-way-we-always-have, or are there legitimate reasons (issues with labs in competition?) why we can't do this the way physics does?
Funny, I just made this suggestion to the editors at Astrobiology last week. That journal is so interdisciplinary, I think 2 reviewers is not really enough for a lot of the papers, so a community discussion would help, especially since a lot of readers often come from different backgrounds and may merely want to ask for clarifications for areas not in their background. I actually would like it if the journals would have a place to comment on/discuss a paper on their website, and restrict it to commenters who are affiliated with institutions with a subscription (to keep out crackpots). This would make it convenient for me to just go look up the paper to find discussion on it, rather than having to search for random blogs. The idea of the journals linking to other people's blogs is helpful, too. I don't know who all keeps their own blog.
ReplyDeletehmmm ... thought I posted a comment yesterday, I'll do it again
ReplyDeleteI don't have experience in the nuts-and-bolts of how it would work, but seems like there could be a set-up where if a blogger includes the DOI of the paper they are commenting on in their blog post it would be collected/aggregated somewhere.
I suppose it's similar to researchblogging dot org, but if there was a way to make such that people didn't have to actively insert code that might make it more used?
Also, even if only a tiny fraction of papers are commented on like this at first, that will allow the system to improve over time through trial/error and user feedback.
But the key, as many people already mention, is to somehow collect/aggregate the commentary so the authors can keep track of it. If it's your own paper, think about how difficult it would be to make sure you see every comment out there. An aggregation would show the questions/issues that are priority and then you can respond.
Seems to me it wouldn't be a great technical challenge for Google to cross their blog search with Google Scholar... and then to add an option to hits in Scholar to see all citing blog posts.
ReplyDeleteGoogle, I'm sure you'll have indexed this comment almost as soon as I've written it, but is there any chance one of you might actually read it?
Once upon a time, there was a blog feature called trackbacks that would have been perfect for this purpose. For those who don't remember, trackbacks were a way to automate links to blog posts that comment on something you've written. I would put a trackback ping on a post on my blog to something on rreasearch, and rresearch would add the link to the bottom of the post, the way that some journals add links to papers that cite this one.
ReplyDeleteAlas, like many of the nice things on the internet, trackback was destroyed by spammers. One way to deal with spam is to raise the cost of linking about zero. But that discourages linking, which is bad.
My modest proposal is that professional society memberships should be used to create whitelists that allow members to submit links to commentary about journal articles via something like trackbacks. Unfortunately, when I raised this idea with ASM Press a couple of years ago, they were not receptive.
I really like Ben's suggestion, and I'm going to send it to Google Scholar.
ReplyDeleteGoogle scholar should be a place to find data supported research in journal articles, not unfounded opinions in blogs. There are plenty of other search engines to use for that, you can already find blogs on whatever topic with regular Google. Do you really want to do away with research journals? Thats just crazy.
ReplyDeleteIt is obvious that any kind of scientific writing is quite sweaty and tedious process which requires previous experience, strong determination, great patience and reliable support.
ReplyDeleteInteresting suggestion. 3R has been suffering from a lack of infrastructure to automate the posting of all bio (or other) papers to provide a comments forum. We are in the process of collaborating with another group and hope to be able to implement something very similar to your suggestions here, using pubmed IDs or DOIs to trawl the internet for related commentary on blogs, as well as to mirror any comments made on journals' own sites. We're not there yet so the site is in dormancy, but you can bet I'll be letting you know if we achieve this! Agreed it's what we all want.
ReplyDeleteHi,
ReplyDeleteI am writing from India. I have been seeing many problems with the peer review process, even with papers published by decent journals. I've tried writing to the editors (especially when these journals are COPE members) but the editors just don't care to respond to anything. I can't really risk going public with my complaints because I've got people who depend on me to keep life going. I was wondering a while ago if I should start a blog dscussing such papers, including the problematic figures, the misinterpretations, the poor experimental design etc. I have never embedded pictures/figures into my blogs so far so I will have to learn how to do these things. But would you happen to know if journals can take legal action against people who reproduce their published papers in part or in their entirety? Personally, I think people who don't care about what they put their names to have no right to take legal action about someone pointing this out, but still I don't have the heart to put the people who depend on me through so much trouble. Would you have any suggestions regarding how to go about doing this (this being forcing journal editors to do their job and acknowledge criticism, making peer reviewers care about what they approve without only paying attention to their friendships with authors, knowing whether blogging would be legally prosecutable, having any insight into how to set such a blog up...?)
Always been a great way.
ReplyDelete