Preparing for Science Online London

In a few hours I'll be part of a Science Online London panel titled:

Linking with the Literature – the Arsenic Story.  How to engage with the peer-reviewed literature: strategies for fellow researchers, science journalists and bloggers. A look at how this was handled in a controversial example from last year – the #arseniclife story.

I 'm trying to figure out what I'll say in my introductory bit.  Our leader has given us four questions to guide our preparations.

The first question is:

1. The #arseniclife story - what deficiencies in engaging with the literature did this case make obvious?

So I think I'll start by considering what researchers might hope to accomplish by writing online about specific peer-reviewed papers.  We can spotlight work we think deserves more attention, and point out problems with work we think flawed.  We might hope to change how the published results affect future work, or what the public thinks science has discovered.

Then I'll do a very quick reminder of the #arseniclife story, emphasizing not the science but the responses of the media, the public, and other researchers. 

I should end my introductory bit by considering the 'deficiencies' of the online responses.  This requires hindsight - what do we wish the online responses had accomplished?  Since everyone but the paper's authors agrees that the big claimed result is false, the responses should have counteracted the media blitz (corrected public understanding), and discouraged researchers from following up on the result.  I think they only partly succeeded at both of these.  Only a very small fraction of the original audience will have seen the criticisms and taken them seriously.  And lots of scientists still refuse to think about material that isn't peer-reviewed.  Is this due to deficiencies on the online responses???

The next two questions will be easier for me to answer:

2. (briefly) Is this only an issue in the life sciences, what are the experiences in the physical sciences?

I think I'm off the hook for this one.

3. Is the peer-reviewed system employed by journals too slow and/or not open enough to deal with such cases? (this might be particularly the case for retractions, which e.g. often are not motivated by editors).  Similarly, what are the online alternatives that have been developed?

Too slow to accomplish what?  It's  not too slow to sort out bad science from good, but it might be too slow to save researchers from wasting their time building on the faulty results.  It's definitely far too slow to counteract the kind of media blitz that occurred with #arseniclife.

The final question is much too big for my limited brain.

4. Possible scenarios for the future. What are the implications and consequences for the various stakeholders (scientists, publishers, journalists/bloggers)?

Can I come up with even one consequence of online engagement with the peer-reviewed literature?  Well, if you're a scientist hoping to incorporate an exciting new paper's results into your planned research you should begin by checking the online response to it.  If you're a publisher (or Google Scholar), providing an easy way to track the online responses to articles would add a lot of value to papers (supplement the 'Cited by' list of links with a 'Discussed by' list).  If you're a blogger, make your posts easy to find.