Editing and Peer Review

During the six years that I was the editor of the Journal of the Medical Library Association it was rare in the extreme that we published an article exactly as submitted.  When I started, I suppose I thought of peer review as strictly a search for error, an opinion as to whether or not the piece should be published.  I came to understand that the reality is far more subtle and complex.

Marcus suggests that librarian journals should evolve into blogs, arguing, in part, that peer review should be a post-publication process rather than a "pre-publication process that sometimes drags out for many months."   Although there is something appealing about this idea, when I think about the actual articles that I was involved in editing, I'm not at all sure that this would be a good thing.

As editor, I considered my responsibilities to be not just to select the most appropriate work for each issue of the JMLA, but to work with each author to present their work and ideas as successfully as possible.  And I began to see the peer review process as just the first step.  Often (perhaps usually) the reviewers (we always had three reviews) disagreed among themselves as to what the major issues or necessary revisions were.  Sometimes they flat out contradicted each other, and part of my role was to sort through those differences and give the author the best advice I could.  Frequently, the writing was simply poor and there were many cases in which it became clear to me as I worked my way through the article that the paragraph that the author had written actually said nearly the opposite of what they intended to say.  Or the article would be full of extraneous and repetitive material that simply got in the way of a busy reader actually getting to the meat of the work.  I believe that most of the articles that we published were more effective in connecting with an audience than they would have been without the pre-publication review and editing.

I'm not at all sure that it would be a service to the library community if all of those articles that I read through in their first iterations had simply been posted to a blog and opened up for comment.  The few experiments that have been done in the last couple of years with post-publication review have not been overwhelmingly successful, the ArXiv experience notwithstanding (extrapolating from the experience of a small, tightly knit, fairly homogeneous scientific community that writes largely in formulas to a much more diverse, narrative literature is a stretch, I think).

Certainly, the lag between completion of an article and its availability to a wide audience needs to be shortened, but this is a matter of efficiency, not a fundamental aspect of "traditional" journal publishing.  The major scientific journals have typically reduced this time lag to a few weeks and there's no structural reason that can't be done with the library literature -- it's a matter of the resources that you're willing to devote to tackling the problem.  Marcus suggests that this could be accomplished by "a carefully designed and managed blog."  The devil is in the details, and I wonder if that careful design and management would result in something that looks more like a typical journal or more like a typical blog. 

I'm not one who is terribly impressed by the "wisdom of crowds" (a concept that seems to be especially dubious during the US election season).   I've rarely seen anything approaching substantive discussion and analysis take place in a comment thread, and the longer the thread, the more worthless it typically is.   Rather than providing vibrant post-publication review, I'm afraid that posting unedited articles for comment would result in much good work being buried and ignored.

But the terrain continues to evolve rapidly, and the opposition of blogs to traditional journals is probably a false distinction.  The traditional journal is rapidly morphing into something else, while adopting features that we associate with blogs (the ability to provide rapid responses being the most obvious).    The underlying architecture of blog software is also developing rapidly and trying to accurately characterize the qualities that define a blog would likely become a contentious debate.   

Marcus is pushing the right questions, and everyone involved in scholarly publishing, at whatever level, should be thinking creatively about how to make the communication and discussion of projects and ideas more effective.  But it isn't a matter of journals vs blogs.  The most effective modes of communication that we develop over the next decade will adopt features that we associate with each, but will be fundamentally different from either.

Comments

[Karin Dalziel]

I think of blogging partially as idea fodder for articles- I put my ideas out there and the community can tell me if I am way off base. If I am, I'll change my direction. If I'm not, there might be something there for an article. My blog posts are rarely in depth enough to serve as articles by themselves.

You're right that a carefully managed peer reviewed blog is pretty much an open access journal. I've thought a lot about what makes a blog a blog, and I've come to the conclusion that it's not just the software. (Comment Press by Institute for the Future of the Book would be an example of blogging software that has morphed into something else.)

Most Journal management software is pretty bad, but it wouldn't be hard to instead use drupal or wordpress to put together a journal where the peer review and publishing all happen in the same place. Streamlining the process would go a long way to getting articles out more quickly.

The problem with blog as peer review is that the comments don't usually affect the content of the post- a system would have to allow the original to be changed after comments are posted, and then push it out to a wider audience.

The advantage, though, would hopefully be a lower barrier of entry for reviewers, and a built in way to track and approve the peer review process.

[Marcus]

Thanks Scott; I'm glad to see that my post has generated some good discussion.

A brief anecdote: Yesterday I learned that an article I co-authored for OCLC Systems & Services: Digital Library Perspectives will be published in Fall 2008. It was submitted on December 1, 2007. In this case I don't even get to see the peer review comments--the editor indicated that revisions will be minor, so perhaps that's the reason.

The article is about emerging alternatives to the impact factor--presumably not that much will change in eight or nine months, but you never know.

I'm not wedded to the idea of a "journal as blog" as we understand blogs now. My real hope is for much faster communication, and a recognition that some level of review can be post-publication. Maybe there could be incentives for thoughtful rather than angry comments on blog threads. And people could become more comfortable with proposing ideas that blossom into a more rigorous investigation, rather than waiting to publish the perfect paper. "Peer review" in this sense would be about improving the kernel of the original idea, just as with the JMLA process. The big difference is that comments would be public; to me that's OK.

This is all very uncharted territory, of course. Who knows where it will all lead? At least we can't say we don't live in interesting times.

[T Scott]

Speed to publication is an important issue. My biggest disappointment, and greatest failure, as editor of the JMLA was that I was unable to shorten that lag. But as I implied in my post, that was strictly a function of the resources put into it. I did all the editing myself and since submissions increased dramatically during my tenure, I could never get ahead. Under Nunzia, there are 3 co-editors and they've cut the lag from submission to acceptance substantially. And since they've been experimenting with putting some articles up on MLANET prior to print publication they're cutting the lag to appearance as well. Shifting to something like OJS as a platform would make it even easier to cut that lag.

As far as your seeing the peer review comments, that's strictly a policy matter -- our policy was that you always saw the reviewers comments, unedited. I'm sorry to hear that the OCLC journal doesn't do it that way, but then, there is tremendous variation among journals as to how they handle almost everything. I've been a member of the World Association of Medical Editors for years, and the discussions on their email list are a constant reminder of how much of a cottage industry much of scholarly publishing still is.

As far as improving the quality of post-publication comments, I'm pretty sceptical about that. Making substantive, thoughtful comments is work, requires time, and doesn't offer much in the way of rewards. Making quick snarky insulting comments (or vapid pointless ones) is easy and, apparently, emotionally satisfying to many. There will always be a very high proportion of noise relative to the amount of value that one can get out of any comment thread.

[K.G. Schneider]

Very good post. I wouldn't even be commenting except to remark, very immaturely, on how the title of this post was truncated. ;-)

[Marcus]

So it comes down to developing an incentive structure for making thoughtful comments, more than whether the publication vehicle is a journal or blog.

Certainly now most comments in blog threads are heated...the big cultural shift would be in understanding these comments as an opportunity to have a real conversation rather than to vent. Will that happen? I don't know, but hope so, even if it takes a long time.

[T Scott]

Marcus, you're more optimistic on this one than I am. Consider the letters to the editor in a local newspaper -- frequently uninformed, often insulting, generally contentious. The letters to the editor in the NYT tend to be thoughtful but that's because they're edited and only present a selection of what gets sent in. I'm afraid that open comments will always draw a high proportion of junk. It's just too easy to spin out a few sentences off the top of your head without having to deal with the consequences of standing behind your ideas.

[Marcus]

The only way this would work is for the comments to be understood and validated as legitimate peer review. There would have to be a reward for making the effort, or nobody would do it. AHIP points seem pretty menial, but something along those lines is what I have in mind; a tangible reason to compose a thoughtful comment is essential. Otherwise, indeed, the junk will win out.

Right now excellent peer review happens in the traditional fashion, because there is prestige attached to being a reviewer in this context. That's what needed online.