Reflections on MJA and Elsevier

Virtually all members of the Medical Journal of Australia (MJA) editorial board have resigned in light of a decision to partner with publishing giant Elsevier to produce future issues of MJA. According to reports, 19 of 20 members of the MJA editorial advisory committee resigned after editor-in-chief Stephen Leeder was sacked. Leeder is quoted as saying dealing with Elsevier was “beyond the reach of my ethical tolerance,” while other members note it was an Elsevier journal that published the now infamous Andrew Wakefield MMR vaccine/autism study and Elsevier in the past created “fake” journals sponsored by drug companies.

This incidence has reminded me of an email exchange and online discussion I had with Jennifer Mindell, editor-in-chief of Elsevier’s Journal of Transport and Health (JTH), and Chris Pringle, an Executive Publisher at Elsevier for Geography & Transport. In my original post, I expressed doubt regarding the legitimacy of JTH as an unbiased source of research as the journal is affiliated with the anti-helmet advocacy group Transport and Health Study Group (THSG).

In her response regarding the debate around bike helmets, Mindell states

“I feel strongly that these issues need to be discussed in a respectful way in a scientific forum, rather than in a series of confidential emails full of allegations or in separate blogs, with difficulty in bringing all the evidence together in one place.”

In a long, detailed email, I detailed how I don’t trust Elsevier or Mindell to handle things in a unbaised manner. Specifically, a journal with ties to an anti-helmet organization should not be the host for the debate.

There are also those that have concerns about Elsevier’s business practices.

To perhaps contribute to the discussion around Elsevier and promote transparency, I’ve inserted my full response to Mindell and Pringle below. This was my explanation in regards to me declining an invitation to contribute a bike helmet article to JTH.

Email to Jennifer Mindell and Chris Pringle (16 April 2014)

Jennifer,

Perhaps some more background would help explain our decision not to contribute to the JTH cycle helmets issue.

I became involved in bicycle helmet research around 2010 because a student of mine at the time, Scott Walter, had an interest. Scott is a commuter cyclist (as was I until I needed to take our small child to daycare). Our interest was piqued by a paper by Alex Voukelatos and Chris Rissel in the Journal of the Australasian College of Road Safety (JACRS) and the published criticisms of that paper by Tim Churches. It was clear to us the Voukelatos/Rissel paper was weak because

(1) no statistical analysis was performed (they merely “eyeballed” their incorrectly drawn graphs to reach their conclusions, and unfortunately the human eye is not a well-defined nor reliable statistical procedure),

(2) the numbers literally didn’t add up (as pointed out by Tim, their results were numerically incorrect, as was the time-scale on the graph which formed the entire basis of their analysis, thus they based their conclusions on a graph that showed incorrect numbers at the wrong position on a date axis),

(3) there was a huge discrepancy between head injuries identified by ICD9 and ICD10 indicating incorrect usage of ICD codes (we accept they would likely not be identical, but Voukelatos/Rissel identified over twice as many head injuries using ICD10 than using ICD9 for 1998/99 (511 vs 1170), we were also never able to reproduce the injury counts reported in their paper, there was a clear problem there)

(4) the use of 1988/89 data was problematic as only one diagnosis code existed in the database at that time (so cyclists with both head and arm injuries would only be listed as one – and most likely head instead of arm thereby inflating the 1988/89 head/arm ratio)

(5) there was no comparison to other body parts and no clear justification arm injuries were a good comparator

As I’m sure you are all aware, the Voukelatos/Rissel paper was later retracted by the JACRS. Although there were clear problems with their study, Scott and I felt that the comparison of head and arm injuries over time was an interesting one and could help deal with the lack of exposure data. We were aware of the criticisms of past research on this topic and we felt this strategy had the best chance of addressing those issues in the most rigorous way possible (if the claims by some authors that helmet legislation led to a 30-40% drop in cycling are true, wouldn’t we see a corresponding decline in non-head injury?). This is certainly not a new strategy as psychologists would call the arm injury comparison a dependent, non-equivalent, no treatment control.

Our goal was to correct those issues and present the results as we found them – no matter if they were supportive or not supportive of helmet legislation. In the end, we reported a 29% decline in cycling head injuries (after adjusting for changes in arm and leg injury). Although we found significant declines in cycling head injury at exactly the time of helmet law introduction, we found no significant declines in cycling arm or leg injury at the time of the NSW helmet law commencement. Note that this change corresponded to approximately a 55% increase in helmet wearing. Thus, per “new” helmet wearer, the decline was actually about 0.29/0.55=52.3%. That figure is not unlike what is usually reported in case-control studies of cycling helmet effectiveness.

Note that our conclusion was not that every jurisdiction in the world should have helmet legislation, as reactionary critics have said that our paper recommends, but merely that the calls to repeal helmet legislation in NSW, Australia are not justified by our analysis. How exactly does that make us pro-helmet?

Once our paper was published, our research was attacked by anti-helmet groups like Helmet Freedom (www.helmetfreedom.com) and the Bicycle Helmet Research Foundation (www.cyclehelmets.org – your colleagues Malcolm Wardlaw and John Franklin are both editorial board members), and elsewhere on the blogosphere. It doesn’t seem to matter to these organisations whether their information is correct or not and, in my opinion, they resemble anti-vaccine and climate change denier websites. One site claims we used less than a year worth of data before the law (we had 18 months of data before the law and would’ve gone further back if usable data existed), while another suggested our paper was a government conspiracy to counter de Jong’s cycle helmets cost-benefit paper and Rissel and Wen’s cycle helmet attitudes survey. The latter claim is quite ludicrous considering our paper was published first of the three, and we actually cited a working draft of de Jong’s paper (as it was the only version available to us at the time), and we published a direct response to the Rissel and Wen paper
(http://www.publish.csiro.au/?act=view_file&file_id=HE12076.pdf). After pointing out to Rissel and Wen that they had misinterpreted their own logistic regression results and ignored their own results that run contrary to their conclusions, they state “Olivier et al. are unashamedly pro-helmet advocates,7,8”. The citations are our 2011 paper and Rissel’s rejoinder to our paper.

Rissel’s rejoinder to our 2011 Accident Analysis and Prevention (AAP) paper was published in 2012 in AAP (another Elsevier journal). The journal’s editors, Karl Kim and Rune Elvik, did not offer us the right of reply or even inform us his rejoinder existed. We found out about it after it was published. In his rejoinder, Rissel cites as primary evidence his own already retracted paper discussed above which is, quite frankly, rather bizarre academic behaviour, and in violation of COPE guidelines. The data and analysis used by Voukelatos and Rissel is clearly wrong and I find it troubling that Rissel was allowed to proclaim his conclusions were unaffected by incorrect data (he never corrected the data or the graph used in his analysis, so he had no evidence to even support his claim). Rissel also misquoted figures/analyses from other studies — a 2011 Rune Elvik paper (who was still editor-in-chief of AAP at the time, this is also the meta-analysis paper that has had at least two corrigendums), an assessment of the Western Australia helmet law (Hendrie et al), and an assessment of UK cycling fatalities (Hynd et al).

One of Rissel’s criticisms was we did not share our data. Since Rissel has an appointment with the NSW Ministry of Health, he is keenly aware of the privacy laws governing the sharing of hospitalisation data. Although we had access to unit record data, we were only approved to release the information contained in our paper, nothing more. I also found his request for our data strange considering he had access to the very same NSW hospitalisation data (even if he used it incorrectly in his paper). Our paper clearly listed the ICD9 codes we used, so he could have easily replicated the aggregated data identified from the unit records.

Additionally, Raphael Grzebieta and I had an email exchange with Chris Rissel two weeks before he submitted his rejoinder. Most of the criticisms found in his rejoinder were in his email. I responded in detail why his criticisms were unfounded. Rissel even alludes to this email exchange in his rejoinder where he states

“A somewhat arbitrary 18 months before and after the official start of the mandatory helmet legislation was used as the basis of regression modelling, limited by poor quality data prior to the legislation (Olivier, personal communication).”

This was actually in reference to there being only 18 months of usable NSW hospitalisation data before the helmet law and not hospitalisation data being of poor quality. Note that since our analysis used the same source data as Voukelatos and Rissel, their retracted paper and Rissel’s post-retraction contention the helmet law had no effect
suffers the same problem (as would any assessment of the NSW helmet law). Note that Rissel never received my permission for publishing the contents of our personal email exchange, and we find his failure to take any heed of our detailed response to the criticisms contained in his AAP rejoinder, which he received two weeks prior to submission of the rejoinder article, to be disingenuous in the extreme. Given these types of behaviour, you will understand our reluctance to submit to a journal with him on the editorial board.

Once we were aware of Rissel’s rejoinder and his bad faith in failing to modify his rejoinder in light of our prior correspondence with him, we alerted these and other issues to Karl Kim, Rune Elvik and, eventually, Chris Pringle. Kim and Elvik’s initial reaction was Rissel’s rejoinder should be retracted. However, Elsevier in the end decided not to
and offered us the option of replying to his criticisms which would go out to peer-review. Much of the response had already been written because we had already responded to Rissel through email.

Note that the retracted Voukelatos/Rissel paper is presently hosted by anti-helmet websites (for example, http://www.cycle-helmets.com/rissel.pdf and http://www.cyclehelmets.org/1146.html) without any indication of the retraction. These organisations, including the Bicycle Helmet Research Foundation – again, your colleagues Wardlaw and Franklin are editorial board members, would be in violation of COPE guidelines if they were, in fact, research organisations.

Tim has also made you aware of Elvik’s papers in AAP and Elsevier’s handling of that situation. I find it incredibly strange that the editor-in-chief of a journal can have the key results in a paper corrected in a major way not once, but twice, without having the paper retracted, nor having the paper re-submitted for peer-review, nor even modifying his conclusions or discussion in light of such major calculation errors. The whole episode reflects extremely poorly on Elsevier’s methods and ethical standards as a scientific publisher of repute.

There were also unfounded criticisms made by you, Malcolm Wardlaw and John Franklin in a THSG presentation (see attached [link: http://www.transportandhealth.
org.uk/wp-content/uploads/2011/10/THSG-seminar_
Cycling-safety.pptx]). You state you want the discussion of cycle helmets to be “less adversarial” as it “does not allow for defining common ground”, yet your actions indicate otherwise. Much of your presentation on helmets is a criticism of other people’s work. When I came across your presentation, it was clear right away that you had manipulated the axes and scales of a graph from our 2011 AAP paper to make your point (these are both massive no-no’s that are usually taught in introductory statistics courses). However, how was I supposed to react to your unfounded criticisms? You didn’t publish them in a journal or conference proceedings – I downloaded it from the THSG website (was it peer-reviewed?). As a researcher, how am I supposed to address such criticisms? You seem to be displeased I expressed my views about the THSG and JTH openly on my blog, yet isn’t that exactly what you did to our paper? If you don’t agree with my blog entry, you can at least leave a comment. We were not offered that courtesy by you.

In your presentation, you include the plot below. I’ve seen this plot elsewhere on anti-helmet websites. I find it interesting, not for what it presents but what it doesn’t. The effect of a trend is presented, but each point represents observations taken on one day taken five years apart. The information conveyed does not indicate how the weather affected census data done in some years and there’s no information about how modes of transportation are categorised. In earlier censuses, multiple modes of travel including a bicycle leg would be categorised as “cycling”. Yet, later years, the categorisation was by primary mode. So, someone taking the Manly ferry coupled with cycling trips to and from may be included with cycling in some censuses and not in others. It also ignores the fact that 1.14% of trips in Australian cities (where most people live and cycle) were by bicycle in the 1986 survey and this figure changed little (1.13%) by the next one in 1991. These are the two censuses on either side of when most Australians were subjected to helmet legislation. It also ignores declines in other modes of transportation except cars (see attached figure created by Tim Churches). Was helmet legislation also the causal factor for declines in other modes of transport?

BHRF Plot

I also note that your THSG co-chair and JTH editorial board member Stephen Watkins has stated “Mass helmet use has not reduced serious head injuries to a noticeable degree relative to general improvements in road safety seen for other road users (i.e. secular trends) (http://www.cyclist.ie/wp-content/uploads/2010/11/Transport-Health-Study-
Group-Cycle-Helmets.pdf). Another quote of his is “The most important is that it has been shown in a number of different jurisdictions that compulsory (and enforced) helmet-wearing reduces cycle use.” These statements are interesting when juxtaposed together. I wholly agree that separating out the effect of helmet legislation (or any other
intervention) from secular trends is important. We did just that in our 2011 analysis. However, this rigour never seems to apply to surveys used to draw conclusions about cycling numbers (whether designed for that purpose or not). There were 4 NSW cycling surveys taken in Oct 1990, Apr 1991, Apr 1992 and Apr 1993. The helmet law came
into effect on 1 Jan 1991 (adults) and 1 July 1991 (kids). The adult data, whose overall counts change little over the 4 surveys, is not found in any of Robinson’s work (it effectively doesn’t exist) and she uses only the 1991-1993 surveys for kids. That means only one data point is used pre-helmet law. How can anyone estimate secular trends in cycling from one data point? Two data points isn’t even realistic as only a line can be estimated.

In fact, trend data does exist for Australian metro areas dating back to 1900 (see attached). Cycling mode share increased up to WWII peaking at about 8-9% mode share. This was followed by a steady decline thereafter. The big change in cycling rates came at the expense of car travel and not helmet legislation.

Australia metro travel

I am confused when you say the THSG is “less likely” to revise their policy as a result of our declining your request to contribute to the JTH helmet issue. Shouldn’t revision of policy happen naturally as more evidence comes to light? Surely the evidence does not have to appear in the JTH in order to be considered by THSG. For example, in reference
of your comment that “many people in public health” believed helmets only protected against “minor issues such as grazes”, a recent article in Accident Analysis and Prevention (http://www.sciencedirect.com/science/article/pii/S000145751400061X) states

“Considering a realistic bicycle accident scenario documented in the literature (Fahlstedt et al., 2012) where a cyclist was thrown at 20 km/h (i.e. 5.6 m/s which corresponds to a drop height of approximately 1.5 m), our analysis indicates that a helmeted cyclist in this situation would have a 9% chance of sustaining the severe brain and skull injuries noted above whereas an unhelmeted cyclist would have sustained these injuries with 99.9% certainty. In other words, a helmet would have reduced the probability of skull fracture or life threatening brain injury from very likely to highly unlikely.”

We also published a paper earlier this year where we found helmets reduced the odds of severe head injury by up to 74% (these were NSW cyclists hospitalised after a motor vehicle crash and reported to the police from 2001-2009). Severe injuries included “Open wound of head with intracranial injury” (S01.83), “Multiple fractures involving skull
and facial bones” (S02.7), “Fracture of skull and facial bones, part unspecified” (S02.9), “Loss of consciousness [30 mins-24hrs]” (S06.03), “Loss of consciousness prolonged without return of consciousness ” (S06.05), “Traumatic cerebral oedema” (S06.1), “Diffuse brain injury” (S06.2), “Other diffuse cerebral & cerebellar injury” (S06.28), “Traumatic subdural haemorrhage” (S06.5), “Traumatic subarachnoid haemorrhage” (S06.6), “Other intracranial injuries” (S06.8), and “Intracranial injury, unspecified” (S06.9). None of these are minor injuries.

One of the main tenet’s of scientific discourse is to re-evaluate one’s position in light of new evidence. Shouldn’t you reflect on the THSG’s policy in light of this and other evidence? Whether we submit an article or not to the JTH?

It takes an awful lot of effort on the part of several researchers, while coordinating everyone’s busy schedules, to produce a high-quality research paper. There has been nothing in my experience with you, the THSG, the JTH or Elsevier that would indicate submitting a “pro-helmet” paper to JTH would be worth the effort. However, I suppose
that could change in light of new evidence

Regards,

Jake

Advertisements