Colin Clarke’s Assessment of Australian Helmet Laws

Colin Clarke, a former engineering student, left a comment to my recent post regarding my experiences with Elsevier. Although this post was about an email I sent to Jennifer Mindell (editor in chief of the Journal of Transport and Health) and Chris Pringle (Elsevier Executive Publisher), Clarke took it upon himself to promote his views on bicycle helmets.

Since his comment was really not relevant to my post, I thought about rejecting his comment for quite a while. However, since I assume he would just criticize me somewhere else for censorship, I allowed his comment to be published.

I’ve had numerous online discussions with Clarke about bike helmets (see here, for example). Years ago, it seems Clarke wrote a standard criticism about helmets and, if you’ve ever read one, you’ve essentially read them all. I’ve pointed out fallacies in his arguments many times, and he continually ignores those criticisms.

Clarke’s primary argument is what I call the ‘helmet law as cycling deterrent’ hypothesis. There’s no real, solid evidence to support this hypothesis, yet it has not stopped those like Clarke to proclaim its truth to anyone who cares to listen.

In his comment, Clarke ‘estimates’ (and I’m being generous here) 413 cycling head injuries in NSW are expected in 1992 (this assumes a 30% drop in cycling from 1991 to 1992 and 590 head injuries reported in 1991, i.e., 590*0.7=413). Instead, there were 648 NSW head injury hospitalizations in 1992 (hospital data can be found here).

There are two problems with this approach. First, NSW adults were subject to helmet legislation for the entirety of 1991 and children for half. Clarke would have needed 1990/91 data to make that point (Jan-Dec for adults, Jul-Jun for children). This also assumes there aren’t any existing trends in cycling rates (no data exists to even attempt to estimate those secular trends, except for 1.5 years of pre-law hospital data that doesn’t support Clarke’s hypothesis). Second, if you apply the same logic to arm injuries, you get an expected 660*0.7=462 arm injuries in 1992 (there were 760). In fact, not adjusting for population, head injuries increased by 9.8% while arm injuries increased 15% between those years. None of this suggests Clarke’s hypothesis is true.

Importantly, the injury data comes from a census of medically diagnosed hospitalizations at that time and is therefore of high quality. To justify the drop in cyclists, Clarke relies on repurposed cycling counts from helmet use surveys. Standing on a street corner observing helmet use among cyclists is prone to bias when used as a measure of cycling exposure (this makes the Victoria and NSW helmet use surveys problematic if used for that purpose). Data collected in this way does not readily capture recreational cyclists (which is the backbone of Australian cycling) and does not/cannot account for changes in cycling routes or patterns (observation locations are fixed from year to year). It is known that many kids stopped cycling to school in Australia because either their parents or their schools started discouraging it or outright forbade it. This was not due to helmet legislation, although the two issues may share a common causal factor (i.e., parents/society concerned about cycling injury). This did not mean kids stopped cycling after school or on weekends.

Data collected through (stratified) random sampling does not suffer from those problems. The only such surveys in Australia, that I’m aware of around the time of helmet legislation, are from South Australia[1] and Western Australia[2]. Below is a summary of their results for proportion of responses to one of four categories of cycling frequency. (Note these reports do not seem to exist online and I’ve decided to host them on my webpage as I feel very few have ever read them or any other source material.)

South Australia Western Australia
1990 1993 1989 1993
At least weekly 21.8 21.0 At least weekly 26.6 27.7
At least monthly 5.2 6.0 At least every 3 months 11.1 11.6
At least every 3 months 3.9 4.4 At least once per year 10.3 11.5
Less often or never 69.1 68.6 Never 52.0 49.2

The results indicate no change in cycling in either SA or WA. We also found that to be true for non-head cycling injury hospitalisations in NSW[3]. Neither arm nor leg injuries declined significantly following helmet legislation, and certainly not at the 20% rate cited by Clarke or the 30-40% figure quoted by others. If cycling in NSW dropped substantially after helmet legislation as suggested, I have always wondered why is there no evidence in the census of injury hospitalizations?

Clarke cites two of his own papers to justify his position. The first is a non-peer reviewed conference paper titled Evaluation of Australia’s bicycle helmet laws. I received numerous spam emails about this conference for months. Unfortunately, I get 10+ emails a day for such conferences or predatory publishers asking me to submit ‘my latest research’. Legitimate conferences and publishes do not need to advertise so aggressively.

As with much of his comments about other people’s work, Clarke’s paper is loaded with cherry-picked data and not a shred of statistical/analytic rigor. I’ll detail a few examples.

Figure 3 on page 6 are counts of cyclists from helmet use surveys in Victoria in 1990 and 1991. Back in June 2013, Tim Churches pointed out this figure omits the 1992 data and does not aggregate by age, which is important considering cycling numbers differ substantially across age groups. Below is a plot created using Tim’s R code.

Vic_Cyc_Counts

The counts of cyclists for two age groups are virtually identical from 1990 to 1992 (age 5-11 and 18+). The only decline was for 12-17 year olds. If you assume this data is an accurate estimate of cycling exposure (keep in mind I don’t agree with this stance), the cycling decline is age dependent and not true overall.

Clarke also cites a WA report from Heathcote. I could not find a copy of this 1993 report anywhere; however, I did find a 1994 report by the same author[2]. According to 1994 report, the 1993 report concerned helmet use and “recommended the need for further research into cycling trends, and attitudes towards the helmet wearing law”. This was the motivation behind the 1994 report. Therefore, the later report is the more relevant here.

The numbers of recreational cyclists reported by Clarke are not found in the 1994 report; however, Table 8 summarizes Usual Destinations with 21.3% in 1989 and 17.4% in 1993 reporting “To a venue for sport/recreation”. This is not near a 57% reduction in recreation cycling as reported by Clarke and, when taken out of context, ignores that there was no real change in WA cycling frequency as was reported in the same surveys (see above table).

In his Table 13, Clarke reports proportions cycling to work in five year intervals (I discussed problems with how this data is interpreted in a previous post). In addition to the usual problems with this data, Clarke creates additional problems by “estimating” the proportion in 1990. He gives absolutely no justification for this figure except for links to anti-helmet advocacy websites. I guess he could have extrapolated that figure, but such an estimate would be fraught with problems from data taken over three days 5 years apart. Estimating trends from such data completely ignores day-to-day variability.

Clarke also cites his 2012 NZMJ paper. Note I have been critical of this paper on at least four occasions [4-7]. The last citation was a systematic review of the methods used to assess the New Zealand bicycle helmet law. Against quality criteria for assessing a population-based intervention, Clarke’s study scored the lowest of five included studies.

Clarke is quite active on the internet sharing his views on helmet legislation. This includes routine editing of the Wikipedia pages on bicycle helmets under the pseudonym “Colin at cycling” where his contributions have been to promote his NZMJ paper (see the edits page). Although I’m generally supportive of Wikipedia (even contributing money), I find it disturbing anti-helmet advocates like Clarke have taken control of the bicycle helmet content. Like Clarke, they all have conflicts of interest which are never reported (see Dorothy Robinson’s Wikipedia page for a flavor of what’s happened, she has a clear conflict of interest since she essentially promotes her own work and strongly opposes the inclusion of other studies that criticize her work).

If anything, Clarke is steadfast in his beliefs helmets are ‘bad’ and he’s not afraid to share his views. However, I’d be more willing to accept his points if he’d stop cherry-picking data and present a balanced view of the topic. My past experiences with him indicate that is highly unlikely.

  1. Marshall, J. & White, M. (1994). Evaluation of the compulsory helmet wearing legislation for bicyclists in South Australia. South Australian Department of Transport, Walkerville, SA.
  2. Heathcote, B. & Maisey, G. (1994) Bicyclist use and attitudes to the helmet wearing law. Traffic Board of Western Australia, Perth.
  3. Walter, S.R., Olivier, J., Churches, T., & Grzebeita, R. (2011). The impact of compulsory cycle helmet legislation on cyclist head injuries in New South Wales, Australia. Accident Analysis and Prevention, 43, 2064–2071.
  4. Olivier, J. (2012) Don’t blame mandatory helmets for cyclist deaths in New Zealand. The Conversation.
  5.  Wang, J., Olivier, J. & Grzebieta, R. (2014). Response to ‘Evaluation of New Zealand’s bicycle helmet law’ article. The New Zealand Medical Journal, 127, 106-108.
  6. Olivier, J., Wang, J.J.J., Walter, S. & Grzebieta, R. (2014). Anti-helmet arguments: lies, damned lies and flawed statistics. Journal of the Australasian College of Road Safety, 25, 10-23.
  7. Olivier, J., Wang, J.J.J. & Grzebieta, R. (2014). A systematic review of methods used to assess mandatory bicycle helmet legislation in New Zealand. Journal of the Australasian College of Road Safety, 25, 24-31.

Reflections on MJA and Elsevier

Virtually all members of the Medical Journal of Australia (MJA) editorial board have resigned in light of a decision to partner with publishing giant Elsevier to produce future issues of MJA. According to reports, 19 of 20 members of the MJA editorial advisory committee resigned after editor-in-chief Stephen Leeder was sacked. Leeder is quoted as saying dealing with Elsevier was “beyond the reach of my ethical tolerance,” while other members note it was an Elsevier journal that published the now infamous Andrew Wakefield MMR vaccine/autism study and Elsevier in the past created “fake” journals sponsored by drug companies.

This incidence has reminded me of an email exchange and online discussion I had with Jennifer Mindell, editor-in-chief of Elsevier’s Journal of Transport and Health (JTH), and Chris Pringle, an Executive Publisher at Elsevier for Geography & Transport. In my original post, I expressed doubt regarding the legitimacy of JTH as an unbiased source of research as the journal is affiliated with the anti-helmet advocacy group Transport and Health Study Group (THSG).

In her response regarding the debate around bike helmets, Mindell states

“I feel strongly that these issues need to be discussed in a respectful way in a scientific forum, rather than in a series of confidential emails full of allegations or in separate blogs, with difficulty in bringing all the evidence together in one place.”

In a long, detailed email, I detailed how I don’t trust Elsevier or Mindell to handle things in a unbaised manner. Specifically, a journal with ties to an anti-helmet organization should not be the host for the debate.

There are also those that have concerns about Elsevier’s business practices.

To perhaps contribute to the discussion around Elsevier and promote transparency, I’ve inserted my full response to Mindell and Pringle below. This was my explanation in regards to me declining an invitation to contribute a bike helmet article to JTH.

Email to Jennifer Mindell and Chris Pringle (16 April 2014)

Jennifer,

Perhaps some more background would help explain our decision not to contribute to the JTH cycle helmets issue.

I became involved in bicycle helmet research around 2010 because a student of mine at the time, Scott Walter, had an interest. Scott is a commuter cyclist (as was I until I needed to take our small child to daycare). Our interest was piqued by a paper by Alex Voukelatos and Chris Rissel in the Journal of the Australasian College of Road Safety (JACRS) and the published criticisms of that paper by Tim Churches. It was clear to us the Voukelatos/Rissel paper was weak because

(1) no statistical analysis was performed (they merely “eyeballed” their incorrectly drawn graphs to reach their conclusions, and unfortunately the human eye is not a well-defined nor reliable statistical procedure),

(2) the numbers literally didn’t add up (as pointed out by Tim, their results were numerically incorrect, as was the time-scale on the graph which formed the entire basis of their analysis, thus they based their conclusions on a graph that showed incorrect numbers at the wrong position on a date axis),

(3) there was a huge discrepancy between head injuries identified by ICD9 and ICD10 indicating incorrect usage of ICD codes (we accept they would likely not be identical, but Voukelatos/Rissel identified over twice as many head injuries using ICD10 than using ICD9 for 1998/99 (511 vs 1170), we were also never able to reproduce the injury counts reported in their paper, there was a clear problem there)

(4) the use of 1988/89 data was problematic as only one diagnosis code existed in the database at that time (so cyclists with both head and arm injuries would only be listed as one – and most likely head instead of arm thereby inflating the 1988/89 head/arm ratio)

(5) there was no comparison to other body parts and no clear justification arm injuries were a good comparator

As I’m sure you are all aware, the Voukelatos/Rissel paper was later retracted by the JACRS. Although there were clear problems with their study, Scott and I felt that the comparison of head and arm injuries over time was an interesting one and could help deal with the lack of exposure data. We were aware of the criticisms of past research on this topic and we felt this strategy had the best chance of addressing those issues in the most rigorous way possible (if the claims by some authors that helmet legislation led to a 30-40% drop in cycling are true, wouldn’t we see a corresponding decline in non-head injury?). This is certainly not a new strategy as psychologists would call the arm injury comparison a dependent, non-equivalent, no treatment control.

Our goal was to correct those issues and present the results as we found them – no matter if they were supportive or not supportive of helmet legislation. In the end, we reported a 29% decline in cycling head injuries (after adjusting for changes in arm and leg injury). Although we found significant declines in cycling head injury at exactly the time of helmet law introduction, we found no significant declines in cycling arm or leg injury at the time of the NSW helmet law commencement. Note that this change corresponded to approximately a 55% increase in helmet wearing. Thus, per “new” helmet wearer, the decline was actually about 0.29/0.55=52.3%. That figure is not unlike what is usually reported in case-control studies of cycling helmet effectiveness.

Note that our conclusion was not that every jurisdiction in the world should have helmet legislation, as reactionary critics have said that our paper recommends, but merely that the calls to repeal helmet legislation in NSW, Australia are not justified by our analysis. How exactly does that make us pro-helmet?

Once our paper was published, our research was attacked by anti-helmet groups like Helmet Freedom (www.helmetfreedom.com) and the Bicycle Helmet Research Foundation (www.cyclehelmets.org – your colleagues Malcolm Wardlaw and John Franklin are both editorial board members), and elsewhere on the blogosphere. It doesn’t seem to matter to these organisations whether their information is correct or not and, in my opinion, they resemble anti-vaccine and climate change denier websites. One site claims we used less than a year worth of data before the law (we had 18 months of data before the law and would’ve gone further back if usable data existed), while another suggested our paper was a government conspiracy to counter de Jong’s cycle helmets cost-benefit paper and Rissel and Wen’s cycle helmet attitudes survey. The latter claim is quite ludicrous considering our paper was published first of the three, and we actually cited a working draft of de Jong’s paper (as it was the only version available to us at the time), and we published a direct response to the Rissel and Wen paper
(http://www.publish.csiro.au/?act=view_file&file_id=HE12076.pdf). After pointing out to Rissel and Wen that they had misinterpreted their own logistic regression results and ignored their own results that run contrary to their conclusions, they state “Olivier et al. are unashamedly pro-helmet advocates,7,8”. The citations are our 2011 paper and Rissel’s rejoinder to our paper.

Rissel’s rejoinder to our 2011 Accident Analysis and Prevention (AAP) paper was published in 2012 in AAP (another Elsevier journal). The journal’s editors, Karl Kim and Rune Elvik, did not offer us the right of reply or even inform us his rejoinder existed. We found out about it after it was published. In his rejoinder, Rissel cites as primary evidence his own already retracted paper discussed above which is, quite frankly, rather bizarre academic behaviour, and in violation of COPE guidelines. The data and analysis used by Voukelatos and Rissel is clearly wrong and I find it troubling that Rissel was allowed to proclaim his conclusions were unaffected by incorrect data (he never corrected the data or the graph used in his analysis, so he had no evidence to even support his claim). Rissel also misquoted figures/analyses from other studies — a 2011 Rune Elvik paper (who was still editor-in-chief of AAP at the time, this is also the meta-analysis paper that has had at least two corrigendums), an assessment of the Western Australia helmet law (Hendrie et al), and an assessment of UK cycling fatalities (Hynd et al).

One of Rissel’s criticisms was we did not share our data. Since Rissel has an appointment with the NSW Ministry of Health, he is keenly aware of the privacy laws governing the sharing of hospitalisation data. Although we had access to unit record data, we were only approved to release the information contained in our paper, nothing more. I also found his request for our data strange considering he had access to the very same NSW hospitalisation data (even if he used it incorrectly in his paper). Our paper clearly listed the ICD9 codes we used, so he could have easily replicated the aggregated data identified from the unit records.

Additionally, Raphael Grzebieta and I had an email exchange with Chris Rissel two weeks before he submitted his rejoinder. Most of the criticisms found in his rejoinder were in his email. I responded in detail why his criticisms were unfounded. Rissel even alludes to this email exchange in his rejoinder where he states

“A somewhat arbitrary 18 months before and after the official start of the mandatory helmet legislation was used as the basis of regression modelling, limited by poor quality data prior to the legislation (Olivier, personal communication).”

This was actually in reference to there being only 18 months of usable NSW hospitalisation data before the helmet law and not hospitalisation data being of poor quality. Note that since our analysis used the same source data as Voukelatos and Rissel, their retracted paper and Rissel’s post-retraction contention the helmet law had no effect
suffers the same problem (as would any assessment of the NSW helmet law). Note that Rissel never received my permission for publishing the contents of our personal email exchange, and we find his failure to take any heed of our detailed response to the criticisms contained in his AAP rejoinder, which he received two weeks prior to submission of the rejoinder article, to be disingenuous in the extreme. Given these types of behaviour, you will understand our reluctance to submit to a journal with him on the editorial board.

Once we were aware of Rissel’s rejoinder and his bad faith in failing to modify his rejoinder in light of our prior correspondence with him, we alerted these and other issues to Karl Kim, Rune Elvik and, eventually, Chris Pringle. Kim and Elvik’s initial reaction was Rissel’s rejoinder should be retracted. However, Elsevier in the end decided not to
and offered us the option of replying to his criticisms which would go out to peer-review. Much of the response had already been written because we had already responded to Rissel through email.

Note that the retracted Voukelatos/Rissel paper is presently hosted by anti-helmet websites (for example, http://www.cycle-helmets.com/rissel.pdf and http://www.cyclehelmets.org/1146.html) without any indication of the retraction. These organisations, including the Bicycle Helmet Research Foundation – again, your colleagues Wardlaw and Franklin are editorial board members, would be in violation of COPE guidelines if they were, in fact, research organisations.

Tim has also made you aware of Elvik’s papers in AAP and Elsevier’s handling of that situation. I find it incredibly strange that the editor-in-chief of a journal can have the key results in a paper corrected in a major way not once, but twice, without having the paper retracted, nor having the paper re-submitted for peer-review, nor even modifying his conclusions or discussion in light of such major calculation errors. The whole episode reflects extremely poorly on Elsevier’s methods and ethical standards as a scientific publisher of repute.

There were also unfounded criticisms made by you, Malcolm Wardlaw and John Franklin in a THSG presentation (see attached [link: http://www.transportandhealth.
org.uk/wp-content/uploads/2011/10/THSG-seminar_
Cycling-safety.pptx]). You state you want the discussion of cycle helmets to be “less adversarial” as it “does not allow for defining common ground”, yet your actions indicate otherwise. Much of your presentation on helmets is a criticism of other people’s work. When I came across your presentation, it was clear right away that you had manipulated the axes and scales of a graph from our 2011 AAP paper to make your point (these are both massive no-no’s that are usually taught in introductory statistics courses). However, how was I supposed to react to your unfounded criticisms? You didn’t publish them in a journal or conference proceedings – I downloaded it from the THSG website (was it peer-reviewed?). As a researcher, how am I supposed to address such criticisms? You seem to be displeased I expressed my views about the THSG and JTH openly on my blog, yet isn’t that exactly what you did to our paper? If you don’t agree with my blog entry, you can at least leave a comment. We were not offered that courtesy by you.

In your presentation, you include the plot below. I’ve seen this plot elsewhere on anti-helmet websites. I find it interesting, not for what it presents but what it doesn’t. The effect of a trend is presented, but each point represents observations taken on one day taken five years apart. The information conveyed does not indicate how the weather affected census data done in some years and there’s no information about how modes of transportation are categorised. In earlier censuses, multiple modes of travel including a bicycle leg would be categorised as “cycling”. Yet, later years, the categorisation was by primary mode. So, someone taking the Manly ferry coupled with cycling trips to and from may be included with cycling in some censuses and not in others. It also ignores the fact that 1.14% of trips in Australian cities (where most people live and cycle) were by bicycle in the 1986 survey and this figure changed little (1.13%) by the next one in 1991. These are the two censuses on either side of when most Australians were subjected to helmet legislation. It also ignores declines in other modes of transportation except cars (see attached figure created by Tim Churches). Was helmet legislation also the causal factor for declines in other modes of transport?

BHRF Plot

I also note that your THSG co-chair and JTH editorial board member Stephen Watkins has stated “Mass helmet use has not reduced serious head injuries to a noticeable degree relative to general improvements in road safety seen for other road users (i.e. secular trends) (http://www.cyclist.ie/wp-content/uploads/2010/11/Transport-Health-Study-
Group-Cycle-Helmets.pdf). Another quote of his is “The most important is that it has been shown in a number of different jurisdictions that compulsory (and enforced) helmet-wearing reduces cycle use.” These statements are interesting when juxtaposed together. I wholly agree that separating out the effect of helmet legislation (or any other
intervention) from secular trends is important. We did just that in our 2011 analysis. However, this rigour never seems to apply to surveys used to draw conclusions about cycling numbers (whether designed for that purpose or not). There were 4 NSW cycling surveys taken in Oct 1990, Apr 1991, Apr 1992 and Apr 1993. The helmet law came
into effect on 1 Jan 1991 (adults) and 1 July 1991 (kids). The adult data, whose overall counts change little over the 4 surveys, is not found in any of Robinson’s work (it effectively doesn’t exist) and she uses only the 1991-1993 surveys for kids. That means only one data point is used pre-helmet law. How can anyone estimate secular trends in cycling from one data point? Two data points isn’t even realistic as only a line can be estimated.

In fact, trend data does exist for Australian metro areas dating back to 1900 (see attached). Cycling mode share increased up to WWII peaking at about 8-9% mode share. This was followed by a steady decline thereafter. The big change in cycling rates came at the expense of car travel and not helmet legislation.

Australia metro travel

I am confused when you say the THSG is “less likely” to revise their policy as a result of our declining your request to contribute to the JTH helmet issue. Shouldn’t revision of policy happen naturally as more evidence comes to light? Surely the evidence does not have to appear in the JTH in order to be considered by THSG. For example, in reference
of your comment that “many people in public health” believed helmets only protected against “minor issues such as grazes”, a recent article in Accident Analysis and Prevention (http://www.sciencedirect.com/science/article/pii/S000145751400061X) states

“Considering a realistic bicycle accident scenario documented in the literature (Fahlstedt et al., 2012) where a cyclist was thrown at 20 km/h (i.e. 5.6 m/s which corresponds to a drop height of approximately 1.5 m), our analysis indicates that a helmeted cyclist in this situation would have a 9% chance of sustaining the severe brain and skull injuries noted above whereas an unhelmeted cyclist would have sustained these injuries with 99.9% certainty. In other words, a helmet would have reduced the probability of skull fracture or life threatening brain injury from very likely to highly unlikely.”

We also published a paper earlier this year where we found helmets reduced the odds of severe head injury by up to 74% (these were NSW cyclists hospitalised after a motor vehicle crash and reported to the police from 2001-2009). Severe injuries included “Open wound of head with intracranial injury” (S01.83), “Multiple fractures involving skull
and facial bones” (S02.7), “Fracture of skull and facial bones, part unspecified” (S02.9), “Loss of consciousness [30 mins-24hrs]” (S06.03), “Loss of consciousness prolonged without return of consciousness ” (S06.05), “Traumatic cerebral oedema” (S06.1), “Diffuse brain injury” (S06.2), “Other diffuse cerebral & cerebellar injury” (S06.28), “Traumatic subdural haemorrhage” (S06.5), “Traumatic subarachnoid haemorrhage” (S06.6), “Other intracranial injuries” (S06.8), and “Intracranial injury, unspecified” (S06.9). None of these are minor injuries.

One of the main tenet’s of scientific discourse is to re-evaluate one’s position in light of new evidence. Shouldn’t you reflect on the THSG’s policy in light of this and other evidence? Whether we submit an article or not to the JTH?

It takes an awful lot of effort on the part of several researchers, while coordinating everyone’s busy schedules, to produce a high-quality research paper. There has been nothing in my experience with you, the THSG, the JTH or Elsevier that would indicate submitting a “pro-helmet” paper to JTH would be worth the effort. However, I suppose
that could change in light of new evidence

Regards,

Jake