Why I Started This Blog

I started this blog late last year, in part, as a response to the plethora of misinformation about bicycle helmets that exists on the internet. I believe much of the problem is a statistical one in that many people either don’t understand or they misuse statistical methods and/or data.

As an academic, the dilemma I found myself in was how to address criticism that originates outside the peer-review environment? When criticism is published in a peer-reviewed journal, it is usually possible to respond in a Letter to the Editor (or similar format). For what it’s worth, the strangely named Bicycle Helmet Research Foundation has an editorial board but no clear avenue for responding to their criticisms — there’s not even a comments section to their posts.

What follows is an email I wrote to Paul Jakma, the author of a blog post that was critical of one of my papers (he mentions he made a similar response here). As I pointed out to him, much of what he wrote was factually incorrect and he misunderstood either our analysis or the data used. To date, he has not corrected his post and, in that time, has left a comment pointing to Chris Rissel’s rejoinder to our paper while ignoring our response. He also seems to have ignored (or is unaware of) our paper showing the benefit of helmet legislation was maintained over the following two decades and there was no “apparent detrimental impact on safety in terms of head injury states” as he states in his post.

I could be wrong, but he doesn’t seem interested in getting his facts correct about our research. I have published my response so that ultimately the reader can make up their mind about the truth.

Email to Paul Jakma (19 January 2012)

I am confused by some of the comments on your blog and on the referring websites.

  1. “Injury rates are seasonal, and they have only very limited data (less than a year) on pre-law rates”

You first point is clearly true which makes monthly injury counts more variable than yearly counts. However, we accounted for that using the X11 method which adjusts for seasonal and abnormal patterns as stated in the paper. Your second point is not true – we used head and limb injury counts 18 months before the law and 18 months afterwards. This gives us plenty of time points (n=36 months) to estimate any trend effects. Inspection of the deviance residuals (not published) indicated we accounted for any seasonal or abnormal variability.

  1. “there appears to be a significant benefit only over a short-period of time”

This point seems to have floated around the blogsphere and is a complete myth. We explicitly state the following in the paper on page 2069.

The tendency towards stability in post-law trends with the inclusion of additional years of data suggests that either 18 months is not sufficient follow up time to accurately detect trends or that the trends shown represent temporally localised changes that did not persist beyond the analysis period. Based on the original analysis there is some evidence that the initial improvement in head injury rates diminished over the 18 months following legislation as shown by the increasing post-law head to limb injury ratios in Fig. 4. Alternatively, the longer term post-law trends being closer to parallel for head and limb injury rates (equivalent to a post-law horizontal line in Fig. 4A and C) supports the idea that the legislation attributable improvement was maintained.

This paragraph seems to be routinely ignored by those who’ve commented about the paper. We chose 18 months post-law to balance out the useable information available pre-law. This is the right approach statistically so that the information before or after the law isn’t dominating the analysis. However, the use of more post-law data demonstrated the benefit was maintained.

  1. “the helmet law has managed to turn a decreasing head injury rate into an increasing head injury rate”

This ties in with my previous point that long term trends (>18 months post law) were flat after the law. Also, note that there were 1288 head injuries in the pre-law period and dropped to 866 in the post-law period, while arm injuries took only a slight dip from 1158 to 1062.

  1. Equal exposure assumption

Your statement is pretty much verbatim from our paper. What we didn’t discuss, and probably should have in retrospect, is that the assumption seemed to work pretty well for arm injures and less so for limb injuries. Their general proximity to the head is likely the reason, i.e., injuries to body parts near each are more correlated than those farther away. Further, the monthly change in the head/arm ratio before the law is 0.997 (95% CI: 0.978-1.016) and 1.006 (95% CI: 0.96-1.06) afterwards. These estimates are virtually textbook examples of no effect/difference (ratio estimate near 1 with tight confidence intervals). Also, for the record, this ratio drops drastically after the law (0.725, 95% CI: 0.539-0.974).

  1. Comments about risk compensation

Many of your arguments seem to be based on #2 and #3, but there are additional reasons I don’t agree with this argument. The primary reason for comparing head to limbs over time is that changes in cycling environment or cycling rates would affect both. I’d recommend reading Povey et al. (1999) as they do a better job explaining this. The idea is that if risk compensation is happening due to more helmet wearing, all injuries would increase (as you point out) but head injuries dropped by almost a third and arm injuries were flat. It has also been suggested the drop is due to less people cycling, but that would mean limb injuries would drop as well and instead dipped slightly. Further, any combination of risk compensation or cycling rate fluctuations would be accounted for by limb injuries (also true for distances cycled). It is true we did not explicitly account for cyclist behaviour or types of cycling, but a person cannot separate their head from their limbs when they get on a bike. So, if a person changes their cycling behaviour after the law, it will affect both.

  1. Comments about fatalities not being included

As I mentioned in my previous email, hospitalisation data can include some fatalities but will not include all. Simply put, Australian data isn’t collected that way. I’ve got a few points about that. First, we used hospitalisation data uniformly before and after the law. If, in fact, helmets turn fatalities into survivable injuries as suggested, you would find an increase in head injuries post-law (which would make it more difficult to find a helmet law benefit). As mentioned above, long term trends for head injuries were flat post-law. That doesn’t mean the point is untrue because fatalities are rare, like I mentioned, and would thus have little influence on the trends in serious bicycle injuries (hospitalisations and fatalities). In a different paper, we found only 39 bicycle-related deaths from all causes in a five year period (2000/01 – 2004/05) (S. Chong et al. / Accident Analysis and Prevention 42 (2010) 290–296). Any cycling-related death is unacceptable, in my opinion, but the inclusion or exclusion of fatalities has little influence on our analysis.

Hospitalised head and limb injuries are quite serious. Most bicycle injuries probably go unreported and most that are reported do not lead to a hospitalisation (most that seem serious on the surface probably only go the ED and don’t get captured as a hospitalisation). These injuries are the ones that can have life-long detrimental effects. In my opinion, a drop in a third of those injuries after a policy change is a clear benefit.

You may have strong views about helmets or helmet laws, but we feel that we have been transparent in our analysis and have given reasonable justification for all aspects of our methods including their limitations. Throughout the process we aimed to be objective and I believe we’ve succeeded on that account.

Regards,

Jake

Advertisements

4 thoughts on “Why I Started This Blog

  1. Hi,

    I’m somewhat disappointed that you have chosen to paint things as if I ignored your comments. Firstly, it was I who emailed you initially to ask for your opinion on some of these points. You emailed a detailed reply, which you have quoted above. However, you have not quoted the email in full. You started it with:

    “I would rather you not quote me directly. ”

    You stated that because in the email you were replying to you, I wrote (commenting on your shorter reply to my first, initial email):

    ” Thanks very much your answer! Would you object if I quoted this? Very interesting paper by the way. Nice paper – I disagree slightly with the more subjective conclusions drawn from it, which seem overly strong based on the post-change trends, but interesting & useful. Thanks!”

    I would appreciate it if, in light of this, you’d revise this blog so as to avoid the impression this gives that I ignored your comments. To be clear:

    1. It was I who emailed you.

    2. I would have been more than happy to have updated my blog with your comments, and explicitly asked you if I could use your comments.

    3. You told me I could not.

    I don’t see what else I could have done.

    If you do not revise this blog I will be more than happy to publish our full correspondence on my own blog.

    Thanks,

    Paul Jakma

  2. Hi Paul,

    I’m confused by your comment. (I note you posted my email on your blog without allowing me to respond first.)

    First off I acknowledge that you contacted me about your blog post and offered me a “right of reply”. I also acknowledge I expressed my desire for you to not post my comment on your blog. However, that is not the issue here and I apologize if that is the impression this post has given you or others. To date, you’re one of a scant few who have been critical of our work that has contacted me (or my co-authors) for their views. You should be commended for that.

    For quite a long time, I was not interested in being involved in the online “helmet wars” debate. I find much of it quite silly generated by people who know just enough about statistics to be dangerous. I also had concerns deviating beyond academic outlets. Over a long period of discussions with colleagues from various areas, I came to the decision the misinformation on the web (including but certainly not limited to your blog) needed to be challenged.

    The issue I have with your blog post is that I presented you with factual errors, which form the basis for your conclusion, and you did not correct them. You do not need my permission to do that. I find it incredibly odd that you could find the time to criticize someone else’s peer-reviewed work, but couldn’t find the time to correct factual errors.

    To reiterate,

    1. We had 18 months of pre-law data (given in the abstract),

    2. We accounted for cyclical (or seasonal) effects using a standard statistical method (bottom of page 2066),

    3. We performed sensitivity analyses to assess whether the “apparent” increase in head injury was real in the long term (page 2069), and

    4. The estimated monthly change in head and arm injuries is about 1 in the pre- and post-law periods (these can be estimated directly from values given in Table 2 of our paper with a bit of algebra and knowledge of generalized linear models).

    Since this paper was published, we have given solid evidence for the comparison between head and arm injury in our response to Chris Rissel. The within-month correlation for these injury types is 0.6. This paper also demonstrates that our model assumptions were valid and there was no residual autocorrelation (which is a main concern if cyclical effects are present).

    These are facts and not my opinions. These are also not very technical, time consuming issues to validate. I’m still left wondering why you did not update your blog after 2.5 years and I believe my criticisms are quite valid. I’d like to be wrong about your motives, but you’ve not given me much to go on.

    • Look, I’m a scientist myself. I have my own opinions and preferences, as does everyone, and I’m sure you do too. Those will affect how we spend our time: which hypotheses we will test before others, the effort we will put into looking for flaws in others’ evaluations of hypotheses. It’s just human nature that we have those and that they affect what we do. The scientific process is there to correct for that, and even exploit it somewhat.

      The blog post wasn’t updated simply because I never went back to it, because of time and because it dropped off my radar. If you had a problem with the blog post, you had several options to correct it that did not depend on “Wait for Paul to validate and write-up my email”. E.g., you could have simply given me permission to post your email. You have just emailed me to notify me of your newer paper.

      Could it be that my own preferences meant I didn’t prioritise following your work, or validating and rewriting your private email to me? Very possibly. However, is it fair for you to expect me to spend my time to redo your email, so as to present your view without it seeming to come from you? I don’t think it’s completely reasonable to think I was required to do that.

      Entia non sunt multiplicanda praeter neccessitatum. The simplest solution to your annoyance with my blog would have been to just email me and say “Actually, yeah, feel free to post my email” or “I’ve got a public paper on that now you can quote/link to”. Further, the simplest explanation for my not updating the blog, in the absence of any communication from you, is just that my attention had moved on from it, and was elsewhere.

      Insinuations about my motives and not being interested in facts seem just a little uncalled for.

      • Some of your comments are reasonable; however, I do question why you don’t seem to be concerned your post contains factual errors. In my own view, you seem like you wanted a discussion regarding your post. Yet, for example, I don’t see how a debate regarding whether 18 months is more or less than a year is worth anyone’s time.

        If someone informed me that one of my posts was factually incorrect, I would have made a quick update pointing to the error. There’s a lot of misinformation out there on cycling safety and factually incorrect information is not helpful.

        The only clues I have about your motives are from what I can gather online. I note that in a recent post titled “UK Cycling Culture” you state “A sub-culture of cyclists are afflicted by Stockholm Syndrome and believe in mandatory helmets for cyclists, even mandatory HiViz.” As a road safety researcher, I find that statement troubling.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s