Pepijn van Erp

Apart from weird theories which are put forward to explain how homeopathy, acupuncture and similar types of quackery work, proponents of these alternative treatments are spreading two other myths: in a breakthrough report the Swiss government has affirmed homeopathy as effective treatment and Dutch research by Prof. Peter Kooreman and Erik Baars shows that General Practioners (GPs) who are trained in Complementary and Alternative Medicine (CAM) work cheaper: their patients have on average 15 percent lower healthcare costs than those of regular GPs.
Both myths can be found for instance together in a Huffington Post blog by ‘Mr Homeopathy’ Dana Ullman. Views on the status of the ‘Swiss report’, which accord more to reality, can be found here and here. In this blog I will tackle the other myth, that CAM GPs have been shown to work 15 percent cheaper in the Netherlands. 

Erik Baars

First critics: The Dutch Society against Quackery
The research of Kooreman and Baars (K&B) was mainly brought to public attention via a newspaper article (De Volkskrant , June 8^th 2010) and rather quickly criticized by dr. Cees Renckens en Jan Willem Nienhuys of the Society against Quackery. A big problem for a direct comparison of the patients of regular and CAM GPs is that these populations differ quite a bit in socioeconomic status and that factor is directly related to health (and costs for healthcare). Strangely enough, higher educated people, who tend to have better socioeconomic circumstances, seem to be more attracted to CAM. So if it might seem that patients of CAM GPs are healthier, it probably has more to do with their better starting position and not by CAM treatments. Of course K&B were aware of this problem and they controlled for this by comparing the data (which they got from big health insurer) on the lowest level of the Dutch zipcode system (about 16 households). On this level socioeconomic differences are expected to be not that big and it looks like a reasonable approach.
The newspaper article mentioned that the study showed 15 percent lower costs for patients of CAM GPs, but Renckens and Nienhuys noticed that this figure could not be found in the article itself. Only a difference of  7 percent was given and this was a not significant difference on the data before correction on socioeconomic factors. They did see that a subgroup, people older than 75 visiting an anthroposophic GP, were 400 euros cheaper (per quarter) which amounts to a difference of about 30 percent in comparison to a regular GP. This huge difference was only just significant, so either the number of patients in this group was quite low or variance in costs was huge (or both). According to the Society against Quackery Kooreman let them know that the analysis of Renckens and Nienhuys clearly showed that the Society lacked the expertise to judge his research.

Discussion in The European Journal of Health Economics
Renckens and Nienhuys had ended their comments remarking that it might be wiser to wait until the article would have been accepted by a serious scientific journal before diving deep in the statistics used. Until then it was only made available via Kooreman’s website as ‘submitted article’. The article was accepted in 2011 and published in December 2012 in The European Journal of Health Economics (EJHE) as Patients whose GP knows complementary medicine tend to have lower costs and live longer (Open Access). Via a Rbutr-link I became aware of this and the recent comments on the article by Christopher Sampson et al.(or pdf from Sampson’s site). The most important problems they have with it, make come down to:

• The lack of attention K&B give to the problem of multiple testing. If you compare the regular and CAM-groups on many subcategories (by age, kind of medical intervention) you have to be very careful not to identify accidental outliers as significant  differences;
• In the article K&B consider only one outcome measure which directly tells something about the health of the patients: mortality. This is a very important observation, I think. A major weakness of this study is that it doesn’t relate any type of medical intervention (regular or CAM) to the costs. You could have done the same analysis classifying the GPs by their favorite car brand. The outcome might then have been that Mercedes-driving GPs work cheaper, but I doubt that anyone would have taken it seriously;
• On the result in the abstract as stated by K&B (‘Patients whose GP has additional CAM training have 0–30% lower healthcare costs and mortality rates.’) they write: ’This is a misleading and,arguably, disingenuous claim.’;
• Sampson et al. also worry that this result will be used for CAM propaganda: ‘We believe these findings could be widely cited, as is commonplace for ‘supportive’ CAM research.‘ That would be a bad thing because ’The study does not demonstrate that GP-CAM training is associated with either reduced healthcare costs or reduced mortality. Academics have a responsibility to communicate their research carefully and without misinterpretation. Kooreman and Baars have failed to do this.’

K&B replied in the same issue. With regard to multiple testing, they note that they found so many significant results (around 17–21 % of the costs coefficients) that it is very unlikely that these are all false positives. From a statistical viewpoint, quite a peculiar way of dealing with this issue. About that 0-30% interval they tell us that the 30% follows from the 400 euros, which I mentioned before (anthroposophic GP, 75+), as well as the lower mortality they found (significant at 10% level,  for men only). And we should also note that ’0′ is included in both intervals, they tell us. How we should look at these rough ranges is still puzzling me. It seems to me a very unclear way of reporting a ‘result’, but maybe they just want to make clear that they do not claim to have found rock solid evidence. The last bit of their reply is:

Throughout, we have been careful to stress the limitations of our study and the need for further research based on better datasets (and we continue to do so in our contacts related to this study with the media, practitioners, and other parties). The commentators have not provided any insights beyond those already available in our original paper. We hope that their future contributions (and the associated journal space) will take the form of substantive research on this important topic.

In my opinion Sampson et al. put forward some very strong arguments and the answers by K&B are not satisfactory. Do they really think they can bluff their way out doing so?

But where is the ’15 percent’ coming from?
I became quite curious to find the origin of the claim that CAM GPs work 15 percent cheaper. First I contacted Sampson. He hadn’t not yet seen the K&B reply and was quite surprised to hear that the study was already cited all over the Internet by CAM supporters, exactly like they had warned for. Sampson will probably come with another reply to K&B to explain why their answers are not satisfactory.

I was getting the idea that the ’15 percent’ might have been wrongfully derived from the result as stated in the abstract.

Patients whose GP has additional CAM training have 0–30% lower healthcare costs and mortality rates, depending on age groups and type of CAM. The lower costs result from fewer hospital stays and fewer prescription drugs

Maybe some people thought of this rough 0-30% range as a proper confidence interval? Then it would make sense to assume the actual result to be exactly in the middle. From the answers K&B gave, it is already clear that is not a confidence interval, but is it a fair summary of what they did found? The figures regarding the costs come from table 3:

Table 3 from the Kooreman & Baars article. First three columns give corrected differences in costs per category (per quarter, in euros). Yellow highlighting by me.

We find the (approximately) 400 euro reduced costs for patients of 75 years and older at anthroposophic GPS, but we also find a group which is worse off: younger than 25 at homeopathic GPs. That difference amounts to 100 euros more health costs per quarter, which is 47 percent higher than the costs for patients of regular GPs. So wouldn’t it be more fair to give the result of this research as an interval running from -47% to 30%? To get a clearer view on what this figures mean, it would have been nice to know the number of patients in each category, standard deviations and p-values. Therefore I wrote Kooreman an e-mail asking about these things.

Before he answered I had already found out that it had not been misinterpretated by De Volkskrant; the ’15 percent’ was in the press release of Tilburg University. And Baars mentioned it in a radio interview (Hoe?Zo! radio June 9^th 2010, starting from 19.05, in Dutch). Kooreman did not show a great appetite in discussing the issues via mail and mainly pointed out that the press release had stated ‘approximately 15 percent’ not just ’15 percent’. It’s a pity he never sent me the underlying figures of table 3 as I had requested (repeatedly). I let him know that I couldn’t find a responsible way of translating a non significant difference of 7% into ’approximately 15 percent’; but alas, no response after that.
In the end I did found out where this ’15 percent’ originates from. K&B wrote an abstract for a congress on Integrative Medicine in December 2010, where they would be presenting their study. Deadline for abstracts was May 31^st 2010. The press release, following a couple of days later, is probably based on the conclusions therein (emphasis by me):

Conclusions: There is evidence that treatment by GPs who completed additional training in complementary medicine (anthroposophic medicine, homeopathy or acupuncture) results in approximately 15% health care cost reduction.

prof. Peter Kooreman

Misleading?
I found more references made by Kooreman to this study: an opinion article in another newspaper and a book chapter, all in Dutch so I’ll just briefly deal with those here. The ’15 percent’ is not mentioned, but he is playing around with the result in an other way. In the newspaper article he writes: “Recent research shows that GPs who have been trained in acupuncture, antroposophic medicine or homoeopathy after their regular studies, work cheaper than regular GPs, even if you correct in the best way for differences in patients backgrounds. Furthermore their patients live longer.” While in the book it is stated as “Based on data from a Dutch health insurer Kooremans and Baars find clues that … etc.” (emphasis by me).
On several websites the K&B article is referenced by a slightly different title: ‘Patients Whose GP Knows Complementary Medicine Have Lower Costs and Live Longer’ in stead of ‘Patients whose GP knows complementary medicine tend to have lower costs and live longer’. I first thought these websites might present it deliberately misleading (mainly because Ullman does so as well), but then I noticed that it was actually the original title. It was changed into the weaker version when published in EJHE.
Kooreman replaced the original submitted version on his website with the one published in EJHE. In doing so all the weblinks point to the most recent version, although in many cases now using the wrong title. Even Kooreman forgot to change the title in his curriculum vitae (up to today at least). The different versions can also be found via the Tilburg University Repository (from which we can see it had already been published before in a German journal on acupuncture).

Conclusions
To me it looks like Kooreman and Baars spiced up the actual result of their research from a realistic ‘we didn’t find a significant difference’ to something that suited them better. It might be a bit annoying that people like Renckens, Nienhuys and myself question the result, but those skeptic sounds can be set away as the biased criticism you can expect from self proclaimed fighters of quackery. It surprises me though, that they try to deal with criticism from scientific peers in similar fashion.
In contrast to this behavior Kooreman wrote to me that he had always insisted on careful interpretation of their findings. In a speech at a recent congress in The Netherlands (on Integrative Medicine) he mentioned that many CAM supporting websites had brought the result as if it proved that CAM treatment is cost effective, while it only can say something about the cost efficiency of CAM GPs, without being clear whether this is actually caused by using CAM treatments instead of regular care. On top of that, he pointed to the reports about the Excellence in Integrative Medicine Research Award Baars and he received at the European Congress for Integrative Medicine for their study. Those reports omit to mention that this prize was sponsored by Heel, a big homeopathic pharma company in Germany. And according to Kooreman, it would have been better to mention this before other parties find out.
This sounds very reasonable and correct, but nowhere does he acknowledge that the main result of the study itself is being communicated in misleading ways, also by the authors themselves. The ’15 percent’ even popped up again in the press release announcing that specific congress.

Tweet