We Cannot Trust The CDC Estimates Of Flu Vaccine Effectiveness

Authored by Eyal Shahar via The Brownstone Institute,Each year the CDC publishes an estimate of the effectiveness of the flu vaccine in the previous flu season. Recently, the NIH director criticized the test-negative design from which the estimates are derived. He was right. The basic premise of the design is a two-edged sword: on the one hand, restricting the sample to people who sought medical care might reduce confounding by healthcare-seeking behavior; on the other hand, that restriction might add another type of bias - colliding bias - which is not as widely appreciated. The net bias remains unknown.This, however, is not the only shortcoming of test-negative case-control studies of the flu vaccine. In this post, I will expose the shaky results of a large study of the flu vaccine in 2022-2023, when the vaccine was well-matched to the dominant strain. The study was based on the VISION Vaccine Effectiveness Network, one of several networks that collaborate with the CDC. Below are the published results.Abbreviations: ED/UC (emergency department/urgent care); ARI (acute respiratory illness); VE (vaccine effectiveness)Confounding by the Background Risk of InfectionThe risk of infection always varies during the flu season. It was high in October through December 2022 and low in January through March 2023 (Figure).Given a changing risk of infection, a valid comparison of the vaccinated and the unvaccinated requires similar distributions of the two populations over time. This is not the case because vaccination is associated with calendar time (rollout).[The authors show the vaccination status at the time of seeking care, but most people got vaccinated by the end of December, and the percentage of vaccinated people stabilized in January at about 45% of the encounters.]As shown below (Table), the share of the vaccinated population in October through December (47%), a period of high background risk, was lower than the comparable share in the unvaccinated population (61%). Of course, the complementary shares in January through March, a period of low risk, were reversed: 53% versus 39%.In technical terms, vaccinated people accumulated more exposure time when the background risk of infection was low (53%), and unvaccinated people accumulated more exposure time when the background risk was high (61%). Moreover, since the authors excluded events that happened within two weeks of vaccination, those who were vaccinated in the second half of December 2022 contributed events only in January 2023, a time of lower risk. I will return to this analytical decision in the next section.It is simple to grasp the bias (left table below) if we consider an extreme example where no one was injected in the first period and everyone was injected a saline solution at the beginning of the second period (right table). If we compare the rate of infection in the "vaccinated" to the rate in the "unvaccinated," the saline injection would appear effective...This bias was explained in the context of the Covid vaccines during the pandemic and was demonstrated in a study from Ontario, Canada. As far as I know, it was not appreciated in the context of the flu vaccine, where the rollout typically follows the rising wave and is completed around the winter peak.Confounding by time trends in the background risk can be avoided in a cohort design with matching an unvaccinated person to a vaccinated person on the vaccination date (and terminating the observation when the former is vaccinated, if they are).Immortal Time BiasAs I mentioned above, the authors excluded some events. They write:"Events among patients with documented vaccination 0.20. Age and calendar time were modeled as natural cubic spline variables. Also, inverse-propensity-to-be-vaccinated weights (IPVWs) were estimated via generalized boosted regression trees and used in logistic regression models to account for additional imbalances between vaccinated and unvaccinated groups."The authors used unusually complex models that included classical covariates (some in a non-linear form) plus inverse probability of treatment weighting (IPTW) to account for "additional imbalance." It is unclear how "additional imbalance" was detected and which variables were used to compute the weights. No one would have been able replicate their analysis based on this description, even if they were given the dataset.Higher Effectiveness in the Elderly?Another set of questionable results is shown below. It was stated in the abstract.We typically expect lower effectiveness in the elderly because of attenuated immune response with aging. Indeed, we observe a somewhat smaller VE against outpatient encounters in the elderly (41% vs. 45%). Unexpectedly, however, effectiveness against hospitalization is much stronger in the elderly (41% vs. 23%). A trustworthy result?Apparently, the authors noticed the peculiar results, and they argue that "VE by age group cannot be directly compared as most young adults received standard-dose inactivated vaccines and most older adults received enhanced products such as high-dose inactivated or adjuvanted vaccines."Well, this explanation does not explain why the remarkable benefit of "enhanced products" was only observed for hospitalization. Indeed, the authors concede: "Despite most vaccinated older adults receiving enhanced vaccine products, VE [for outpatient encounters] was similar compared to younger adults who mostly received standard-dose inactivated vaccines."In short, these findings remain unexplained. They cannot be trusted.The Outcome of Hospitalized Flu PatientsAlthough not explicitly stated, the authors show a set of results from a nested cohort design.They write:"As a secondary objective, to explore whether patient characteristics and in-hospital outcomes were different between vaccinated and unvaccinated influenza-positive cases, we compared proportions of patients with more severe clinical outcomes by influenza vaccination status stratified by age (18-64, ≥65 years)..."Stated differently, they compared the outcome, including death, of hospitalized flu patients according to their vaccination status. Unfortunately, they did not try to adjust for baseline characteristics, so the inference is limited. I will focus on the case fatality in the elderly (almost 70% of all deaths).In the population of hospitalized elderly flu patients, the vaccinated were older and sicker than the unvaccinated, but the differences were generally small. The fatality of the former was 50% higher: risk ratio = 4.0/2.7 = 1.5. Although no formal adjustment is possible, we can try some hypothetical examples.Suppose vaccination was helpful and the true effect ranged from a risk ratio of 0.5 to 0.75 (50% to 25% effectiveness against death) if an elderly person is hospitalized because of the flu. Then, confounding should have changed a risk ratio of 0.75 (true) or 0.5 (true) to 1.5 (biased), which is a two-fold (1.5/0.75) or a three-fold (1.5/0.5) shift on the ratio scale. If true, that's extreme confounding, perhaps stronger than what might be expected from the reported differences in age and some baseline characteristics. Did the flu vaccine offer any protection in those patients?It is interesting to read the authors' account of these data. They write:"Baseline demographic characteristics and underlying medical conditions were similar across vaccinated and unvaccinated groups within this age strata...The percentage experiencing severe in-hospital clinical outcomes including ICU admission, receipt of IMV, or death, was similar across vaccination groups."They don't even claim any hidden benefit. They assume no meaningful confounding but consider the different fatality (4.0% vs. 2.7%) as "similar." If we follow their reasoning, we may wonder whether vaccination increased the risk of death in these patients.EpilogueI chose to examine one paper closely rather than criticize the generic methodology because this paper demonstrated a series of problems, some of which are shared by other CDC-based studies of the flu vaccine.Since randomized trials will never be conducted, other designs should be sought. I mentioned two possibilities in previous posts: 1) A cohort study with two outcomes: symptomatic infection and asymptomatic infection; 2) Regression discontinuity design. So far, neither showed promising effects of the annual flu shot.Constructed from Figure 1