Most people know: If you got a flu shot this year, next year you’ll need it again.

This is because the virus changes, usually rendering the previous year’s vaccine partly or totally useless. And it’s no secret that the flu vaccine’s effectiveness falls well short of what scientists and public health officials would like to see. Yes, it reduces the severity of influenza infections and prevents thousands of deaths and hospitalizations every year, but nowhere near other recommended vaccines.

But why does the virus change so much every year? Why does the vaccine’s effectiveness vary so much? Here’s the explainer you’ve been waiting for.

The big picture

First, a primer on how vaccines work: Vaccines include pieces of the pathogens—the viruses or bacteria—that cause disease. Some vaccines contain the whole pathogen, but whether it’s the whole thing or just pieces, the pathogen in the vaccine has been killed or modified so it can’t cause illness. (That’s why the flu vaccine can’t give you the flu.) What it can do is get a healthy immune system’s attention.

The body’s immune response to a vaccine resembles a military drill for fighting a specific enemy. When a vaccine enters the body, the immune system recognizes the deactivated pathogen as an intruder. Among other things, the body builds an army of antibodies, proteins in the blood specialized to fight that specific pathogen. Even though the vaccine’s components don’t cause infection, the body is primed for battle nonetheless, with antibodies on standby. If the body ever encounters the real pathogen, the antibody army wipes it out before it causes disease.

But, this process works only if the enemy always looks relatively the same. Measles, pertussis, hepatitis, and other viruses we have vaccines for don’t change from year to year, says Dr. Ruth Karron, an epidemiology professor and director of the Center for Immunization Research at the Johns Hopkins Bloomberg School of Public Health in Baltimore. So the vaccine doesn’t need to change either.

Unfortunately, the influenza virus is a ruthless master of disguise.

“Not only does flu change, but it’s a survival strategy for the virus. Its change is how it makes people so sick,” Dr. Karron explained. That’s because the part of the virus that changes is the very part the immune system targets.

Meet the influenza virus

Like all viruses, the flu virus has one goal: replicate. And it can do that only by hijacking other cells. The virus enters a cell and takes over, shutting off the cell’s antiviral response and then using the cell’s machinery to make copies of itself.

Once the immune system learns what a flu virus looks like—whether from a vaccine or a past infection—it seeks out and destroys flu viruses before they enter more cells. But the virus needs to replicate to keep surviving. If parts of the immune system recognize change slightly, the immune system won’t see the virus.

How the influenza virus mutates

DNA is made of two strands—the double helix you may have heard of—and each strand has building blocks that match in pairs.

The flu virus’s genetic material, called RNA, is similar to DNA but with one crucial difference. RNA viruses don’t have “proofreading capacity,” Dr. Karron said.

“If for some reason the machinery that puts in the building blocks is wrong in DNA and it doesn’t match, it has to be fixed,” Dr. Karron said. But RNA has only one strand.

“In RNA, if there’s a mistake, it doesn’t have to be corrected to survive,” Dr. Karron said. “The viruses play this roulette. It’s not that they know which changes will outwit the human immune system, but they’re replicating very rapidly.”

So rapidly that lots of mistakes—mutations—occur. Hijacked cells release new viruses after just 6 hours.

“If there’s a change in one of those viruses that the human immune system doesn’t recognize so well, that gives that virus an advantage,” Dr. Karron said. Those changes often happen in two proteins, called hemagglutinin (HA) and neuraminidase (NA), the H and the N in flu names like H1N1. These proteins are also the parts of the flu virus that the immune system recognizes; so, viruses with a slightly different HA or NA can escape the immune system’s attention.

It’s evolution at work: Tiny mutations help the virus evade detection, and as it evades detection, it’s more successful at multiplying.

“The vaccines we have target the part of the virus that changes so rapidly, so our vaccines have to evolve along with the virus,” Dr. Karron said.

How big changes in viruses start pandemics

That whole evolutionary process of small changes helping the flu virus escape the immune system is called antigenic drift. Drifts happen within flu seasons and from one to the next. But there’s another much larger, and more sinister, change called antigenic shift.

The flu virus genome has eight separate segments, including one for the HA protein and one for NA. These proteins determine a flu virus’s subtype (such as H1N1 or H3N2), and nearly all subtypes occur in birds, the natural host of the influenza virus. If two subtypes infect the same cell, their gene segments can mix and match, called reassortment. This often happens in pigs and birds.

Humans often catch flu infections from pigs and birds at places, such as state fairs, farms, and live animal markets, but these avian and swine viruses don’t replicate well in humans.

“It has to be readily able to pass from person to person,” Dr. Karron said. “A lot of these animal-human interfaces that happen are one-off.”

But what if a bird virus and a human virus end up in the same cell together? That’s called a superinfection. Now, reassortment becomes possible.

“You can get a hemagglutinin and a neuraminidase protein from the avian virus and the other six from the human viruses,” said Dr. Scott Hensley, an associate professor of microbiology at the University of Pennsylvania in Philadelphia. Now there’s a new, mostly human virus that will replicate easily in human cells, but the avian HA and NA make it look very different from any other known human flu viruses, Dr. Hensley said.

That’s an antigenic shift—reason to worry.

“When that happens, you have a pandemic because the human population doesn’t have immunity against those viruses, so the viruses can spread very quickly,” Dr. Hensley said.

The largest pandemic in history was the Spanish flu of 1918, an H1N1 subtype. It infected one-third of the world’s population and killed an estimated 50 million people, hitting young adults especially hard.

“That circulated in humans for about 40 years and continued to acquire HA and NA mutations that allowed it to infect people already exposed to the 1918 strain,” Dr. Hensley said.

An H2N2 subtype replaced the 1918 H1N1 subtype in 1957, causing a new pandemic. But remember when scientists freaked out about the new H1N1 virus in 2009? They had good reason.

“The H1N1 was a triple reassortment, with the gene segments of three viruses,” Dr. Hensley said. And the HA in the new subtype was very similar to the HA in the 1918 subtype, which had been circulating in pigs all that time.

What this means for your annual flu shot

Considering all the influenza virus’s tricks for infecting humans, it’s remarkable we’ve created a vaccine that works at all. And in fact, the vaccine prevents thousands of deaths every year. Most years, the vaccine cuts your chance of catching the flu in half, but it’s hard to get much better odds than that from it.

“Our vaccine process is sort of always one step behind because of this long production process and trying to update the vaccine to what’s circulating,” Dr. Hensley said. The WHO meets twice a year, once for the Southern Hemisphere and once for the Northern Hemisphere, to examine global flu surveillance and recommend the flu strains they expect will cause the most illness.

“This puts a huge burden on manufacturers,” Dr. Karron said. In the Northern Hemisphere, they learn the WHO recommendations in February and have only about 6 months to make vaccines. They sell it very cheaply, Dr. Karron said, about $10 to 20 a dose, and have to throw away anything unsold at the end of the season. “The flu vaccine is not a blockbuster moneymaker,” she said.

Understanding the flu virus’s trickery explains some of the vaccine’s quirks. Why do some vaccine strains partly protect against nonvaccine strains? The HA and NA may look similar enough to the vaccine strain that antibodies attack it anyway.

Why isn’t there a universal flu vaccine yet? The immune system can’t learn every possible combination of HA and NA. Scientists are trying to develop a vaccine that teaches the immune system to recognize a different part of the virus than HA or NA, but that’s hard to do when the immune system automatically notices those proteins first.

Why does the vaccine work better in some people than in others? Different people’s bodies can respond to different parts of the same virus, Dr. Hensley said. Both genetics and the microbiome, for example, can influence immune response.

“My thought is that the minor antigenic changes happening are much more significant than we appreciate,” Dr. Hensley said. Scientists used to think major antigenic changes happened only every few years. But emerging research suggests even tiny changes occurring each season “are actually very big changes for some individuals,” he said.

His research team also recently discovered that growing flu viruses in eggs to make vaccines, the way many flu vaccines have been made for decades, causes small mutations that might lessen the vaccine’s effectiveness. Viruses that replicate well in chicken cells differ slightly from ones that replicate well in human cells.

Finally, it’s impossible for vaccine strains and circulating strains ever to match perfectly when circulating strains accumulate mutations all season long. But getting the flu vaccine means your body has at least some sense of the viruses out there.

“Even when you have these mismatches, you probably are preventing severe disease,” Dr. Hensley said, “which in my mind is one of the most important goals, preventing someone from dying.”