Cornell mathematicians are using game theory to model how this competition could be leveraged, so cancer treatment—which also takes a toll on the patient's body—might be administered more sparingly, with maximized effect.

Their paper, "Optimizing Adaptive Cancer Therapy: Dynamic Programming and Evolutionary Game Theory," published April 22 in Proceedings of the Royal Society B: Biological Sciences.

"There are many game theoretic approaches for modeling how humans interact, how biological systems interact, how economic entities interact," said the paper's senior author, Alex Vladimirsky, professor of mathematics in the College of Arts and Sciences. "You could also model interactions between different types of cancer cells, which are competing to proliferate inside the tumor. If you know exactly how they're competing, you can try to leverage this to fight cancer better."

Vladimirsky and the paper's lead author, doctoral student Mark Gluzman, collaborated with oncologist and co-author Jacob Scott of the Cleveland Clinic. They used evolutionary game theory to model the interactions of three subpopulations of lung cancer cells that are differentiated by their relationship to oxygen: glycoltyic cells (GLY), vascular overproducers (VOP) and defectors (DEF).

In this model, previously co-developed by Scott, GLY cells are anaerobic (i.e., they do not require oxygen); VOP and DEF cells both use oxygen, but only VOP cells are willing to expend extra energy to produce a protein that will improve the vasculature and bring more oxygen to the cells.

Vladimirsky likens their competition to a game of rock, paper, scissors in which a million people are vying against each other. If the majority of participants choose to play rock, a greater number of players will be tempted to switch to paper. As the number of people switching to paper increases, fewer people will play rock and many more will shift to playing scissors. As the popularity of scissors grows, rock will become an attractive option again, and so on.

"So you have three populations, three competitive strategies, undergoing these cyclic oscillations," said Vladimirsky, who directs the Center for Applied Mathematics. "Without a drug therapy, the three subtypes of cancer cells may follow similar oscillating trajectories. Administering drugs can be viewed as temporarily changing the rules of the game.

"A natural question is how and when to change the rules to achieve our goals at a minimal cost—both in terms of the time to recovery and the total amount of drugs administered to the patient," he said. "Our main contribution is in computing how to optimally time these periods of drug treatment adaptively. We basically developed a map that shows when to administer drugs based on the current ratio of different subtypes of cancer."

In current clinical practice, cancer patients typically receive chemotherapy at the highest dosage their body can safely tolerate, and the side effects can be harsh. In addition, such a continuous treatment regimen often leads the surviving cancer cells to develop drug resistance, making further therapy far more difficult. The team's paper shows that a well-timed "adaptive" application could potentially lead to a patient's recovery with a greatly reduced amount of drugs.

But Vladimirsky cautions that, as is often the case in mathematical modeling, reality is much messier than theory. Biological interactions are complicated, often random, and can vary from patient to patient.

"Our optimization approach and computational experiments were all based on a particular simplified model of cancer evolution," he said. "In principle, the same ideas should also be applicable to much more detailed, and even patient-specific, models, but we are still a long way from there. We view this paper as a necessary early step on the road to practical use of adaptive, personalized drug-therapy. Our results are a strong argument for incorporating timing optimization into the protocol of future clinical trials."