Long associated with reward and pleasure, dopamine and serotonin may also be involved in general cognition, shaping how people perceive the world and act on those perceptions, a new study finds.

For the first time, researchers have continuously and simultaneously monitored the two neuromodulators in the human brain. The results, published October 12 in Neuron, offer new opportunities to test hypotheses previously studied mostly in animal models. 

“This study isn’t just measuring dopamine and serotonin; it’s building upon the deep foundation looking at neural mechanisms for perceptual decisions in animals and humans” and linking the findings of these studies together, Tim Hanks, a neuroscientist at the University of California, Davis, who was not involved in the study, tells The Scientist. “There’s a growing recognition that [dopamine and serotonin] have more refined and nuanced roles than what may have once been believed, and this study really makes that case clear in human decision-making.”

Both neuromodulators have been heavily studied in animals, but animals require training to carry out decision-making tasks—training that often comes with a reward. As a result, it can be difficult to tease apart the decision-making from the reinforcement they receive in return. “Animals are a limited model of the rich thoughts and behaviors that we see in humans,” says Dan Bang, a neuroscientist at University College London and the lead author of the new study. 

To study dopamine and serotonin signaling in humans, the team recruited five volunteers who were set to undergo brain surgery to treat either Parkinson’s or essential tremors and agreed to have their neurochemicals monitored during the procedure. Surgeons keep patients awake during the operation and use probes to measure brain activity for safety. Bang’s group was able to insert its own microelectrode into the caudate nucleus of four of the volunteers and the putamen of the fifth. Both structures are regions of the striatum and are involved in movement, learning, and reward.

As they underwent surgery, each participant completed a modified version of a common visual task called the random dot motion paradigm. In each round of the task, a person was shown a cloud of flickering dots moving across a screen. Some dots moved together in the same direction, while the rest moved randomly; the proportions undergoing each type of movement determined the difficulty of the task. In the standard test, the dots disappear, and the subject must indicate whether they had, on average, been moving toward the left or the right. In the amended protocol, participants were instead shown a random angle after the dots had disappeared and had to decide whether the dots had been moving to the left or right of that angle. 

In this way, the scientists were able to vary the difficulty and uncertainty of a person’s perception by changing both the number of dots moving in synchrony and how close their path of motion came to the randomly selected reference angle. After making their choice, participants rated how sure they were of their decision. 

A microelectrode continuously measured both dopamine and serotonin levels in the caudate nucleus or putamen, taking 10 measurements each second. Scientists have never before been able to monitor these neurotransmitters at such biologically relevant speeds in humans. Less-invasive methods such as PET scanning or fMRI typically take only one measurement per minute. 

Within the caudate nucleus, serotonin levels were linked to uncertainty around perceptions in three of the four participants. When the task was more difficult and the outcome uncertain, as estimated by the task variables and the participants’ self-reported uncertainty about their decisions, serotonin levels began to drop shortly after the dots appeared on the screen. When the task was easier, serotonin spiked. In some previous human and animal studies, dopamine has had the opposite relationship with serotonin, and therefore with uncertainty, but in the new study, variations in the caudate nucleus’s dopamine levels did not track consistently with task uncertainty. 

In the putamen, however, the team did find strong evidence in support of opposing roles for  dopamine and serotonin in relation to action, as evidenced by the time it took participants to make their choice about the direction of the dots. Both an increase in dopamine and a corresponding decrease in serotonin were associated with the subject’s choice to act, and both the change in neuromodulator levels and the decision itself happened more quickly when the task was easier and less uncertain.

Taken together, these findings suggest that beyond their role as reward chemicals, dopamine and serotonin may contribute to cognition more generally, linking “how we perceive the world and how we then go on to make decisions,” says Ken Kishida, a neuroscientist at the Wake Forest School of Medicine and a coauthor on the study. “This is definitely putting the importance of dopamine and serotonin into a new light.”

Even though this is a new finding in humans, it dovetails with what some researchers have begun to find in animals, says Armin Lak, a neuroscientist at the University of Oxford who was not involved in the study. In his own work, Lak has found links between dopamine and perception in rodents. “It’s really nice, for those of us working in neuroscience, to see this spectrum of studies all the way from animals to human volunteers.”

The biggest limitation of the new study, Lak adds, is the small sample size. Some of the team’s results, such as their data on the putamen, stem from only a single person. Kishida also points out that while dopamine levels varied more between people than did serotonin levels, that may be because some of the patients had Parkinson’s, a disease caused by dysregulated dopamine signaling.

Moving forward, the team plans to refine its microelectrode to recognize additional neurochemicals, such as norepinephrine. Having shown that the responses of neuromodulators can differ by brain region, they would also like to expand to include the cortex, amygdala, and hippocampus. 

Better understanding of how dopamine and serotonin interact and their roles in different parts of the brain will also have important implications for treating neuropsychiatric disorders such as Parkinson’s and depression, says Hanks. Many treatments target these two modulators, but they do so across the entire brain and over longer time scales, so more knowledge could lead to more targeted and effective therapies.

“Because these neuromodulators have complex roles that depend on brain region, some will see it as a challenge, because it means that we can’t just use a medication that’s affecting [the brain] diffusely,” Hanks tells The Scientist. “But at the same time, I would argue that this represents a tremendous opportunity to make [therapies] even more effective.”

D. Bang et al., “Sub-second dopamine and serotonin signaling in human striatum during perceptual decision-making,” Neuron, 108:1–12, 2020.

Clarification (October 16): The title of this article has been updated to better reflect the study’s findings.