Making the best decisions in our ever-changing world requires learning. Evolution rewards those of us who rapidly learn from wins and from mistakes. So how does the brain learn to make good decisions in the face of uncertainty?
A new paper led by Dr Armin Lak (now based in our department at Oxford) and colleagues at UCL suggests learning mainly happens when we are not confident about the outcome, and illustrates the brain signals that underlie this.
Several studies have explored how humans make decisions. Some have focused on sensory decisions; how decisions are made when the stimulus is either clear or unclear, affecting confidence in the outcome. Others have focused on reward value; how economic decisions are made based on the value of rewards. This new study from Lak et al brings these two considerations together by changing both the strength of sensory signals as well as the reward value, and does so for the first time in mice, allowing for harnessing advanced tools to examine brain activity during decision making.
The test aimed to assess how the mice chose when stimuli were hard or easy to detect and also when the rewards they predicted differed in values. The mice viewed stimuli on a monitor; some were clearly visible and others were hard to see. In addition, some items were of greater value than others, changing the level of reward the animals received if they choose the stimulus. An item with high reward may be hard to detect, whereas an item with low reward may be easy to see. Therefore, the tasks combined different forms of uncertainty – whether the mouse can confidently see the stimuli and whether each reward carries a lower or higher value.
The mice performed this task very well: when the stimulus was easy and the reward was guaranteed, they chose it. On the other hand, when the stimulus was hard to detect and hence they were unsure about their choice, they opted to try their chance for getting the large reward. In particular, they showed the strongest learning when they were not confident of what the outcome of the choice would be.
If they are not confident, and end up making the correct choice and receiving the reward, that's when they learn most and change their future decisions. Our computer models were showing this, and surprisingly, animals showed the very same phenomenon. So, it is best to monitor your decision confidence and learn from unsure decisions. You learn more about how to make decisions, and in the long run, you will win big and often. - Dr Lak
The researchers were also examining the physiology in specific regions of the brain during task performance. Their experiments recorded the neural activity in both the frontal cortex and the dopamine system - the cognitively complex regions associated with learning, valuation, personality and emotions. While it has been known that these regions of the brain work together to guide economic decisions, it is not known how they combine our experience of reward value and how confident we are in achieving it. As a consequence of the experiments, researchers were able to see which of these regions were more involved in each stage of the decision-making process, suggesting they do in fact have different parts to play in this process.
The team closely observed and recorded the neurons both when the mice perceived the stimulus and at the action of obtaining the reward. “The neural activity recorded precisely showed a combination of the reward value the mouse is expecting and how confident the animal is of receiving the reward.” (Dr Lak). Both frontal cortex neurons and dopamine neurons showed this activity. But do these neurons actually have different roles in the decision-making process?
In a second set of experiments, the team attempted to address this question by manipulating the neurons using optogenetics, a technique that uses light to control the activity of nerve cells. Researchers turned neurons on and off at each moment of the decision-making process, to ascertain which were most important at the time of stimulus and at the time of outcome. Activating either the dopamine neurons or the frontal cortex neurons achieved different behavioural effects. Activating the frontal cortex affected decisions only if the neurons were manipulated at the time of perceiving the stimulus. Activating the dopamine system only changed decisions if the neurons were manipulated at the time of receiving the reward. The results suggested that both the frontal cortex and dopamine system are important for learning but in a very different manner. The former by predicting the future outcome, and the latter by evaluating the prediction at the time of reward.
For the first time, this study has compared the value of a reward and the confidence in a reward during decision making in mice, in order to shed light on how we learn to make the best decisions under uncertainty. It has also presented strong evidence to suggest which specific regions of the brain are most involved in controlling each stage of the decision-making process.
The full paper, Dopaminergic and Prefrontal Basis of Learning from Sensory Confidence and Reward Value, published in Neuron, is available to read here.