seductive
This link has been bookmarked by 44 people . It was first bookmarked on 09 Aug 2008, by dhtobey Tobey.
-
10 Oct 09
-
14 Feb 09
-
What wasn't yet clear was that dopamine is also a profoundly important source of information. It doesn't merely let us take pleasure in the world; it allows us to understand the world.
-
But after years of searching in vain, Schultz started to notice something odd about these dopamine neurons: They began to fire just before the monkeys got a reward.
-
His experiments observed a simple protocol: He played a loud tone, waited for a few seconds, and then squirted a few drops of apple juice into the mouth of a monkey. While the experiment was unfolding, Schultz was probing the dopamine-rich areas of the monkey brain with a needle that monitored the electrical activity inside individual cells. At first the dopamine neurons didn't fire until the juice was delivered; they were responding to the actual reward. However, once the animal learned that the tone preceded the arrival of juice — this requires only a few trials — the same neurons began firing at the sound of the tone instead of the sweet reward. And then eventually, if the tone kept on predicting the juice, the cells went silent. They stopped firing altogether.
-
a model called temporal difference reinforcement learning (TDRL)
-
Computer scientists Rich Sutton and Andrew Barto, who both worked on models of artificial intelligence, had pioneered the model. Sutton and Barto wanted to develop a "neuronlike" program that could learn simple rules and behaviors in order to achieve a goal. The basic premise is straightforward: The software makes predictions about what will happen — about how a checkers game will unfold for example — and then compares these predictions with what actually happens. If the prediction is right, that series of predictions gets reinforced. However, if the prediction is wrong, the software reevaluates its representation of the game.
-
We wrote that paper 11 times," Montague says. "It got bounced from every journal
-
The crucial feature of these dopamine neurons, say Montague and Dayan, is that they are more concerned with predicting rewards than with the rewards themselves. Once the cells memorize the simple pattern — a loud tone predicts the arrival of juice — they become exquisitely sensitive to variations on the pattern. If the cellular predictions proved correct and the primates experienced a surge of dopamine, the prediction was reinforced. However, if the pattern was violated — if the tone sounded but the juice never arrived — then the monkey's dopamine neurons abruptly decreased their firing rate. This is known as the "prediction error signal." The monkey got upset because its predictions of juice were wrong.
-
A Neural Substrate of Prediction and Reward
-
The paper has since been cited more than 1,200 times, and it remains the definitive explanation of how the brain parses reality into a set of accurate expectations, which are measured out in short bursts of dopamine. A crucial part of the cellular code had been cracked.
-
"One of the distinguishing traits of human beings is that we chase ideas, not just primary rewards,"
-
Montague's insight, however, was that ideas are just like apple juice. From the perspective of the brain, an abstraction can be just as rewarding as the tone that predicts the reward. Evolution essentially bootstrapped our penchant for intellectual concepts to the same reward circuits that govern our animal appetites.
-
According to Montague, the reason abstract thoughts can be so rewarding, is that the brain relies on a common neural currency for evaluating alternatives. "It's clear that you need some way to compare your options, even if your options come from very different categories," he says. By representing everything in terms of neuron firing rates, the human brain is able to choose the abstract thought over the visceral reward, as long as the abstraction excites our cells more than apple juice. That's what makes ideas so powerful: No matter how esoteric or ethereal they get, they are ultimately fed back into the same system that makes us want sex and sugar. As Montague notes, "You don't have to dig very far before it all comes back to your loins."
-
But what exactly is the caudate computing? How do we decide whom to trust with our money? And why do we sometimes decide to stop trusting those people? It turned out that the caudate worked just like the reward cells in the monkey brain. At first the caudate didn't get excited until the subjects actually trusted one another and garnered their separate rewards. But over time this brain area started to expect trust, so that it fired long before the reward actually arrived. Of course, if the bond was broken — if someone cheated and stole money — then the neurons stopped firing; social assumptions were proven wrong. (Montague is currently repeating this experiment with a collaborating lab in China so that he can detect the influence of culture on social interactions.) The point, he says, is that people were using this TDRL strategy — a strategy that evolved to help animals find caloric rewards — to model another mind. Instead of predicting the arrival of juice, the neurons were predicting the behavior of someone else's brain.
-
Although dopamine neurons excelled at measuring the mismatch between their predictions of rewards and those that actually arrived — these errors provided the input for learning — they'd learn much quicker if they could also incorporate the prediction errors of others. Montague called this a "fictive error learning signal," since the brain would be benefiting from hypothetical scenarios: "You'd be updating your expectations based not just on what happened, but on what might have happened if you'd done something differently." As Montague saw it, this would be a very valuable addition to our cognitive software.
-
The more we regret a decision, the more likely we are to do something different the next time around. As a result investors in the experiment naturally adapted their investments to the ebb and flow of the market. When markets were booming, as in the Nasdaq bubble of the late 1990s, people perpetually increased their investments.
-
Last year Montague decided to replicate his stock market study with a large group of chronic smokers. It turned out that smokers were perfectly able to compute a "what if" learning signal, which allowed them to experience regret. Like nonsmokers they realized that they should have invested differently in the stock market. Unfortunately, this signal had no impact on their decision making, which led them to make significantly less money during the investing game. According to Montague, this data helps explain why smokers continue to smoke even when they regret it. Although their dopamine neurons correctly compute the rewards of an extended life versus a hit of nicotine — they are, in essence, asking themselves, "What if I don't smoke this cigarette?" — their brain doesn't process the result. That feeling of regret is conveniently ignored. They just keep on lighting up.
-
-
28 Nov 08
-
23 Nov 08
-
30 Oct 08
Ilan GordonThere is so much good stuff in this article.
science evolution for:ryanholiday for:celestethespectacular for:tuckermax
-
15 Oct 08
-
25 Sep 08
-
You can't really understand the brain until you understand how these social behaviors happen, or what happens when they go haywire."
-
What began as an investigation into a single neurotransmitter has morphed into an exploration of the social brain:
-
What began as an investigation into a single neurotransmitter has morphed into an exploration of the social brain:
-
You can't really understand the brain until you understand how these social behaviors happen, or what happens when they go haywire."
-
" I get bored rather easily," he says — and his lab is constantly shifting direction, transitioning from dopamine to neuroeconomics to social neuroscience.
-
He buzzes with ideas for new e
-
st a collection of discrete individuals," he says. "It's something else entirely. You would do things in a group that you would never do by yourself
-
If I'd listened to the naysayers," he says, "I'd still be studying honeybees."
-
-
15 Sep 08
-
22 Aug 08
-
21 Aug 08
-
15 Aug 08
review from Mind Hacks - "One thing I notice a little of in the quotes from Montague, which is incredibly common in discussion of dopamine and reward, is a kind of 'reward system dogma'." http://www.mindhacks.com/blog/2008/08/the_best_is_yet_to_
brain anatomy neuro behavior learning dopamine addiction fmri computer application prediction science_is_a_method model theory publishing risk social_network
-
That’s when Montague discovered the powers of dopamine, a neurotransmitter in the brain. His research on the singular chemical has drawn tantalizing connections between the peculiar habits of our neurons and the peculiar habits of real people, so that the various levels of psychological description — the macro and the micro, the behavioral and the cellular — no longer seem so distinct. What began as an investigation into a single neurotransmitter has morphed into an exploration of the social brain: Montague has pioneered research that allows him to link the obscure details of the cortex to all sorts of important phenomena, from stock market bubbles to cigarette addiction to the development of trust. “We are profoundly social animals,” he says. “You can’t really understand the brain until you understand how these social behaviors happen, or what happens when they go haywire.”
-
-
nucleus accumbens (NAcc), a part of the brain dense with dopamine neurons and involved with the processing of pleasurable rewards, like food and sex
-
temporal difference reinforcement learning (TDRL). Computer scientists Rich Sutton and Andrew Barto, who both worked on models of artificial intelligence, had pioneered the model. Sutton and Barto wanted to develop a “neuronlike” program that could learn simple rules and behaviors in order to achieve a goal. The basic premise is straightforward: The software makes predictions about what will happen
-
“The only reason we could see it so clearly,” Montague says, “is because we came at it from this theoretical angle. If you were an experimentalist seeing this data, it would have been extremely confusing.
-
It got bounced from every journal. I came this close to leaving the field. I realized that neuroscience just wasn’t ready for theory, even if the theory made sense.”
-
The crucial feature of these dopamine neurons, say Montague and Dayan, is that they are more concerned with predicting rewards than with the rewards themselves.
-
Our knowledge, in other words, emerges from our cellular mistakes. The brain learns how to be right by focusing on what it got wrong.
-
“A Neural Substrate of Prediction and Reward.” The paper has since been cited more than 1,200 times, and it remains the definitive explanation of how the brain parses reality into a set of accurate expectations, which are measured out in short bursts of dopamine.
-
Evolution essentially bootstrapped our penchant for intellectual concepts to the same reward circuits that govern our animal appetites. “The guy who’s on hunger strike for some political cause is still relying on his midbrain dopamine neurons, just like a monkey getting a treat,” Montague says. “His brain simply values the cause more than it values dinner.”
-
By representing everything in terms of neuron firing rates, the human brain is able to choose the abstract thought over the visceral reward, as long as the abstraction excites our cells more than apple juice. That’s what makes ideas so powerful: No matter how esoteric or ethereal they get, they are ultimately fed back into the same system that makes us want sex and sugar
-
Although dopamine neurons excelled at measuring the mismatch between their predictions of rewards and those that actually arrived — these errors provided the input for learning — they’d learn much quicker if they could also incorporate the prediction errors of others. Montague called this a “fictive error learning signal,” since the brain would be benefiting from hypothetical scenarios: “You’d be updating your expectations based not just on what happened, but on what might have happened if you’d done something differently.”
-
addiction is a disease of valuation: Dopamine cells have lost track of what’s really important.
-
The purpose of this particular experiment is to see how “one bad apple” can lead perfect strangers to also act badly. While Montague isn’t ready to share the results — he’s still gathering data — what he’s found so far is, he says, “stunning, shocking even…. For me the lesson has been that people act very badly in groups. And now we can see why.”
-
-
14 Aug 08
Pablo StafforiniNew research is linking dopamine to complex social phenomena and changing neuroscience in the process.
shared_old new-import-delicious neuroscience dopamine shared_with_delicious
-
13 Aug 08
-
12 Aug 08
urban sheepNew research is linking dopamine to complex social phenomena and changing neuroscience in the process.
-
11 Aug 08
chinesejapaneseNew research is linking dopamine to complex social phenomena and changing neuroscience in the process.
dopamine article science neuroscience brain imaging MRI fMRI neuron neurotransmitter delicious import from ImportDelicious 2016 05 22
-
10 Aug 08
Rudy Garns"New research is linking dopamine to complex social phenomena and changing neuroscience in the process." (Seed)
-
The importance of dopamine was discovered by accident. In 1954 James Olds and Peter Milner
-
At first the dopamine neurons didn’t fire until the juice was delivered; they were responding to the actual reward. However, once the animal learned that the tone preceded the arrival of juice — this requires only a few trials — the same neurons began firing at the sound of the tone instead of the sweet reward. And then eventually, if the tone kept on predicting the juice, the cells went silent. They stopped firing altogether.
-
The software makes predictions about what will happen — about how a checkers game will unfold for example — and then compares these predictions with what actually happens. If the prediction is right, that series of predictions gets reinforced. However, if the prediction is wrong, the software reevaluates its representation of the game.
-
The crucial feature of these dopamine neurons, say Montague and Dayan, is that they are more concerned with predicting rewards than with the rewards themselves. Once the cells memorize the simple pattern — a loud tone predicts the arrival of juice — they become exquisitely sensitive to variations on the pattern. If the cellular predictions proved correct and the primates experienced a surge of dopamine, the prediction was reinforced. However, if the pattern was violated — if the tone sounded but the juice never arrived — then the monkey’s dopamine neurons abruptly decreased their firing rate. This is known as the “prediction error signal.” The monkey got upset because its predictions of juice were wrong.
-
According to Montague, the reason abstract thoughts can be so rewarding, is that the brain relies on a common neural currency for evaluating alternatives. “It’s clear that you need some way to compare your options, even if your options come from very different categories,” he says. By representing everything in terms of neuron firing rates, the human brain is able to choose the abstract thought over the visceral reward, as long as the abstraction excites our cells more than apple juice. That’s what makes ideas so powerful: No matter how esoteric or ethereal they get, they are ultimately fed back into the same system that makes us want sex and sugar
-
-
09 Aug 08
Jack ParkWhile Montague isn’t ready to share the results — he’s still gathering data — what he’s found so far is, he says, “stunning, shocking even…. For me the lesson has been that people act very badly in groups. And now we can see why.”
-
-
The centerpiece of the lab, however, isn’t visible. Montague has access to five state-of-the-art fMRI machines, which occupy the perimeter of the room. Each of the scanners is hidden behind a thick concrete wall, but when the scanners are in use — and they almost always are — the entire lab seems to quiver with a high-pitched buzz. Montague, though, doesn’t seem to mind. “It’s not the prettiest sound,” he admits. “But it’s the sound of data.”
-
The human brain, however, is an incredibly well-encrypted machine. For starters it’s hard to even know what the code is: Our cells express themselves in so many different ways.
-
There’s the language of chemistry, with brain activity measured in squirts of neurotransmitter and kinase enzymes. And then there’s the electrical conversation of the cortex, so that each neuron acts like a biological transistor, emitting a binary code of action potentials.
-
“We are profoundly social animals,” he says. “You can’t really understand the brain until you understand how these social behaviors happen, or what happens when they go haywire.”
-
But that view of the neurotransmitter was vastly oversimplified. What wasn’t yet clear was that dopamine is also a profoundly important source of information. It doesn’t merely let us take pleasure in the world; it allows us to understand the world.
-
He knew that the cells were learning something about the juice and the tone, but he couldn’t figure out how they were learning it. The code remained impenetrable.
-
“The only reason we could see it so clearly,” Montague says, “is because we came at it from this theoretical angle. If you were an experimentalist seeing this data, it would have been extremely confusing. What the hell are these cells doing? Why aren’t they just responding to the juice?” That same day Montague and Dayan began writing a technical paper that laid out their insight, explaining how these neurons were making precise predictions about future rewards. But the paper — an awkward mix of Schultz’s dopamine recordings and equations borrowed from computer science — went nowhere.
-
“We had to drag the experimentalists kicking and screaming,” Montague says. “They just didn’t understand how these funny-looking equations could explain their data. They told us, ‘We need more data.’ But what’s the point of data if you can’t figure it out?”
-
Our knowledge, in other words, emerges from our cellular mistakes. The brain learns how to be right by focusing on what it got wrong.
-
ideas are just like apple juice. From the perspective of the brain, an abstraction can be just as rewarding as the tone that predicts the reward
-
“The most unrealistic element [of fMRI experiments] is that we could only study the brain by itself,” Montague says. “But when are brains ever by themselves?” And so Montague pioneered a technique known as hyper-scanning, allowing subjects in different fMRI machines to interact in real tim
-
“The most unrealistic element [of fMRI experiments] is that we could only study the brain by itself,” Montague says. “But when are brains ever by themselves?” And so Montague pioneered a technique known as hyper-scanning, allowing subjects in different fMRI machines to interact in real time.
-
At first the caudate didn’t get excited until the subjects actually trusted one another and garnered their separate rewards. But over time this brain area started to expect trust, so that it fired long before the reward actually arrived. Of course, if the bond was broken — if someone cheated and stole money — then the neurons stopped firing; social assumptions were proven wrong.
-
“I just assumed that evolution would use this approach, because it’s too good an idea not to use,” he says.
-
addiction is a disease of valuation
-
Although their dopamine neurons correctly compute the rewards of an extended life versus a hit of nicotine — they are, in essence, asking themselves, “What if I don’t smoke this cigarette?” — their brain doesn’t process the result.
-
what he’s found so far is, he says, “stunning, shocking even…. For me the lesson has been that people act very badly in groups. And now we can see why.”
-
-
-
New research is linking dopamine to complex social phenomena and changing neuroscience in the process
-
Read Montague
-
Read Montague
-
is director of the Human Neuroimaging Lab at Baylor College of Medicine in downtown Houston.
-
The importance of dopamine was discovered by accident. In 1954 James Olds and Peter Milner, two neuroscientists at McGill University, decided to implant an electrode deep into the center of a rat’s brain. The precise placement of the electrode was largely happenstance: At the time the geography of the mind remained a mystery. But Olds and Milner got lucky. They inserted the needle right next to the nucleus accumbens (NAcc), a part of the brain dense with dopamine neurons and involved with the processing of pleasurable rewards, like food and sex.
Olds and Milner quickly discovered that too much pleasure can be fatal. After they ran a small current into the wire, so that the NAcc was continually excited, the scientists noticed that the rodents lost interest in everything else. They stopped eating and drinking. All courtship behavior ceased. The rats would just cower in the corner of their cage, transfixed by their bliss. Within days all of the animals had perished. They had died of thirst.
-
What wasn’t yet clear was that dopamine is also a profoundly important source of information. It doesn’t merely let us take pleasure in the world; it allows us to understand the world.
The first experimental insight into this aspect of the dopamine system came from the pioneering research of Wolfram Schultz, a neuroscientist at Cambridge University.
-
Schultz started to notice something odd about these dopamine neurons: They began to fire just before the monkeys got a reward. (Originally, the reward was a way of getting the monkeys to move.) “
-
After hundreds of experimental trials, Schultz began to believe his own data: He realized that he had found, by accident, the reward mechanism at work in the primate brain.
-
He played a loud tone, waited for a few seconds, and then squirted a few drops of apple juice into the mouth of a monkey. While the experiment was unfolding, Schultz was probing the dopamine-rich areas of the monkey brain with a needle that monitored the electrical activity inside individual cells. At first the dopamine neurons didn’t fire until the juice was delivered; they were responding to the actual reward. However, once the animal learned that the tone preceded the arrival of juice — this requires only a few trials — the same neurons began firing at the sound of the tone instead of the sweet reward. And then eventually, if the tone kept on predicting the juice, the cells went silent. They stopped firing altogether.
-
Peter Dayan, a colleague of Montague’s at Salk, had introduced him to a model called temporal difference reinforcement learning (TDRL). Computer scientists Rich Sutton and Andrew Barto, who both worked on models of artificial intelligence, had pioneered the model. Sutton and Barto wanted to develop a “neuronlike” program that could learn simple rules and behaviors in order to achieve a goal. The basic premise is straightforward: The software makes predictions about what will happen — about how a checkers game will unfold for example — and then compares these predictions with what actually happens. If the prediction is right, that series of predictions gets reinforced. However, if the prediction is wrong, the software reevaluates its representation of the game.
-
The crucial feature of these dopamine neurons, say Montague and Dayan, is that they are more concerned with predicting rewards than with the rewards themselves. Once the cells memorize the simple pattern — a loud tone predicts the arrival of juice — they become exquisitely sensitive to variations on the pattern. If the cellular predictions proved correct and the primates experienced a surge of dopamine, the prediction was reinforced. However, if the pattern was violated — if the tone sounded but the juice never arrived — then the monkey’s dopamine neurons abruptly decreased their firing rate. This is known as the “prediction error signal.” The monkey got upset because its predictions of juice were wrong.
-
What’s interesting about this system is that it’s all about expectation.
-
“You learn how the world works by focusing on the prediction errors, on the events that you didn’t expect.” Our knowledge, in other words, emerges from our cellular mistakes. The brain learns how to be right by focusing on what it got wrong.
-
Montague continued to work on dopamine. In 1997 he published a Science paper with Dayan and Schultz whose short title was audaciously grand: “A Neural Substrate of Prediction and Reward.” The paper has since been cited more than 1,200 times, and it remains the definitive explanation of how the brain parses reality into a set of accurate expectations, which are measured out in short bursts of dopamine.
-
Montague’s insight, however, was that ideas are just like apple juice. From the perspective of the brain, an abstraction can be just as rewarding as the tone that predicts the reward. Evolution essentially bootstrapped our penchant for intellectual concepts to the same reward circuits that govern our animal appetites.
-
According to Montague, the reason abstract thoughts can be so rewarding, is that the brain relies on a common neural currency for evaluating alternatives.
-
In recent years Montague has shown how this basic computational mechanism is a fundamental feature of the human mind. Consider a paper on the neural foundations of trust, recently published in Science. The experiment was born out of Montague’s frustration with the limitations of conventional fMRI. “The most unrealistic element [of fMRI experiments] is that we could only study the brain by itself,” Montague says. “But when are brains ever by themselves?” And so Montague pioneered a technique known as hyper-scanning, allowing subjects in different fMRI machines to interact in real time. His experiment revolved around a simple economic game in which getting the maximum reward required the strangers to trust one another. However, if one of the players grew especially selfish, he or she could always steal from the pot and erase the tenuous bond of trust. By monitoring the players’ brains, Montague was able to predict whether or not someone would steal money several seconds before the theft actually occurred. The secret was a cortical area known as the caudate nucleus, which closely tracked the payouts from the other player. Montague noticed that whenever the caudate exhibited reduced activity, trust tended to break down.
-
At first the caudate didn’t get excited until the subjects actually trusted one another and garnered their separate rewards. But over time this brain area started to expect trust, so that it fired long before the reward actually arrived. Of course, if the bond was broken — if someone cheated and stole money — then the neurons stopped firing; social assumptions were proven wrong.
-
Although dopamine neurons excelled at measuring the mismatch between their predictions of rewards and those that actually arrived — these errors provided the input for learning — they’d learn much quicker if they could also incorporate the prediction errors of others. Montague called this a “fictive error learning signal,” since the brain would be benefiting from hypothetical scenarios: “You’d be updating your expectations based not just on what happened, but on what might have happened if you’d done something differently.”
-
The question, of course, is how to find this “what if” signal in the brain. Montague’s clever solution was to use the stock market. After all, Wall Street investors are constantly comparing their actual returns against the returns that might have been, if only they’d sold their shares before the crash or bought Google stock when the company first went public.
-
One interesting twist was that instead of using random simulations of the stock market, Montague relied on distillations of data from famous historical markets. Montague had people “play” the Dow of 1929, the Nasdaq of 1998, and the S&P 500 of 1987, so the neural responses of investors reflected real-life bubbles and crashes.
The scientists immediately discovered a strong neural signal that drove many of the investment decisions. The signal was fictive learning. Take, for example, this situation. A player has decided to wager 10 percent of her total portfolio in the market, which is a rather small bet. Then she watches as the market rises dramatically in value. At this point, the regret signal in the brain — a swell of activity in the ventral caudate, a reward area rich in dopamine neurons — lights up. While people enjoy their earnings, their brain is fixated on the profits they missed, figuring out the difference between the actual return and the best return “that could have been.” The more we regret a decision, the more likely we are to do something different the next time around. As a result investors in the experiment naturally adapted their investments to the ebb and flow of the market. When markets were booming, as in the Nasdaq bubble of the late 1990s, people perpetually increased their investments.
-
But fictive learning isn’t always adaptive. Montague argues that these computational signals are also a main cause of financial bubbles.
-
Montague’s experiments take advantage of his unique fMRI setup. He has four people negotiate with one another as they decide how much to offer someone else during an investing game. While the group is bickering, Montague is monitoring the brain activity of everyone involved. He’s also infiltrated the group with a computer player that has been programmed to act just like a person with borderline personality disorder. The purpose of this particular experiment is to see how “one bad apple” can lead perfect strangers to also act badly.
-
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.