Many object-related actions can be recognized both by their sound and by their vision. Here we describe a population of neurons in the ventral premotor cortex of the monkey that discharge both when the animal performs a specific action and when it hears or sees the same action performed by another individual. These 'audiovisual mirror neurons' therefore represent actions independently of whether these actions are performed, heard or seen. The magnitude of auditory and visual responses did not differ significantly in half the neurons. A neurometric analysis revealed that based on the response of these neurons, two actions could be discriminated with 97% accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this article Subscribe and saveSpringer+ Basic
€34.99 /Month
Price includes VAT (Germany)
Instant access to the full article PDF.
Similar content being viewed by others Explore related subjectsDiscover the latest articles and news from researchers in related subjects, suggested using machine learning. ReferencesBach M, Bouis D, Fischer B (1983) An accurate and linear infrared oculometer. J Neurosci Methods 9:9–14
Bi G, Poo M (2001) Synaptic modification by correlated activity: Hebb's postulate revisited. Annu Rev Neurosci 24:139–166
Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13:400–404
Cantalupo C, Hopkins WD (2001) Asymmetric Broca's area in great apes. Nature 414:505
Decety J, Grezes J (1999) Neural mechanisms subserving the perception of human actions. Trends Cogn Sci 3:172–178
Fadiga L, Fogassi L, Pavesi G, Rizzolatti G (1995) Motor facilitation during action observation: a magnetic stimulation study. J Neurophysiol 73:2608–2611
Fries W (1984) Cortical projections to the superior colliculus in the macaque monkey: a retrograde study using horseradish peroxidase. J Comp Neurol 230:55–76
Gallese V, Fadiga L, Fogassi L, Rizzolatti G (1996) Action recognition in the premotor cortex. Brain 119:593–609
Grafton ST, Arbib MA, Fadiga L, Rizzolatti G (1996) Localization of grasp representations in humans by positron emission tomography. 2. Observation compared with imagination. Exp Brain Res 112:103–111
Graziano MS, Reiss LA, Gross CG (1999) A neuronal representation of the location of nearby sounds. Nature 397:428–430
Hebb DO (1949) The organization of behavior. Wiley, New York
Keysers C, Xiao DK, Foldiak P, Perrett DI (2001) The speed of sight. J Cogn Neurosci 13:90–101
Kohler E, Keysers C, Umiltà MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848
Newsome WT, Britten KH, Movshon JA (1989) Neuronal correlates of a perceptual decision. Nature 341:52–54
Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends Neurosci 21:188–194
Rizzolatti G, Fadiga L, Gallese V, Fogassi L (1996) Premotor cortex and the recognition of motor actions. Brain Res Cogn Brain Res 3:131–141
Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP (1999) Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci 12:1131–1136
Umiltà MA, Kohler E, Gallese V, Fogassi L, Fadiga L, Keysers C, Rizzolatti G (2001) I know what you are doing. A neurophysiological study. Neuron 31:155–165
Watanabe M (1992) Frontal units of the monkey coding the associative significance of visual and auditory stimuli. Exp Brain Res 89:233–247
The research was funded by a MIURST and an ESF Grant. E.K. was supported by a Fonds fuer Medizinische Forschung der Universitaet Zuerich fellowship, C.K. by an EU Marie-Curie Fellowship. We thank G. Rizzolatti for help and advice on all aspects of the work, G. Pavan, M. Manghi and C. Fossati for their invaluable help in computing and acoustics, S. Rozzi, M. Matelli, and G. Luppino for their anatomical advice and F. Orzi and P. Rossi for caring for the monkeys.
Author information Authors and AffiliationsDepartment of Neuroscience, Università di Parma, Via Volturno 39, 43100, Parma, Italy
C. Keysers, E. Kohler, M. A. Umiltà & V. Gallese
Department of Mathematics, Università di Ferrara, Via Machiavelli, 44100, Ferrara, Italy
L. Nanetti
Department of Psychology, Università di Parma, Bgo Carissimi 10, 43100, Parma, Italy
L. Fogassi
Correspondence to C. Keysers or V. Gallese.
Additional informationThe first two authors contributed equally to the work
Appendix: Box 1: the Receiver Operator Characteristic Analysis Appendix: Box 1: the Receiver Operator Characteristic AnalysisThe fundamental issue behind the Receiver Operator Characteristic (ROC) analysis is simple: the brain contains no photoreceptors, and thus has no direct vision of the outside world—instead, it contains neurons that fire a certain number of times. The brain then has to analyze what happens in the outside world based on the firing of its neurons. The ROC analysis calculates how well an imaginary observer of the firing activity of a given neuron could be at detecting a particular target stimulus. In the case at hand, this ideal observer has to decide on each trial if a particular action (his target action or best action) was performed. He has to take this perceptual decision based on the number of spikes produced by an audiovisual mirror neuron on each individual trial.
The decision rule used by the imaginary observer of the neural activity is simple: he decides on a threshold spike-count value theta, and compares the count x i on a given trial i with this threshold. If x i ≥theta, the observer responds that his target action occurred. If x i <theta, he reports that another action must have been performed. The decision of the observer is then compared with what action was really performed on that trial.
The working of the ROC analysis can be best explained by applying the method to two different neurons: one, which firing has nothing to do with what action the experimenter performed in front of the monkey, and one—an audiovisual mirror neuron—that fires more when its best action was performed.
Figure 2A shows the spike-count distribution for the audiovisual mirror neuron. Note how the histogram when the experimenter performed the best action (top of Fig. 2A) is shifted rightwards compared with the lower histogram when the less effective action was performed (bottom). This shift means that the neuron is more likely to produce large spike-counts when the best action is performed. In the example of Fig. 2A, the observer places his theta at an intermediate value (e.g., 20, dotted vertical line in Fig. 2A). Not knowing what action was really performed by the experimenter, the observer applies his decision rule blindly to both conditions. He reports the best action when the spike count is larger than or equal to theta (i.e., at the right of his threshold) in both cases. In our example, he reports the best action in nine out of the ten best action trials (i.e., when the experimenter really performed the best action), and in two of the ten less effective action trials. The former 9/10 are correct decisions, called 'hits' (black bars in Fig. 2A), and his hit rate is thus 90%. The latter 2/10 are errors, called false alarms (gray bars), and his false alarm rate is thus 20%. The hit and false alarm rate depend not only on the response of the neuron, but also on the theta used by the observer: Placing theta very low (i.e., moving the dotted line leftwards), the observer would always report the best action, and both the hit and false alarm rate would approach 1. Choosing very large theta, he would never report the best action, and both the rates will approach 0. Figure 2B illustrates the relationship between the hit rate and the false alarm rate for this audiovisual mirror neuron as a function of theta. This curve, called the receiver operator characteristic curve, is obtained by testing all the possible theta values, plotting the hit and false alarm rates for each theta, and connecting all the points. For this neuron, this curve is very far away from the dotted diagonal, and the surface under the curve is close to 1 (0.96). What does this large surface mean? Given that the two histograms overlap very little, if the observer starts at a theta=42 (bottom left of Fig. 2B), and reduces theta until 23, the observer only responds with best action for best action trials, i.e., the criterion does not include trials of the lower histogram in Fig. 2A, and thus the hit rate increases without increasing the false alarm rate. Only if the observer reduces theta below 23 will he respond with best action also in trials where the worst action was really performed. The curve thus rises vertically until the 80% hit rate, and then moves almost horizontally towards a false alarm of 100%, covering almost all the surface in the box. This is symptomatic for cases where the neuron accurately discriminates between the two types of actions.
In contrast to the audiovisual mirror neuron, let us now consider an imaginary neuron that does not discriminate between the two actions. Figure 2C illustrates the spike counts for this neuron. Note how much overlap there is between the two histograms. Under such conditions, placing theta at 20 means that the observer will respond with best action 6/10 times when the best action and 5/10 times when the worst action was really performed. The result is that the hit and false alarm rate (60 and 50% respectively) are almost equal. Figure 1D illustrates the ROC curve in this case. The curve remains very close to the diagonal, meaning that a decrease of theta increases the hit rate, but at the cost of equally increasing the false alarm rate. The observer is essentially randomly guessing: the spike count gives him no information about what action was performed. The surface under the curve will be close to 0.5 (here 0.54), which is symptomatic for the cases when the activity of the neuron is unrelated to the action performed by the experimenter.
The surface under the ROC curve is thus an indication of the proportion of correct decisions that the observer makes, taking all the thetas into account. From the two examples, it is intuitive that a surface close to 0.5 represents random performance while a surface close to 1 indicates that the observer is very good at telling what action was performed. This surface then gives us valuable information about what function the neuron might have in the brain. If the imaginary observer is very good at telling what action was performed (large surface under the ROC curve), then the brain too could use this neuron to discriminate between the two actions. The neuron could thus participate in the perception of the actions. If the observer is very poor at telling the difference between the two actions using the spike count of this neuron, so would the brain be, and the neuron therefore is unlikely to be involved in the perception of the actions.
About this article Cite this articleKeysers, C., Kohler, E., Umiltà, M.A. et al. Audiovisual mirror neurons and action recognition. Exp Brain Res 153, 628–636 (2003). https://doi.org/10.1007/s00221-003-1603-5
Received: 25 October 2002
Accepted: 14 May 2003
Published: 23 August 2003
Issue Date: December 2003
DOI: https://doi.org/10.1007/s00221-003-1603-5
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4