Publications
2022
-
(2022) Trends in Cognitive Sciences. 26, 4, Abstract
Having established that spontaneous brain activity follows meaningful coactivation patterns and correlates with behavior, researchers have turned their attention to understanding its function and behavioral significance. We suggest closed-loop neuromodulation as a neural perturbation tool uniquely well suited for this task. Closed-loop neuromodulation has primarily been viewed as an interventionist tool to teach subjects to directly control their own brain activity. We examine an alternative operant conditioning model of closed-loop neuromodulation which, through implicit feedback, can manipulate spontaneous activity at the network level, without violating the spontaneous or endogenous nature of the signal, thereby providing a direct test of network causality.
2021
-
(2021) Autism Research. 14, 8, p. 1670-1683 Abstract
Eye tracking provides insights into social processing deficits in autism spectrum disorder (ASD), especially in conjunction with dynamic, naturalistic free-viewing stimuli. However, the question remains whether gaze characteristics, such as preference for specific facial features, can be considered a stable individual trait, particularly in those with ASD. If so, how much data are needed for consistent estimations? To address these questions, we assessed the stability and robustness of gaze preference for facial features as incremental amounts of movie data were introduced for analysis. We trained an artificial neural network to create an object-based segmentation of naturalistic movie clips (14 s each, 7410 frames total). Thirty-three high-functioning individuals with ASD and 36 age- and IQ-equated typically developing individuals (age range: 12–30 years) viewed 22 Hollywood movie clips, each depicting a social interaction. As we evaluated combinations of one, three, five, eight, and 11 movie clips, gaze dwell times on core facial features became increasingly stable at within-subject, within-group, and between-group levels. Using a number of movie clips deemed sufficient by our analysis, we found that individuals with ASD displayed significantly less face-centered gaze (centralized on the nose; p
2020
-
-
A Deep Neural Network Tool for Automatic Segmentation of Human Body Parts in Natural Scenes(2020) arXiv. 2009.09900. Abstract
This short article describes a deep neural network trained to perform automatic segmentation of human body parts in natural scenes. More specifically, we trained a Bayesian SegNet with concrete dropout on the Pascal-Parts dataset to predict whether each pixel in a given frame was part of a person's hair, head, ear, eyebrows, legs, arms, mouth, neck, nose, or torso.
-
(2020) BioRxiv. Abstract
Eye tracking provides insights into social processing and its deficits in disorders such as autism spectrum disorder (ASD), especially in conjunction with dynamic, naturalistic stimuli. However, reliance on manual stimuli segmentation severely limits scalability. We assessed how the amount of available data impacts individual reliability of fixation preference for different facial features, and the effect of this reliability on between-group differences. We trained an artificial neural network to segment 22 Hollywood movie clips (7410 frames). We then analyzed fixation preferences in typically developing participants and participants with ASD as we incrementally introduced movie data for analysis. Although fixations were initially variable, results stabilized as more data was added. Additionally, while those with ASD displayed significantly fewer face-centered fixations (plt;.001), they did not differ in eye or mouth fixations. Our results highlight the validity of treating fixation preferences as a stable individual trait, and the risk of misinterpretation with insufficient data.Competing Interest StatementThe authors have declared no competing interest.
-
(2020) Communications Biology. 3, 1, 48. Abstract
Extensive study of typically developing individuals and those on the autism spectrum has identified a large number of brain regions associated with our ability to navigate the social world. Although it is widely appreciated that this so-called “social brain” is composed of distinct, interacting systems, these component parts have yet to be clearly elucidated. Here we used measures of eye movement and neural typicality—based on the degree to which subjects deviated from the norm—while typically developing (N = 62) and individuals with autism (N = 36) watched a large battery of movies depicting social interactions. Our findings provide clear evidence for distinct, but overlapping, neural systems underpinning two major components of the “social brain,” social orienting, and inferring the mental state of others.
2019
-
(2019) Journal of Neuroscience. 39, 25, p. 4976-4985 Abstract
Our ability to recognize others by their facial features is at the core of human social interaction, yet this ability varies widely within the general population, ranging from developmental prosopagnosia to “super-recognizers”. Previous work has focused mainly on the contribution of neural activity within the well described face network to this variance. However, given the nature of face memory in everyday life, and the social context in which it takes place, we were interested in exploring how the collaboration between different networks outside the face network in humans (measured through resting state connectivity) affects face memory performance. Fifty participants (men and women) were scanned with fMRI. Our data revealed that although the nodes of the face-processing network were tightly coupled at rest, the strength of these connections did not predict face memory performance. Instead, face recognition memory was dependent on multiple connections between these face patches and regions of the medial temporal lobe memory system (including the hippocampus), and the social processing system. Moreover, this network was selective for memory for faces, and did not predict memory for other visual objects (cars). These findings suggest that in the general population, variability in face memory is dependent on how well the face processing system interacts with other processing networks, with interaction among the face patches themselves accounting for little of the variance in memory ability.
-
(2019) Progress in Neuro-Psychopharmacology and Biological Psychiatry. 90, p. 28-36 Abstract
Autism spectrum disorder (ASD) is characterized by profound impairments in social abilities and by restricted interests and repetitive behaviors. Much work in the past decade has been dedicated to understanding the brain-bases of ASD, and in the context of resting-state functional connectivity fMRI in high-functioning adolescents and adults, the field has established a set of reliable findings: decreased cortico-cortical interactions among brain regions thought to be engaged in social processing, along with a simultaneous increase in thalamo-cortical and striato-cortical interactions. However, few studies have attempted to manipulate these altered patterns, leading to the question of whether such patterns are actually causally involved in producing the corresponding behavioral impairments. We discuss a few such recent attempts in the domains of fMRI neurofeedback and overt social interaction during scanning, and we conclude that the evidence of causal involvement is somewhat mixed. We highlight the potential role of the thalamus and striatum in ASD and emphasize the need for studies that directly compare scanning during multiple cognitive states in addition to the resting-state.
-
(2019) NeuroImage. 188, p. 322-334 Abstract
Interest in real-time fMRI neurofeedback has grown exponentially over the past few years, both for use as a basic science research tool, and as part of the search for novel clinical interventions for neurological and psychiatric illnesses. In order to expand the range of questions which can be addressed with this tool however, new neurofeedback methods must be developed, going beyond feedback of activations in a single region. These new methods, several of which have already been proposed, are by their nature complex, involving many possible parameters. Here we suggest a framework for evaluating and optimizing algorithms for use in a real-time setting, before beginning the neurofeedback experiment, by offline simulations of algorithm output using a previously collected dataset. We demonstrate the application of this framework on the instantaneous proxy for correlations which we developed for training connectivity between different network nodes, identify the optimal parameters for use with this algorithm, and compare it to more traditional correlation methods. We also examine the effects of advanced imaging techniques, such as multi-echo acquisition, and the integration of these into the real-time processing stream.
2017
-
(2017) eLife. 6, e28974. Abstract
The existence of abnormal connectivity patterns between resting state networks in neuropsychiatric disorders, including Autism Spectrum Disorder (ASD), has been well established. Traditional treatment methods in ASD are limited, and do not address the aberrant network structure. Using real-time fMRI neurofeedback, we directly trained three brain nodes in participants with ASD, in which the aberrant connectivity has been shown to correlate with symptom severity. Desired network connectivity patterns were reinforced in real-time, without participants’ awareness of the training taking place. This training regimen produced large, significant long-term changes in correlations at the network level, and whole brain analysis revealed that the greatest changes were focused on the areas being trained. These changes were not found in the control group. Moreover, changes in ASD resting state connectivity following the training were correlated to changes in behavior, suggesting that neurofeedback can be used to directly alter complex, clinically relevant network connectivity patterns.
2016
-
Diminished Auditory Responses during NREM Sleep Correlate with the Hierarchy of Language Processing.(2016) PLoS ONE. 11, 6, e0157143. Abstract
Natural sleep provides a powerful model system for studying the neuronal correlates of awareness and state changes in the human brain. To quantitatively map the nature of sleep-induced modulations in sensory responses we presented participants with auditory stimuli possessing different levels of linguistic complexity. Ten participants were scanned using functional magnetic resonance imaging (fMRI) during the waking state and after falling asleep. Sleep staging was based on heart rate measures validated independently on 20 participants using concurrent EEG and heart rate measurements and the results were confirmed using permutation analysis. Participants were exposed to three types of auditory stimuli: scrambled sounds, meaningless word sentences and comprehensible sentences. During non-rapid eye movement (NREM) sleep, we found diminishing brain activation along the hierarchy of language processing, more pronounced in higher processing regions. Specifically, the auditory thalamus showed similar activation levels during sleep and waking states, primary auditory cortex remained activated but showed a significant reduction in auditory responses during sleep, and the high order language-related representation in inferior frontal gyrus (IFG) cortex showed a complete abolishment of responses during NREM sleep. In addition to an overall activation decrease in language processing regions in superior temporal gyrus and IFG, those areas manifested a loss of semantic selectivity during NREM sleep. Our results suggest that the decreased awareness to linguistic auditory stimuli during NREM sleep is linked to diminished activity in high order processing stations.
-
(2016) Proceedings of the National Academy of Sciences of the United States of America. 113, 17, p. E2413-E2420 Abstract
Recent advances in blood oxygen level-dependent-functional MRI (BOLD-fMRI)-based neurofeedback reveal that participants can modulate neuronal properties. However, it is unknown whether such training effects can be introduced in the absence of participants-awareness that they are being trained. Here, we show unconscious neurofeedback training, which consequently produced changes in functional connectivity, introduced in participants who received positive and negative rewards that were covertly coupled to activity in two category-selective visual cortex regions. The results indicate that brain networks can be modified even in the complete absence of intention and awareness of the learning situation, raising intriguing possibilities for clinical interventions.
2014
-
(2014) Proceedings of the 6th International Brain-Computer Interface Conference 2014. Brunner C., Steyrl D., Müller-Putz G., Bauernfeind G., Scherer R. & Wriessnegger S.(eds.). p. 133-136 Abstract
Objective : We have developed an efficient generic machine learning (ML) tool for realtime fMRI whole brain classification, which can be used to explore novel brain-computer interface (BCI) or advanced neurofeedback (NF) strategies.Approach : We use information gain for isolating the most relevant voxels in the brain and a support vector machine classifier.Main results : We have used our tool in three types of experiments: motor movement, motor imagery and visual categories.Significance : We show high accuracy results in real-time, using an optimal number of voxels, with a shorter delay compared to the previous method based on regions of interest (ROI). Finally, our tool is integrated with a virtual environment and can be used to control a virtual avatar or a robot.
2013
-
(2013) Journal of Neurophysiology. 109, 9, p. 2272-2281 Abstract
A fundamental debate in the study of cortical sensory systems concerns the scale of functional selectivity in cortical networks. Brain imaging studies have repeatedly demonstrated functional selectivity in entire cortical areas and networks using predetermined stimuli. However, it is not clear to what extent these networks are heterogeneous, i.e., whether the selectivity profiles in subregions within each sensory network show significant dissimilarity. Here, we studied local functional selectivity in the human cortex using naturalistic movie clips shown to 12 patients implanted with intracranial electrocorticography electrodes (590 in total), providing extensive cortical coverage. We examined the similarity of response profiles (40 to 80-Hz gamma-power modulations) across electrodes using a novel data driven approach without assuming any predefined category. Our results show that the functional selectivity of each highly responsive electrode was different from that of all other electrodes across the sensory cortex. Thus most responsive electrodes showed an activation profile that was unique in each patient and was similar to that of only 0.3% (1-2) of all other electrodes across all patients. Functional similarity between electrodes was linked to anatomical proximity. While in most electrodes the source of selectivity was complex, a small subset showed the welldocumented selectivity to faces and actions. Our results indicate that the human sensory cortex is organized as a mosaic of functionally unique subregions in which each site manifests its own special response profile.
-
(2013) Journal of Neuroscience. 33, 37, p. 14715-14728 Abstract
Despite the profound reduction in conscious awareness associated with sleep, sensory cortex remains highly active during the different sleep stages, exhibiting complex interactions between different cortical sites. The potential functional significance of such spatial patterns and how they change between different sleep stages is presently unknown. In this electrocorticography study of human patients, we examined this question by studying spatial patterns of activity (broadband gamma power) that emerge during sleep (sleep patterns) and comparing them to the functional organization of sensory cortex that is activated by naturalistic stimuli during the awake state. Our results show a high correlation (p
-
(2013) Journal of Neuroscience. 33, 3, p. 1228-1240 Abstract
One of the puzzling aspects in the visual attention literature is the discrepancy between electrophysiological and fMRI findings: whereas fMRI studies reveal strong attentional modulation in the earliest visual areas, single-unit and local field potential studies yielded mixed results. In addition, it is not clear to what extent spatial attention effects extend from early to high-order visual areas. Here we addressed these issues using electrocorticography recordings in epileptic patients. The patients performed a task that allowed simultaneous manipulation of both spatial and object-based attention. They were presented with composite stimuli, consisting of a small object (face or house) superimposed on a large one, and in separate blocks, were instructed to attend one of the objects. We found a consistent increase in broadband high-frequency (30 -90 Hz) power, but not in visual evoked potentials, associated with spatial attention starting with V1/V2 and continuing throughout the visual hierarchy. The magnitude of the attentional modulation was correlated with the spatial selectivity of each electrode and its distance from the occipital pole. Interestingly, the latency of the attentional modulation showed a significant decrease along the visual hierarchy. In addition, electrodes placed over high-order visual areas (e.g., fusiform gyrus) showed both effects of spatial and object-based attention. Overall, our results help to reconcile previous observations of discrepancy between fMRI and electrophysiology. They also imply that spatial attention effects can be found both in early and high-order visual cortical areas, in parallel with their stimulus tuning properties.
2012
-
(2012) Journal of Neuroscience. 32, 31, p. 10458-10469 Abstract
While research of human cortical function has typically focused on task-related increases in neuronal activity, there is a growing interest in the complementary phenomenon-namely, task-induced reductions. Recent human BOLD fMRI studies have associated such reductions with a specific network termed the default mode network (DMN). However, detailed understanding of the spatiotemporal patterns of task-negative responses and particularly how they compare across different cortical networks is lacking. Here we examined this issue in a large-scale electrocorticography study in patients performing a demanding backward masking task. Our results uncovered rapid (
-
(2012) PLoS ONE. 7, 5, e37238. Abstract
Clinical diagnosis of disorders of consciousness (DOC) caused by brain injury poses great challenges since patients are often behaviorally unresponsive. A promising new approach towards objective DOC diagnosis may be offered by the analysis of ultra-slow (
2011
-
(2011) NeuroImage. 58, 1, p. 213-225 Abstract
The recent discovery of incessant spontaneous fluctuations in human brain activity (also termed resting state fMRI) has been a focus of intense research in brain imaging. The spontaneous BOLD activity shows organized anatomical specialization as well as disruption in a number of brain pathologies. The link between the spontaneous fMRI fluctuations and human behavior is therefore of acute interest and importance. Here we report that a highly significant correlation exists between spontaneous BOLD fluctuations and eye movements which occur subliminally and spontaneously in the absence of any visual stimulation. Of the various eye movement parameters tested, we found robust and anatomically consistent correlations with both the amplitude and velocity of spontaneous eye movements. Control experiments ruled out a contribution of spatial and visual attention as well as smooth pursuit eye movements to the effect. The consistent anatomical specificity of the correlation patterns and their tight temporal link at the proper hemodynamic delay argues against a non-neuronal explanation of the effect, such as cardiac or respiratory cycles. Our results thus demonstrate a link between resting state and spontaneously emerging subconscious oculo-motor behavior.
2009
-
(2009) Neuron. 64, 4, p. 562-574 Abstract
Human recognition performance is characterized by abrupt changes in perceptual states. Understanding the neuronal dynamics underlying such transitions could provide important insights into mechanisms of recognition and perceptual awareness. Here we examined patients monitored for clinical purposes with multiple subdural electrodes. The patients participated in a backward masking experiment in which pictures of various object categories were presented briefly followed by a mask. We recorded ECoG from 445 electrodes placed in 11 patients. We found a striking increase in gamma power (30-70 Hz) and evoked responses specifically associated with successful recognition. The enhanced activation occurred 150-200 ms after stimulus onset and consistently outlasted the stimulus presentation. We propose that the gamma and evoked potential activations reflect a rapid increase in recurrent neuronal activity that plays a critical role in the emergence of a recognizable visual percept in conscious awareness.