Integrated Calendar

The spiking of dopamine neurons in animals, and apparently analogous BOLD signals at dopaminergic targets in humans, appear to report predictions of future reward. Prominent computational theories of these responses suggest that they both support and reflect trial-and-error learning about which actions have been successful, based on simple associations with past rewards. This is essentially a neural implementation of Thorndike's (1911) behaviorist principle that reinforced behaviors should be repeated. However, it has long been known that organisms are not condemned merely to repeat previously successful actions, but instead that even rodents' decisions can under some circumstances reflect other sorts of knowledge about task structure and contingencies. The neural and computational bases for these additional effects, and their interaction with the putative reinforcement systems in the basal ganglia, are poorly understood.
Such interactions are of considerable practical importance because, for instance, disorders of compulsion in humans, such as substance abuse, are thought to arise from runaway reinforcement processes unfettered by more deliberative influences.

I first discuss how such extra-reinforcement effects – e.g., planning novel routes based on cognitive maps, or incorporating "counterfactual" feedback about foregone actions – can be incorporated in the framework of existing computational theories, via algorithms for “model-based reinforcement learning." Rather than learning about actions' past successes directly, such algorithms learn a representation of the task structure, and can use it to evaluate candidate actions via mental simulation of their consequences. This computational characterization allows reasoning about (and explaining empirical data concerning) under which circumstances the brain might efficiently adopt either this strategy or the reinforcement one. It also allows quantifying and dissociating either strategy's effects on decision making and associated neural signaling.

Next, I discuss human fMRI experiments characterizing these influences in learning tasks. By fitting computational models to decision behavior and BOLD signals, we demonstrate that neither choices nor (putatively dopamine-related) BOLD signals in striatum can be explained by past reinforcement alone, but instead that both reflect additional learning and reasoning about task structure and contingencies. That such influences are prominent even at the level of striatum challenges current models of the computations there and suggest that the system is a common target for many different sorts of learning. Additional experiments examine individual variation in the tendency to employ either system; the patterns of both spontaneous and experimentally induced variation suggest that the dominance of model-based decision influence over simpler reinforcement systems employs cognitive control mechanisms that have previously been studied in other areas of cognitive neuroscience. Finally, I report results showing that patients with several disorders involving compulsion show abnormally reinforcement-bound choices on our tasks, supporting a link between these neurocomputational learning mechanisms and pathological habits.

Orbitofrontal cortex (OFC) has long been known to play an important role in decision making. However, the exact nature of that role has remained elusive. The OFC does not seem necessary for almost anything---animals and humans can learn, unlearn and reverse previous learning even without an OFC, albeit more slowly than their healthy counterparts. What role, then, can the OFC be playing such that its absence would cause subtle but broadly permeating deficits? We propose a new unifying theory of OFC function. Specifically, we hypothesize that OFC encodes a map of the states of the current task and their inter-relations, which provides a state space for reinforcement learning elsewhere in the brain. I will first use a simple perceptual judgement task to demonstrate that state spaces, a critical ingredient in any reinforcement learning algorithm, are learned from data. I will then use our hypothesis that the OFC encodes the learned state space to explain recent experimental findings in an odor-guided choice task (Takahashi et al, Nature Neuroscience 2012) as well as classic findings in reversal learning and extinction. Finally, I will lay out a number of testable experimental predictions that can distinguish our theory from other accounts of OFC function.

Using modern micro and nano-fabrication techniques combined with superconducting materials we realize electronic circuits the dynamics of which are governed by the laws of quantum mechanics. Making use of the strong interaction of photons with superconducting quantum two-level systems realized in these circuits we investigate both fundamental quantum effects of light and applications in quantum information processing. In this presentation, I will discuss novel methods to investigate the quantum properties of microwave frequency radiation emitted from solid state devices. Instead of employing photon counters, as commonly done at optical frequencies, we use linear amplifiers and measure the amplitude and phase of the emitted electromagnetic fields at the quantum level. For this purpose we have developed efficient methods to separate the quantum signal of interest from the noise added by the linear amplifiers [1]. To demonstrate the power of these techniques we have realized on-demand single microwave photon sources which we characterize using correlation function measurements which display anti-bunching of the detected photons [2] and full quantum state tomography which display the negativity of the extracted Wigner functions [3]. The presented techniques are readily applicable for investigating other solid state emitters in the microwave frequency domain, such as quantum dots for example. We have also explored the entanglement created between stationary emitters and freely propagating microwave photons [4] and the Hong-Ou-Mandel effect in this way.

[1] C. Eichler et al., Phys. Rev. A 86, 032106 (2012)
[2] D. Bozyigit et al., Nat. Phys. 7, 154 (2011)
[3] C. Eichler et al., Phys. Rev. Lett. 106, 220503 (2011)
[4] C. Eichler et al., Phys. Rev. Lett., in print (2012), also arXiv:1209.0441