לוח שנה משולב

Neural circuits in the brain must be plastic enough to allow an animal to adapt to and learn from new experiences yet they must also remain functionally stable such that previously learned skills and information are retained. Thus, fundamental questions in neuroscience concern the molecular, cellular, and circuit mechanisms that balance the plasticity and stability of neural circuits. During my studies, I investigated these mechanisms in three studies that focused on sensory- and behavioral state-dependent changes in transcription and GABAergic inhibition in the visual cortex of adult mice. In my Ph.D. defense, I will elaborate on the novel molecular-cellular mechanisms that I discovered in these studies and discuss their role in conveying both plasticity and stability to visual processing and perception.

I will describe an attempt to describe turbulence using the methods of quantum field theory. We consider waves that interact via four-wave scattering (such as sea waves, plasma waves, spin waves, and many others). By summing the series of the most UV-divergent terms in the perturbation theory, we show that the true dimensionless coupling is different from the naive estimate, and find that the effective interaction either decays or grows explosively with the cascade extent, depending on the sign of the new coupling. The explosive growth possibly signals the appearance of a multi-wave bound state (solitons, shocks, cusps) similar to confinement in quantum chromodynamics.

In this talk, I present recent progress in interpreting intermediate representations in vision models.
First, I demonstrate the existence of common intermediate representations (neurons) across a wide range of vision models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). I present an algorithm for finding these universal neurons and show that they can be used for model-to-model translation, enabling various zero-shot inversion-based image manipulations (e.g. shifting, zooming).
Second, I analyze the intermediate representations in CLIP, by investigating how they affect the final representation. I show that CLIP image representation can be decomposed as a sum across individual image patches, model layers, and attention heads and that CLIP's text representation can be used to interpret the summands. This decomposition enables an automatic characterization of attention head roles and reveals that some heads capture specific image properties (e.g. location or shape). It also uncovers emergent spatial localization within CLIP. Finally, this understanding helps to remove spurious features from CLIP and to create a strong zero-shot image segmenter.
This talk is based on two papers: "Rosetta Neurons: Mining the Common Units in a Model Zoo", and "Interpreting CLIP's Image Representation via Text-Based Decomposition".