Integrated Calendar

In this talk, I present recent progress in interpreting intermediate representations in vision models.
First, I demonstrate the existence of common intermediate representations (neurons) across a wide range of vision models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). I present an algorithm for finding these universal neurons and show that they can be used for model-to-model translation, enabling various zero-shot inversion-based image manipulations (e.g. shifting, zooming).
Second, I analyze the intermediate representations in CLIP, by investigating how they affect the final representation. I show that CLIP image representation can be decomposed as a sum across individual image patches, model layers, and attention heads and that CLIP's text representation can be used to interpret the summands. This decomposition enables an automatic characterization of attention head roles and reveals that some heads capture specific image properties (e.g. location or shape). It also uncovers emergent spatial localization within CLIP. Finally, this understanding helps to remove spurious features from CLIP and to create a strong zero-shot image segmenter.
This talk is based on two papers: "Rosetta Neurons: Mining the Common Units in a Model Zoo", and "Interpreting CLIP's Image Representation via Text-Based Decomposition".

Strong light-matter interactions are at the core of many electromagnetic phenomena. In this talk, I will give an overview of several nanophotonic systems which support polaritons – hybrid light-matter states, as well as try to demonstrate their potential usefulness in applications. I will start with transition metal dichalcogenides (TMDs) and specifically discuss one-dimensional edges in these two-dimensional materials (1-2). I will show that TMDs can be etched along certain crystallographic axes, such that the obtained edges are nearly atomically sharp and exclusively zigzag-terminated, while still supporting polaritonic regime. Furthermore, I will show that Fabry-Pérot resonators, one of the most important workhorses of nanophotonics, can spontaneously form in an aqueous solution of gold nanoflakes (3-4). This effect is possible due to the balance between attractive Casimir-Lifshitz forces and repulsive electrostatic forces acting between the flakes. There is a hope that this technology is going to be useful for future developments in self-assembly, nanomachinery, polaritonic devices, and perhaps other disciplines.
References: 1) Nat. Commun., 11, 4604, (2020) 2) Laser & Photonics Rev., 17, 2200057, (2023) 3) Nature 597, 214-219, (2021) 4) Nat. Phys. 19, 271-278, (2023)