לוח שנה משולב

In this last talk of the seminar, we will pack together all the topics discussed over the semester. In particular, we will suggest a path toward finding a non sofic group. 

While cognitive neuroscientists have uncovered principles of perceptual decision-making by analyzing choices and neuronal firing across thousands of trials, we do not yet know the behavioral or neuronal dynamics underlying one SINGLE choice. For instance, why might a subject judge a given stimulus in category A 70% of the time but in category B 30%? Until we can work out precisely what determines single-decisions – this choice, right now – the mechanisms of real-world decision-making will remain unknown. In tactile psychophysical tasks with rats and humans, we are trying to sort out factors that explain the variability in judgments (across trials) to the identical stimulus input. We identify four factors: (i) trial-to-trial fluctuations in sensory coding, (ii) temporal context, namely, the history of preceding stimuli and choices, (iii) attention, and (iv) bias (predictions originating in beliefs about the environment’s probabilistic structure). The strategy is to bring these factors under experimental control, rather than leaving them to vary according to uninterrogated states within the subject. Psychophysics from rats and humans show that large chunks of variability are accounted for by these factors; evidence from cortical neuronal populations in rats provides some mechanistic grounding.

While language models (LMs) show impressive text manipulation capabilities, they also lack commonsense and reasoning abilities and are known to be brittle. In this talk, I will suggest a different LMs design paradigm, inspired by how humans understand it. I will present two papers, both shedding light on human-inspired NLP architectures aimed at delving deeper into the meaning beyond words.
 The first paper [1] accounts for the lack of commonsense and reasoning abilities by proposing a paradigm shift in language understanding, drawing inspiration from embodied cognitive linguistics (ECL). In this position paper we propose a new architecture that treats language as inherently executable, grounded in embodied interaction, and driven by metaphoric reasoning.
 The second paper [2] shows that LMs are brittle and far from human performance in their concept-understanding and abstraction capabilities. We argue this is due to their token-based objectives, and implement a concept-aware post-processing manipulation, showing it matches human intuition better. We then pave the way for more concept-aware training paradigms.  

[1] Language (Re)modelling: Towards Embodied Language Understanding
Ronen Tamari, Chen Shani, Tom Hope, Miriam R L Petruck, Omri Abend, and Dafna Shahaf. 2020.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6268–6281, Online. Association for Computational Linguistics. 

[2] Towards Concept-Aware Large Language Models
Shani, Chen, Jilles Vreeken, and Dafna Shahaf.
In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13158-13170. 2023.
 
Bio: Chen Shani is a post-doctoral researcher at Stanford's NLP group, collaborating with Prof. Dan Jurafsky. Previously, she pursued her Ph.D. at the Hebrew University under the guidance of Prof. Dafna Shahaf and worked at Amazon Research. Her focus lies at the intersection of humans and NLP, where she implements insights from human cognition to improve NLP systems.

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific fine-tuning or changing the model weights? Inspired by prompting in NLP, Visual Prompting is an exciting new paradigm that guides computer vision models to perform downstream tasks by providing visual examples. This talk will focus on designing the underlying self-supervised learning objectives, architectures, and various methods to prompt vision models. Finally, I will discuss in-context learning from a mechanistic interpretability standpoint and share new insights into how transformer models can encode task-specific information in their activation space.

Bio: Amir Bar is a PhD student at Tel Aviv University and Berkeley AI Research. During his PhD, Amir has focused on self-supervised learning in computer vision and pioneered Visual Prompting (or visual in-context learning). Before starting his PhD, he led the research team at the healthcare startup Zebra Medical Vision (recently acquired), where his team developed multiple FDA-approved algorithms now in clinical use in hospitals worldwide. Starting in August 2024, Amir will join FAIR Labs as a postdoctoral researcher, working with Yann LeCun.