לוח שנה משולב

Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets. This talk presents LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained automatic speech recognition (ASR) serves as the classifier. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals than competing methods (demo page: https://lipvoicer.github.io). The presented LipVoicer is a joint work of Yochai Yemini, Aviv Shamsian, Lior Bracha, Sharon Gannot, and Ethan Fetaya.

Bio:

Sharon Gannot obtained his Ph.D. in electrical engineering from Tel-Aviv University, Israel, in 2000. He is a full professor in the Faculty of Engineering at Bar-Ilan University, Israel. He serves as a senior area chair for IEEE Transactions on Audio, Speech, and Language Processing, a member of the senior editorial board of IEEE Signal Processing Magazine, a member of the editorial board of IEEE SPS Education Center, and the chair of the IEEE Signal Processing Society Data Science Initiative. Previously, he was chair of the IEEE Audio and Acoustic Signal Processing Technical Committee in 2017–2018. He has also held other roles, such as the general co-chair of the 2010 International Workshop on Acoustic Signal Enhancement and the 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. He is the general co-chair of Interspeech 2024, which will be held in September in Greece. His research interests include statistical signal processing and machine learning in the audio processing domain. The methods he develops utilize multi-microphone and multi-modal information. Applications include speech enhancement, noise reduction, speaker separation and diarization, dereverberation, speaker localization, and tracking. Sharon Gannot is the recipient of the 2022 European Association for Signal Processing Group Technical Achievement Award and a Fellow of the IEEE.

 

 

Learning from data has led to paradigm shifts in a multitude of disciplines, including web, text and image search and generation, speech recognition, as well as bioinformatics. Can machine learning enable similar breakthroughs in understanding (quantum) dynamics of molecules and materials? This is an especially relevant question given that quantum mechanical predictions are now within experimental uncertainties for complex molecular systems (https://www.nature.com/articles/s41586-023-06587-3).

Aiming towards a unified machine learning (ML) model of molecular interactions in chemical space, I will discuss the potential and challenges for using ML techniques in chemistry and physics. ML methods can not only accurately estimate molecular properties of large datasets, but they can also lead to new insights into chemical similarity, aromaticity, reactivity, and molecular dynamics. For example, the combination of reliable molecular data with ML methods has enabled a fully quantitative simulation of protein dynamics in water (https://www.science.org/doi/full/10.1126/sciadv.adn4397). While the potential of machine learning for revealing insights into molecules and materials is high, I will conclude my talk by discussing the many remaining avenues for development.

In this talk I will present a useful construction introduced by Kassabov and Pak, called diagonal products. They arise naturally in the study of the space of marked groups, that is, the normal subgroups of a free group with the Chabauty topology. It turns out that diagonal products are extremely rich, and proved to be a useful tool for providing a full spectrum of various growth functions for groups. This includes answers to questions such as "How fast do Følner sets grow in an amenable group" and "How fast do residual chains grow in residually finite groups" etc. We will elaborate on a joint work with Arie Levit and Itamar Vigdorovich, where we give a general classification result for characters (á la Thoma) of many diagonal products. As a result, we deduce that there are uncountably many Hilbert–Schmidt stable groups, which are as unstable as one wants.

The interplay between spontaneous symmetry breaking and topology can result in exotic quantum states of matter. A celebrated example is the quantum anomalous Hall (QAH) effect, which exhibits an integer quantum Hall effect at zero magnetic field due to topologically nontrivial bands and intrinsic magnetism. In the presence of strong electron-electron interactions, fractional-QAH (FQAH) effect at zero magnetic field can emerge, which is a lattice analog of fractional quantum Hall effect without Landau level formation. In this talk, I will present experimental observation of FQAH effect in twisted MoTe2 bilayer, using combined magneto-optical and -transport measurements. In addition, we find an anomalous Hall state near the filling factor -1/2, whose behavior resembles that of the composite Fermi liquid phase in the half-filled lowest Landau level of a two-dimensional electron gas at high magnetic field. Direct observation of the FQAH and associated effects paves the way for researching charge fractionalization and anyonic statistics at zero magnetic field.

Reference
1. Observation of Fractionally Quantized Anomalous Hall Effect, Heonjoon Park et al., Nature, https://www.nature.com/articles/s41586-023-06536-0 (2023);
2. Signatures of Fractional Quantum Anomalous Hall States in Twisted MoTe2 Bilayer, Jiaqi Cai et al., Nature, https://www.nature.com/articles/s41586-023-06289-w (2023);
3. Programming Correlated Magnetic States via Gate Controlled Moiré Geometry, Eric Anderson et al., Science, https://www.science.org/doi/full/10.1126/science.adg4268 (2023).

In this talk, we will prove the topological overlap property for 2-complexes that generalizes the Boros–Furedi theorem, following Yehudayoff's exposition of Gromov's method. The main idea is the notion of a folding map or obstruction, which also has a cohomological interpretation. We will further discuss homological connectivity of random 2-complexes based on a work of Linial and Meshulam.