Revolutionizing vision to unveil the hidden world

In the realm of computer vision, the conventional camera, which mimics the human eye, has long been the go-to sensor of researchers. Challenging the limitations of this approach and its inability to capture all the information present in our surroundings, Dr. Mark Sheinin, a new scientist in the Department of Computer Science and Applied Mathematics, is embarking on a mission to build cameras that reveal the hidden layers of our environment.

“Because traditional cameras have remained relatively unchanged for a century—basically designed to capture what the human eye sees, they inherently miss out on crucial information,” explains Dr. Sheinin. “I envision a world where cameras would be able to transcend their conventional role and capture nuances hidden in plain sight.”

A flicker of insight

During his PhD research at the Technion–Israel Institute of Technology in Haifa, Dr. Sheinin built an imaging device that captured the imperceptible fluctuations of lightbulbs. These minute flickers, undetectable by the human eye, became a rich source of information for his studies.

One notable application involved separating captured scenes into individual light sources. In a lamppost-lit street, for example, this innovative camera could distinguish each light source and provide a visual representation of the scene with individual streetlights switched on or off without physical intervention.

Additionally, Dr. Sheinin harnessed the camera’s capacity to analyze individual lights in-depth, extracting valuable information about the bulbs themselves. Not only could the camera identify the bulb type, but it also detected electrical perturbations and discerned the distribution of specific electric phases, offering insights that could help advance electric grid design.

Continuing his mission to build novel imaging devices during his postdoctoral studies at Carnegie Mellon University, Dr. Sheinin worked on various cameras designed to capture different phenomena.

His first postdoctoral project involved creating a specialized camera that used a 1D sensor to capture a 2D scene. This novel approach addresses the speed limitation of traditional cameras, which allocate bandwidth equally between all camera pixels, regardless of the imaging tasks’ specific objective.

However, in scenarios like motion capture―where the only objects of interest are the illuminated points on a motion-capture suit―conventional cameras expend resources on redundant dark pixels, hindering high-speed operations needed for capturing the action of rapidly moving objects. To overcome this, Dr. Sheinin devised a solution: encoding and capturing the light only from the relevant points with 1D sensors. This breakthrough allowed for a significantly faster operation—up to 50 times the speed.

Seeing sound

Another standout postdoctoral project was the development of a camera that “sees sound” by sensing tiny vibrations on object surfaces. For example, in a scene where two musicians are playing simultaneously, rather than capturing a blend of both sounds as a traditional camera microphone would, Dr. Sheinin’s device was able to record both players separately: by directing a laser at a specific point, the camera isolated the vibrations produced by each musician, “listening” to each one without interference from the other, and yielding separate recordings of both.

Additionally, this camera could use its vibration-detecting laser to identify an object’s point of impact. In an experimental setup beneath a ping pong table,

Dr. Sheinin’s camera accurately detected where a ping pong ball bounced based solely on table vibrations―eliminating the need for direct visual observation.

“The scientific applications for this type of camera are extensive,” he explains. “By being able to detect tiny imperceptible vibrations of an object, I see it potentially being used for things like testing the structural integrity of bridges, or locating individuals buried in post-earthquake wreckage.”

Thermal breadcrumbs

In his latest project, Dr. Sheinin developed a cutting-edge device that employs a laser projector and thermal camera to remotely generate intricate heat patterns on object surfaces. Unlike projected patterns, these thermal patterns, consisting of thousands of heat points, “stick” to a moving object.

This distinctive feature enables this technology to track and reconstruct 3D models of uniform, textureless objects whose features would be challenging to capture with a regular camera.

In another application, this device, when affixed to a moving object, leaves a trail of thermal “breadcrumbs,” allowing it to accurately trace the object’s path around a room.

“It’s a whole new sensing modality,” says Dr. Sheinin. “That’s what drives my research: finding the boundaries of our perception and breaking through them.”

He plans to continue pushing the boundaries of these innovative technologies in his new lab at the Weizmann Institute. One of his goals is to create a camera capable of perceiving all vibrations in an environment, such as those on every object surface in a room. This technology could potentially allow machines to infer information about events occurring beyond the line of sight by observing the vibrations they generate (e.g., a person walking on the upper floor).

“I’m constantly trying to imagine how we would operate in the world if our vision system was different,” Dr. 

Sheinin states, “and I think the Weizmann Institute is the ideal place to do that—a place where I can not only think outside the box but build an entirely new box.”

Dr. Sheinin joined the Weizmann faculty in January 2024.

Education and select awards

BSc, Technion–Israel Institute of Technology (2011)

MSc-PhD (direct track), Technion–Israel Institute of Technology (2019)

Postdoc, Carnegie Mellon University, Pennsylvania (2019-2023)

Best Student Paper Award at the Institute of Electrical and Electronics Engineers Conference on Computer Vision and Pattern Recognition (IEEE CVPR) (2017); Jacobs-Qualcomm Fellowship (2017, 2018); Porat Award for Outstanding Graduate Students (2017); Jacobs Distinguished Publication Award (2018); Viterbi Fellowship for Nurturing Future Faculty Members (2019-2020); Best Paper Honorable Mention Award at IEEE CVPR (2022)