Raja Giryes - On the Visual Capabilities of Multimodal Models

0:02 hello everyone. I want to thank the organizers and all the previous speakers for the interesting talks. my talk is taking a slightly different angle than what was presented before. So, everyone here was talking about coding agents and how we can extend them. Here I want to talk about adding new capabilities and which is going to multimodal. So, most current models focus on how to improve coding because like it's easier to work like that.

0:33 But when you hear me now, you don't just see a text. You see me. You see everything. So, we live in a multimodal world. And the question is, what are the current visual capabilities that we have in this in multimodal models? So, there are various challenges when we look at multimodal models. And I think that one of the interesting thing that we see in the field is that once people start asking the right question in one or two years, we start seeing solutions.

1:08 So, few years ago with my group, we had some works on compositional reasoning. There were some observations that one of the problems of old multimodal models when they look at captioning of images. So, for example, if I have if I want to generate or want to describe an image with like three zebras and two giraffes. So, basically all the numbers and names are mixed together and the model is treating everything as bag of words.

1:43 And then there is some problem with compositionality. And then with my group, we had various work on improving the compositionality. And since then, things improved. So, we used some tree structures. We took the words and decomposed them into trees. We basically had more dense and aligned captions. And basically we taught this model to be a structured. So you can see them in this works. Afterwards, we found other problems that exist in these models. So in two works, we tried to analyze what happens if you try to work with these models and see what happened in the workflow of understanding. So in one work that was done with my colleagues, which is called performance gap in entity knowledge and was published in ACL one year ago, we basically tried to see where in multimodal models we can we can see the reasoning. And one of the problems that we see in current multimodal models is that basically if you look at all modern multimodal models, you have very strong LLM and a nice visual head attached to it.

3:02 So we invest lots of resources in training LLMs, but then we have some nice visual visual language model that we found somewhere and you attach to it. We train them together afterwards with some post training. And then when you analyze what happens when these models try to do some multimodal understanding, you see that most of the analysis that happened happened in the LLM part and not in the visual part.

3:32 Another work, we tried to analyze what happened if we tried to do in-context learning. So in LLMs, we know that in-context learning is great, but then the strange thing that we found in VLMs or multimodal models is that if you give in-context learning that is too long with too many images, basically the models start to get confused. And again, the reason for that is that currently when we train multimodal models, we mainly rely on the language part. And the question is, maybe we need to make a certain shift.

4:05 So, the focus in my talk would be on a specific ability of multimodal models, which is spatial cognition. And there is an interesting work of my colleagues that was published 2 years ago, which is whether spatial cognition emerge in frontier models. And what they did is they checked various spatial cognition tests on multimodal models. And they checked some task like mental rotation. We get a shape, and then we ask, which shape is basically the same shape but rotated?

4:44 Another interesting task is perspective taking. I am asking, where the dog What What is, for example, the angle between the bat and the dog? And then, the question is whether such model can answer that. Another question is maze completion. We want to ask a model to solve this maze. And then, another interesting task is shortcut discovery. In shortcut discovery, basically, you show a model a walk-through through a place, but you don't go in the shortest path. And then, you ask the model to go in the shortest path, and you check whether the model is capable to do that. So, all of these are very important task in in a in spatial cognition. The interesting thing is that 2 years ago, all the models were very bad. If you would take the frontier models, like GPT and other frontier models, they would act really bad. But the interesting thing is that once you start asking the question, you start to see improvement throughout time. So, this work that was 2 years ago, you get that frontier models are quite close to chance. They're a bit better.

5:57 January this year, there is was an interesting work called Baby Vision. They made a similar exploration, and then they found that on January this year, the frontier models were basically at the level of 3-year-old child. So, from models that don't know anything, we grew up to 3 years. So, the interesting thing is that you see that once question is asked, sometimes it's more important to ask the question. People start to pursue that, and in 1 and 1/2 year, we got to 3 years.

6:27 Which is nice to grow to become 3 years old in 1 and 1/2 year. But usually we want to grow slower. Another interesting work which is asked compared how much time it takes for a human to solve the problem, and then checked compared this time to what the models can do. And basically they found that current frontier models, and this happened in February 2026, can solve special cognition tasks that take for us as human 10 seconds.

7:04 So, if you think about it, like mental rotation, maze solving, it take us sometimes to solve. So, these models are able to solve only the tasks that we are able to solve in 10 seconds. Which is not so great, but better than what we had before. And then, if we look at the task that I talked before, so here we can see the state of advance since the initial work I mentioned by my colleagues was published 2 years ago till today. So, 2 years ago, for example, in perspective taking, the best models were close to chance.

7:44 Two years after now, we can see that for example, Opus 4.6 and Gemini 3.1 are basically almost close to 100%. Very impressive improvement in just two years. In mass completion, we are still not so great, but just three months ago from close to chance, we see that for example, GPT 5.4 became very got a great leap. And the same happened with shorter discovery. So we see that great improvement in these models. And then the question is, why this happens?

8:17 So in a recent work that I that in a work that we I have done with my colleagues at Apple, we asked the question of whether multimodal models imagine electric ship, and the answer is yes. So I don't have much time. I will just mention briefly and then you can look at the paper. Basically, what we did, we studied various spatial cognition tasks, and we studied them in an open-loop case. Open-loop case mean that we just show a model the first frame, and then we ask the model to try to solve the game by giving us what what step it should do.

8:58 And the interesting thing is that we found that if we look at the model that is not generating any image, it's just it can receive images, but cannot generate image, and basically we probe the weights, we can we can take a transformer, attach to the weights, probe it. The interesting thing that we found is that the model is imagining internally how to solve the problem. So in different tasks, for example, if we ask if two shapes have the same chirality, we basically able to see that the model in its internal weights is reconstructing the visual thing that we would do. If we would look at two shapes and want to understand if we rotate them they become the same. So, we found that the model is doing it in its internal weights. And maybe this is the reason for the great improvement that we see that model started to imagine.

9:54 And the same happened, for example, if you want to assemble characters. So, we can see that models in their internal weights assemble characters. And if we see that a model that is not able to generate visual thing is able in internal way to do that. So, what we done, we added the ability to the model to imagine. We supervised it to imagine and suddenly we saw we have seen improvement in performance by adding this imagination ability.

10:24 So, we have some more results. Because of time I will skip them. But the interesting point is or there are two points I want to mention. One is asking the right question is very important to advance science. Second thing is that if we think about multimodal models, it's very important for the models not that to focus on text, but also to focus on generating the visual part and it helped the model to improve a lot. Thank you very much.