Multimodal Vision-Language Models
An image and the sentence describing it are, on the surface, completely different kinds of data — a grid of pixels versus a sequence of tokens. CLIP's contribution was to train both an image encoder and a text encoder so that matching pairs land in the same embedding space, close together — and the moment that works, classification becomes a search problem, with no fixed label set required.