Visual question answering (VQA) is the task of answering a natural-language question about an image. Given a photograph and a question such as “How many people are wearing hats?” or “What is the man holding?”, a VQA system produces a short answer, which may be a word, a number or a phrase. The task requires a combination of abilities: perceiving objects and attributes in the picture, understanding the question, and reasoning about the two together. VQA is a central benchmark for multimodal AI and closely connected to image captioning, scene understanding and natural language understanding.
Overview
VQA became a prominent research problem around 2015 because it tests visual and linguistic understanding at once, and because an answer can be checked more easily than a free-form description. Questions can be of many kinds.
- Recognition. “What animal is this?”
- Attributes. “What color is the umbrella?”
- Counting. “How many chairs are there?”
- Spatial relations. “What is to the left of the lamp?”
- Activity. “What is the woman doing?”
- Reading text. “What does the sign say?”
- Knowledge and reasoning. “Why might these people be wearing coats?” or “Which of these foods is usually eaten for breakfast?”
- Yes or no. Binary questions about the content.
Answers can be chosen from a fixed list of common answers (a classification setting), or generated freely as text, which is the approach taken by modern generative models. Some versions use multiple-choice formats, and others extend the idea to video question answering, where the system must reason over time.
History
Before VQA, the idea of answering questions about pictures appeared in early work on question answering for restricted domains. In 2015, Antol and colleagues released the VQA data set of about 250,000 images and 760,000 open-ended questions, with ten human answers collected per question, which defined the standard task and metric [1]. Early models combined image features from a convolutional network with a question vector from a recurrent network and classified over possible answers. Researchers soon found that models could get high scores by exploiting language priors: guessing “two” for “how many” or “yes” for yes-no questions, without really looking at the image. VQA v2 responded by pairing each question with two similar images that have different answers, which forces models to use the picture [2]. Attention mechanisms let the model focus on question-relevant regions, and later transformer models pretrained on large collections of image-text pairs improved results markedly. Today, general vision-language models answer visual questions in open-ended dialogue.
How VQA works
Most systems have the following components.
- Image encoder. A vision model, such as a convolutional network, an object-detection-based encoder or a vision transformer, converts the image into features for regions or patches.
- Question encoder. A language model converts the question into a representation that captures its meaning.
- Multimodal fusion. The two are combined, often with cross-attention, which lets question words attend to image regions and the reverse; see self-attention.
- Answer prediction. The system selects an answer from a list or generates it word by word.
In recent systems, an image encoder is connected to a large language model through a learned projection, and the whole model is fine-tuned with instruction tuning on image-question-answer examples. Such models inherit the language model’s world knowledge and reasoning, which helps with questions that need outside facts. Some use external tools, such as optical character recognition for text in images, or retrieval of facts from a knowledge base.
Data sets and evaluation
Standard benchmarks include the VQA and VQA v2 collections, GQA, which tests compositional reasoning with questions generated from scene graphs, OK-VQA, which requires outside knowledge, TextVQA, which needs reading text in images, and VizWiz, which contains questions asked by blind users about photos they took, often blurry or badly framed [3]. The VQA metric scores an answer by how many of the ten human annotators gave it, so partly agreed answers get partial credit. Newer evaluations for general multimodal models test college-level questions, charts, diagrams and documents. Scores on benchmarks can overstate real ability, because of dataset biases and possible overlap between test items and training data.
Applications
- Accessibility. Letting blind and low-vision users ask about their surroundings, labels and documents. VizWiz was designed for this use [3].
- Image and document analysis. Asking questions about charts, forms, receipts and slides.
- Education. Tutoring from diagrams, textbook figures and science images.
- Healthcare. Asking questions about radiology or pathology images, an area that needs rigorous validation and clinician oversight.
- E-commerce and customer support. Answering questions about product photos, or helping users troubleshoot from a picture.
- Robotics and agents. Querying what a robot sees, as part of robotic perception and computer-use AI.
- Search and retrieval. Conversational image search; see visual search.
- Content moderation and fact-checking. Examining images in context.
Challenges
- Language bias. Models can answer from question patterns without looking at the image.
- Hallucination. Generative models may state details that are not visible, a form of hallucination.
- Counting and spatial reasoning. Models still make errors on exact counts, small objects and relative positions.
- Reading and fine detail. Small text and dense charts remain difficult.
- Knowledge and common sense. Some questions need background facts or reasoning beyond what is in the picture.
- Robustness. Image quality, unusual framing and rephrased questions can change answers.
- Fairness and privacy. Questions about people can elicit sensitive inferences, such as guessing identity or health, which systems should decline.
- Evaluation. Open-ended answers can be correct in different words, which makes scoring hard.
Reasoning and compositional questions
Some questions need several steps. To answer “Is the cup to the right of the laptop the same color as the book?”, a system must find the laptop, locate the cup on its right, find the book and compare colors. Datasets such as GQA were built to test this kind of compositional reasoning. Approaches include neural module networks, which assemble small specialized modules for each step, and prompting methods in which a language model writes out its reasoning before answering, related to chain-of-thought prompting. Evaluations that ask models to explain their answer, or to point to the region they used, help show whether the reasoning is genuine.
Document and chart question answering
A large share of practical visual questions concern documents, screenshots, charts and tables rather than natural photographs. Here the system must read text, understand layout and interpret graphical conventions such as axes and legends. Specialized models combine text, position and image information, and general vision-language models now handle many such tasks directly. Accuracy on numbers is a recurring weak point, so important figures should be checked against the source.
Using VQA responsibly
A useful system should say when it is unsure, decline questions that would require guessing sensitive attributes of people, and avoid presenting uncertain readings of medical images or safety-relevant scenes as fact. Developers should test on realistic inputs from the target users, such as photos taken by people with low vision, and on varied wording, and they should give users a way to check the answer, for instance by highlighting the part of the image it relies on.
FAQs
What is visual question answering? It is the task of answering a natural-language question about an image using both visual and language understanding.
How is VQA different from image captioning? Captioning produces a general description. VQA answers a specific question, which usually needs focused attention on part of the image.
What is the VQA data set? It is a benchmark released in 2015 with about 250,000 images and 760,000 questions, each with human-provided answers.
Why is language bias a problem in VQA? Models can score well by guessing common answers from the question alone, which hides whether they actually understand the image.
What is VQA used for? It is used in accessibility tools, document and chart analysis, education, customer support and multimodal assistants.
More in Computer Vision
Key facts
- The VQA data set, introduced by Antol and colleagues in 2015, contains about 250,000 images and 760,000 open-ended questions with human answers1.
- The VQA v2 data set balanced questions with pairs of similar images that have different answers to reduce language-only guessing2.
- The VizWiz data set, built from questions asked by blind people about photos they took, was created to study real-world visual question answering for accessibility3.
Related concepts
Quick checkWhat is visual question answering?Show answer
Visual question answering (VQA) is the task of giving a correct natural-language answer to a question about a given image, such as how many people are in a photo or what color a car is, by combining image understanding with language understanding.
Sources
- Stanislaw Antol et al., VQA: Visual Question Answering, ICCV 2015, https://arxiv.org/abs/1505.00468
- Yash Goyal et al., Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, https://arxiv.org/abs/1612.00837
- Danna Gurari et al., VizWiz Grand Challenge: Answering Visual Questions from Blind People, https://arxiv.org/abs/1802.08218
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
