Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Image Captioning

nounComputer Visionalso automatic image captioning, image description generation, alt text generation

In one line

Image captioning is the task of automatically generating a natural-language description of the content of an image.

Image captioning is the task of automatically generating a natural-language description of an image. Given a photograph, a captioning system might write “A brown dog runs across a grassy field with a ball in its mouth.” The task joins two fields of artificial intelligence: computer vision, which has to understand what is in the picture, and natural language generation, which has to express that understanding in fluent text. Image captioning supports accessibility through alt text for people who cannot see images, helps organize and search photo collections, and serves as a building block for broader multimodal AI systems. It is closely related to visual question answering and image-to-text technology.

Overview

A good caption names the main objects, describes their attributes and actions, and captures relationships, such as who is doing what and where. It should be accurate, relevant and readable. A caption is not a complete inventory of the image, because it must decide which details matter. Captions can be of different kinds.

  • Generic captions. Short, one-sentence descriptions of the main content.
  • Dense captions. Descriptions of many separate regions in an image.
  • Detailed or paragraph descriptions. Longer text that covers more content.
  • Controlled or styled captions. Descriptions in a requested tone, such as humorous or factual, or focused on a given aspect.
  • Alt text. Descriptions written specifically for people who use screen readers, which should convey the purpose of the image on a page.

History

Early captioning systems used templates. They detected objects and attributes with computer vision and inserted them into sentence patterns, or retrieved existing captions from similar images. In 2014 and 2015, researchers adapted the encoder-decoder idea from machine translation: a convolutional network encoded the image into a vector, and a recurrent decoder generated the caption word by word. The Show and Tell model from Google is a well-known example [1]. Attention mechanisms let the decoder look at different parts of the image as it wrote each word; see attention mechanism. Transformers then replaced recurrent decoders, and large models pretrained on enormous numbers of image-text pairs produced much more detailed and fluent captions. Today’s vision-language models can caption, answer questions about and reason over images in one system.

How image captioning works

  1. Image encoding. A vision model, such as a convolutional network or vision transformer, converts the image into features, often one vector per region or patch.
  2. Language decoding. A language model generates the caption one token at a time, using the image features and the words written so far.
  3. Attention. At each step, the decoder attends to the most relevant image regions.
  4. Search. Instead of always choosing the most likely word, decoding often uses beam search, which keeps several candidate sentences and picks the best, or sampling for more variety; see temperature.
  5. Training. The model learns from images paired with human captions by predicting each next word. Many systems are first pretrained on large collections of web image-text pairs and then fine-tuned on curated caption data.

Reinforcement learning has been used to optimize captions directly for evaluation metrics, and instruction tuning allows modern models to follow requests such as “describe the image for a blind user.”

Data sets and evaluation

The Microsoft COCO Captions data set, with more than 1.5 million human-written captions for over 330,000 images, is the standard benchmark [2]. Other collections include Flickr30k and large web-scraped sets. Captions are evaluated with automatic metrics that compare a generated sentence with several human references.

  • BLEU and METEOR. Borrowed from machine translation, they count matching words and phrases.
  • ROUGE. Measures overlap, often used in summarization.
  • CIDEr. Weights words by how distinctive they are and measures agreement with consensus human descriptions [3].
  • SPICE. Compares the objects, attributes and relations in the generated and reference captions.
  • Learned metrics. Use image-text models to judge how well a caption matches the picture, and can work without references.

Automatic metrics correlate imperfectly with human judgment, so people are still asked to rate accuracy, completeness and usefulness. A fluent caption can still contain errors.

Applications

  • Accessibility. Generating alt text for images so that blind and low-vision users can understand them, with human review preferred for important content.
  • Search and organization. Making photo libraries and archives searchable with text, related to visual search.
  • Social media and content management. Suggesting descriptions and tags, and supporting moderation.
  • E-commerce. Producing product descriptions from photos.
  • Robotics and navigation aids. Describing surroundings to users or to planning software.
  • Healthcare. Drafting descriptions of medical images, which must be checked by clinicians.
  • Education and journalism. Describing diagrams, charts and photographs.
  • Video. Summarizing and describing clips, an extension linked to text summarization.

Challenges

  • Hallucination. Models sometimes mention objects that are not in the image or get relationships wrong, a form of hallucination.
  • Generic output. Captions can be bland, describing what is obvious and missing the point of the image.
  • Counting and spatial detail. Numbers, left-right positions and small text are often wrong.
  • Context and knowledge. A caption may miss cultural, historical or situational meaning, such as why an image of a flag or a protest matters.
  • Bias and harm. Models can reflect stereotypes about gender, race or occupation in training data, and can mislabel people. Describing people’s identity or attributes needs particular care.
  • Evaluation. Several valid captions exist, which makes single-reference metrics unreliable.
  • Sensitive content. Images may contain private or distressing material.

Writing good alt text

Automatic captions are often used to produce alt text, but accessibility guidelines ask for more than a literal description. Good alt text conveys the purpose of an image in its context: a chart should be described by its main finding, a product photo by the features that matter, and a decorative image may need no description at all. Guidance from accessibility groups suggests keeping descriptions concise, avoiding phrases such as “image of,” and including any text that appears in the picture. Machine-generated captions are a helpful starting point, and a person who knows the context should review them for important pages.

Beyond single sentences

Research has moved from one-line captions toward richer outputs. Dense captioning describes many regions, and grounded captioning links each phrase to a box or mask in the image, which makes errors easier to spot. Video captioning describes events over time, and story-like descriptions connect a sequence of images. Large vision-language models allow follow-up conversation, so a user can ask for more detail about a part of an image, which blurs the line between captioning and visual question answering.

Practical tips

Teams that use captioning in products should test it on their own images, since models trained on everyday photographs may do poorly on diagrams, medical scans or product catalogs. It helps to set rules for what the system must never state, such as guessing a person’s identity, ethnicity or emotions, to measure accuracy on a labeled sample and to offer users a simple way to correct captions. Keeping a record of which descriptions were generated by a model supports transparency and later quality checks.

FAQs

What is image captioning? It is the automatic generation of a text description of an image by an AI system.

How does image captioning work? A vision model encodes the image into features, and a language model uses them to write the description one word at a time.

What is image captioning used for? It is used for accessibility alt text, photo search, content management, e-commerce descriptions and as part of multimodal assistants.

How is image captioning evaluated? With metrics such as BLEU, CIDEr and SPICE that compare generated captions with human references, and with human ratings of accuracy.

Can AI captions be wrong? Yes. Models can invent details, miss important content or reflect bias, so important captions should be reviewed by a person.

Key facts

  • The Show and Tell model, published by Google researchers in 2015, used a convolutional network as an encoder and a recurrent network as a decoder to generate captions1.
  • The Microsoft COCO Captions data set contains over 1.5 million human-written captions for more than 330,000 images2.
  • CIDEr, a metric proposed in 2015, scores captions by their agreement with human consensus descriptions3.

Related concepts

Quick checkWhat is image captioning?Show answer

Image captioning is the task of automatically writing a natural-language sentence that describes what an image shows, combining computer vision to understand the picture with language generation to express it in words.

Sources

  1. Oriol Vinyals et al., Show and Tell: A Neural Image Caption Generator, https://arxiv.org/abs/1411.4555
  2. Xinlei Chen et al., Microsoft COCO Captions: Data Collection and Evaluation Server, https://arxiv.org/abs/1504.00325
  3. Ramakrishna Vedantam, C. Lawrence Zitnick and Devi Parikh, CIDEr: Consensus-based Image Description Evaluation, https://arxiv.org/abs/1411.5726

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue