YOLOE: object detection without fine-tuning

YOLOE: A Universal Object Detector with Text and Visual Prompts

YOLOE enables the detection of arbitrary objects in images using text descriptions or visual examples instead of fixed classes. The backbone and PAN are borrowed from standard YOLO, but the detector head compares object embeddings with vectors from a text encoder. The RepRTA module refines text embeddings and integrates them into the model through re-parameterization, maintaining inference speed without overhead.

The model is initialized from pre-trained weights, with classes set dynamically:

from ultralytics import YOLOE

model = YOLOE("yoloe-26m-seg.pt")
model.set_classes(["person", "bus"])
results = model.predict("path/to/image.jpg")

Architecture and Key Modules

Standard YOLO is limited to classes from the training dataset. In YOLOE, class embeddings are generated by a text encoder before inference and embedded into the weights. This eliminates the need for dataset collection, labeling, and training iterations.

Additionally, visual prompts are supported via SAVPE (Semantic-Activated Visual Prompt Encoder). The user provides a reference image with bounding boxes, and the model searches for similar objects:

import numpy as np
from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-26m-seg.pt")
visual_prompts = dict(
    bboxes=np.array([
        [221.52, 405.8, 344.98, 857.54],  # Person
        [120, 425, 160, 445],              # Bus
    ]),
    cls=np.array([0, 1]),
)
results = model.predict(
    "ultralytics/assets/bus.jpg",
    refer_image="reference.jpg",
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
)
results[0].show()

The no-prompt mode uses a built-in dictionary of 1200+ categories:

from ultralytics import YOLOE

model = YOLOE("yoloe-26m-seg-pf.pt")
results = model.predict("path/to/image.jpg")
results[0].show()

Application Scenarios

YOLOE is effective for tasks with clearly describable objects. Here are typical use cases:

• Labeling control: Detection of QR codes, damaged labels, or packaging orientation on a conveyor belt.
• Zone monitoring: Searching for foreign objects or missing tools.
• Warehouse inventory: Counting and classifying changing products.
• Visual inspection: Scratches, chips, or deformations on surfaces.

In these scenarios, prompts replace datasets, speeding up development.

Limitations and Approach Selection

The model falls short in tasks with visually similar classes, specific objects, or requirements for >95% accuracy. Distinguishing fine details or unseen objects requires fine-tuning.

Strategy: Start with YOLOE for a baseline assessment; if insufficient, refine encoders or switch to classical training.

Key Points

• Dynamic classes: Text or visual prompts instead of fixed labels.
• Speed retention: RepRTA re-parameterizes embeddings, eliminating runtime overhead.
• Flexibility: SAVPE for visual examples, built-in dictionary for zero-prompt.
• Practicality: Suitable for industrial tasks with changing objects.
• Limits: Not for high-precision on exotic data.

— Editorial Team