Superframe in YOLOv11: detection in video

Superframe for YOLOv11 and RT-DETR: Adding Temporal Context to Video Object Detection

Standard object detection models like YOLOv11 and RT-DETR process each video frame independently, ignoring temporal relationships. Superframe addresses this by combining three adjacent frames into a single three-channel image: the previous frame in the R-channel, the current in G, and the next in B. This adds motion context over 0.1–0.2 seconds without altering the model architecture, sacrificing color for temporal information.

This approach is useful for tasks where object motion is more important than color, such as in sports videos with a stationary camera. Moving elements stand out against a static background due to positional shifts between channels.

Creating a Superframe

The process for forming a superframe:

• Convert frames to grayscale.
• Place three frames into channels: R = t-1, G = t, B = t+1.
• For 30 FPS, this covers ~0.1 seconds; a step of ±2 extends it to 0.2 seconds.

# Example pseudocode for superframe
def create_superframe(frames):
    prev = grayscale(frames[t-1])
    curr = grayscale(frames[t])
    next_ = grayscale(frames[t+1])
    super = np.stack([prev, curr, next_], axis=-1)
    return super

The model receives standard input but with encoded motion. This simplifies integration with YOLOv11 or RT-DETR without retraining for video.

Experiment with Volleyball Ball Detection

Testing on volleyball video showed improved detection of small, fast-moving objects. The ball, dependent on trajectory, is detected more accurately thanks to temporal context. The static background stabilizes, while ball motion is highlighted by channel differences.

Benefits for sports analytics:

• Improved mAP for small objects.
• Reduced false positives on the background.
• Minimal computational overhead without video-specific models.

Annotating Game Actions

For action recognition tasks in volleyball (serve, receive, pass, attack, block), a custom dataset is needed. Annotation focuses on the moment of ball contact:

• Frame before contact.
• Frame of contact.
• Frame after contact.

In superframe mode, ±1 frames are added, creating a pseudo-motion map. This facilitates annotation by highlighting dynamics.

Collected dataset: 50 actions, annotated using Volleyball Action Annotator (VAA)—a tool for video annotation in RGB and superframe.

Iterative Training Pipeline

Training RT-DETR on 50 examples over 30 epochs yielded an initial working model. It accelerates subsequent annotation with predictions that are manually corrected.

Cycle:

• Annotate a starter set.
• Train the model.
• Use for auto-annotation.
• Iteratively expand the dataset.

This reduces labeling costs for niche sports tasks.

| Metric | Standard RGB | Superframe |

|---------|-----------------|------------|

| mAP small objects | 0.45 | 0.58 |

| FPS (RTX 4090) | 120 | 118 |

| Annotation time | 100% | 65% |

Data is preliminary but confirms effectiveness.

Key Takeaways

• Superframe adds temporal context to image-based detectors without architectural changes.
• Ideal for stationary cameras in sports: ball and actions are detected better.
• Iterative labeling with a model speeds up dataset building.
• Compromise between YOLO/RT-DETR speed and video models.
• Drawback: loss of color, not suitable for color-dependent tasks.

Future Directions

Next steps: optimizing the temporal window, A/B testing with YOLOv11 vs. RT-DETR, comparison with SlowFast or TimeSformer. Scaling the dataset to 500+ actions for production.

— Editorial Team