TRIBE v2 predicts brain reactions from video

# TRIBE v2: Predicting Brain BOLD Signals from Video and Audio

The TRIBE v2 model from the Brain & AI team at Meta FAIR generates maps of the brain's hemodynamic response to visual and auditory stimuli. The input is video: the model extracts images, sound, and text, processes them through V-JEPA 2, Wav2Vec-BERT, and Llama 3.2, then a transformer predicts BOLD signals across the entire cortex at ~20,000 vertices resolution. This enables simulating neuroscience experiments in silico, avoiding scanning live subjects.

Multimodal Model Architecture

TRIBE v2 integrates three modalities:

• Video: V-JEPA 2 extracts spatio-temporal features.
• Audio: Wav2Vec-BERT encodes acoustic patterns.
• Text: Llama 3.2 processes subtitles or transcripts.

The combined embeddings pass through a transformer that maps them to voxel-wise fMRI predictions. The model doesn't require fine-tuning for new subjects and generalizes to previously unseen individuals.

Training Data and Scaling

The first version of TRIBE was trained on 4 subjects watching the series Friends (6 seasons) and movies, with the cortex divided into 1000 regions. TRIBE v2 uses >500 hours of fMRI from >700 volunteers. This increased the resolution to 20k vertices and prediction accuracy.

Key improvements:

• Dataset scale: from hours to hundreds of hours of recordings.
• Resolution: from 1k to 20k cortical vertices.
• Generalization: works on new subjects without retraining.
• Multimodality: video + audio + text for realistic stimuli.

Applications in Neuroscience and BCI

The model speeds up stimulus testing: from verifying visual perception to developing brain-computer interfaces (BCI). In clinical practice, it analyzes deviations in cortical activation in neurological disorders. Researchers get a tool for hypothesis testing without expensive equipment.

TRIBE v2 focuses on hemodynamics (BOLD), not thoughts or emotions, making its predictions reproducible and comparable to real fMRI.

Key Takeaways

• TRIBE v2 predicts voxel-level BOLD signals from video without real scanning.
• Trained on 500+ hours of fMRI from 700+ subjects, 20k vertex resolution.
• Multimodal: V-JEPA 2 (video), Wav2Vec-BERT (audio), Llama 3.2 (text).
• Available under a non-commercial license for researchers.
• Generalizes to new people without fine-tuning.

— Editorial Team