PyTorch Hooks for audio from LLM activations

Hacking LLM Activations for Audio Generation Using PyTorch Hooks

In the PaleoSonic Engine experiment, visual patches from SigLIP are directly transformed into MusicGen audio latents. Instead of the usual Image -> Text -> Audio pipeline, a linear layer (nn.Linear) projects 196 patches of size 196 into sound vectors. Models are converted to bfloat16 to run on 16 GB RAM.

The key step is monkey-patching MusicGen's text encoder. The generator expects 196 text tokens but gets image geometry instead. Random initialization of the bridge produces raw noise: harsh static mimicking an architecture clash without fine-tuning.

# Save the original "brain"
original_text_encoder = self.audio_decoder.text_encoder.forward

# Create a fake class
class VisualThoughts:
    def __init__(self, hidden_states):
        self.last_hidden_state = hidden_states
    def __getitem__(self, idx):
        return [self.last_hidden_state][idx]
        
def spoofed_text_encoder(*args, **kwargs):
    # Slip in visual tensors instead of text!
    return VisualThoughts(audio_conditioning)
    
# Infect the generator
self.audio_decoder.text_encoder.forward = spoofed_text_encoder

This approach exposes a raw translation of pixel geometry into acoustics, bypassing text semantics.

PyTorch Hooks for Extracting LLM 'Thoughts' in Real Time

In the second experiment, Neural-Analog Engine uses Qwen2.5-1.5B-Instruct to control a DSP oscillator. A forward hook is registered on layer 15—the hub of abstract representations. Activations are detached, converted to float32, and compressed by averaging over the batch axis.

neural_activations = []

# Spy function to steal thoughts
def steal_thoughts_hook(module, input, output):
    # Grab the current neuron state
    current_thought = output[0].detach().cpu().to(torch.float32).numpy()
    compressed_thought = np.mean(current_thought, axis=1)[0] 
    neural_activations.append(compressed_thought)

# Inject the "needle" into Qwen's 15th layer
hook_handle = model.model.layers[15].register_forward_hook(steal_thoughts_hook)

Raw vectors feed into a NumPy oscillator and SciPy filters. The LLM generates a prompt like "Silence of a frozen quantum star," but the sound is modulated by activation pulses: frequencies, phases, and decay driven by neural values. The result is pure dark ambient without neural audio generators.

Technical Details of Cross-Modal Bridging

• Vision Backbone: google/siglip-base-patch16-224 outputs 196 patches after self-attention.
• Audio Target: facebook/musicgen-small expects text_encoder with hidden_state [1, 196, dim].
• Bridge: nn.Linear(196dim_vision, 196dim_audio) with random init; no fine-tuning to keep the 'grit'.
• Hook Mechanics: register_forward_hook captures output after MLP in layer 15; detach prevents gradients.
• DSP Pipeline: np.sin for oscillators + scipy.signal.butter for low-pass filters; amplitude modulation from activations.

Sound differences:

• PaleoSonic — chaotic noise from latent collisions.
• Neural-Analog — structured ambient from physical waves.

Scaling and Optimizations for Production

Both approaches run in Hugging Face Spaces. For scale:

• Use torch.compile to speed up hooks.
• Quantize Qwen to 4-bit with bitsandbytes to cut memory use.
• Parallelize activations via DataParallel on multiple GPUs.

Potential: Integrate into real-time audio (JACK/WASAPI) for live performances. Experiment with other layers—early ones yield low-level features, later ones semantics.

Key Takeaways

• Monkey patching bypasses generate() without retraining, swapping the encoder on the fly.
• PyTorch forward hooks are standard for interpretability; detach is essential for inference.
• Activation compression (mean axis=1) reduces dimensionality without losing dynamics.
• bfloat16 is crucial for low-RAM setups; avoid float16 on CPU.
• NumPy/SciPy DSP beats torch.nn for simple oscillators.

— Editorial Team