Neural Networks and Multiplication: SwiGLU in Transformers

Why Neural Networks Don't Multiply: From Perceptrons to SwiGLU

Neural networks, despite their success in generating text and images, fundamentally do not perform multiplication of input data. In a perceptron, inputs undergo a linear combination: they are weighted and summed. The formula for a basic neuron is: $$h = \sigma(\sum w_i x_i + b)$$, where there is no product of two variables $x_1 \cdot x_2$.

For a task like calculating cost (price × quantity), the network approximates the result by memorizing patterns but does not compute multiplication directly. Adding hidden layers preserves the principle: at each stage, only sums of weighted signals are involved.

In the first layer:

$$h_1^{(1)} = x_1 w_{11}^{(1)} + x_2 w_{12}^{(1)}$$

$$h_2^{(1)} = x_1 w_{21}^{(1)} + x_2 w_{22}^{(1)}$$

In the second layer:

$$h_1^{(2)} = h_1^{(1)} w_{11}^{(2)} + h_2^{(1)} w_{12}^{(2)}$$

Substitution shows: $x_1$ and $x_2$ remain in sums, their product does not arise. Weights multiply between layers, but inputs do not.

Workarounds: Squares and Precomputations

Engineers add pre-multiplied values ($x_1 x_2$) or squares to the input to simulate multiplication. A school formula allows expressing the product:

$$x y = \frac{1}{2} \left( (x+y)^2 - x^2 - y^2 \right)$$

This reduces multiplication to addition, subtraction, and squaring. A hypothetical activation $f(z) = z^2$ could help, but standard ReLU, Sigmoid, and Tanh do not provide the nonlinearity of squaring.

Practice shows: such workarounds work for demonstrations, but in real-world tasks with many inputs, they require knowing the necessary combinations in advance.

• Precomputations: Feeding $x_1 x_2$ as input.
• Squares: Using $(x+y)^2$, $x^2$, $y^2$.
• Limitations: Does not scale, compromises the universal approximator property.

Transformer Breakthrough: The Role of GLU Variants

The Transformer architecture (2017) introduced attention, but a key step was replacing activations with Gated Linear Units (GLU) in 2020 (Noam Shazeer, "GLU Variants Improve Transformer"). ReGLU, GEGLU, and SwiGLU outperform ReLU in convergence.

| Activation | Error |

|-----------|--------|

| ReLU | 2.45 |

| ReGLU | 2.32 |

| GEGLU | 2.25 |

| SwiGLU | 2.24 |

SwiGLU is preferred due to speed. The formula:

$$\text{SwiGLU}(x) = \text{Swish}(x W + b) \odot (x V + c)$$

Here $\odot$ is element-wise multiplication of two projections. This introduces quadratic dependencies, allowing modeling of $x_1 x_2$ without workarounds. SwiGLU is used in FFN blocks of modern LLMs.

Mechanism:

• Input $x$ is projected into two paths: $xW + b$ and $xV + c$.
• The first passes through Swish ($\text{Swish}(z) = z \sigma(\beta z)$).
• Element-wise multiplication.

Practical Implications for Models

SwiGLU improves training but does not make AI a precise calculator. Generative models are probabilistic: "2×2=4" is a likely token, not a deduction. For complex calculations, LLMs delegate to Python code in a sandbox.

• Pros of GLU: Better convergence, modeling of nonlinearities.
• Cons: The network does not always find the necessary weights automatically.
• Requires large amounts of data.
• Probabilistic nature persists.

Future integrations (WebAssembly in Transformer) will allow running native code within the model.

Key Takeaways

• Classical perceptrons add but do not multiply inputs.
• GLU (SwiGLU) introduces element-wise multiplication of projections for quadratic dependencies.
• Gated units replace ReLU in modern transformers, reducing error by 5–10%.
• For precise calculations, LLMs use external tools.

— Editorial Team