Large Language Models: Basics and Practice

# Basics of Large Language Models: From Fundamentals to Practice

A language model estimates the probability of the next token given the context: P(w_t | w_1, ..., w_{t-1}). This is the core principle of all modern systems, including LLMs.

Development has gone through three stages:

• Statistical n-grams: counting frequencies of 2–3 word sequences. Fast, but they ignore long-range context and semantics.
• Vector embeddings (word2vec, GloVe): words as points in multidimensional space, where distance reflects semantic similarity.
• Transformers: the attention mechanism allows considering connections between distant tokens, enabling deep text understanding.

Transformers form the basis of LLMs capable of generating coherent text.

Scale of Large Language Models

LLMs differ in scale: hundreds of billions of tokens in training data, parameters from 7B to 175B. Parameters are the neural network weights that capture statistical language patterns.

7B–14B models can run locally in quantized form on a GPU. Larger ones require clusters. At scale, LLMs capture not only linguistics but also encoded knowledge from texts—from facts to logic. These are foundation models for chatbots, analytics, and generation.

Instruct and Conversational LLMs

Base LLMs predict tokens but don't always follow tasks. Instruct models are fine-tuned on instruction-response pairs with RLHF: human evaluations shape preferences for politeness and consistency.

Result: conversational systems like ChatGPT, Claude, Gemini. They clarify queries, structure responses, and explain reasoning.

Tokenization and Context Window

LLMs work with tokens—subword units (words, morphemes, punctuation). The tokenizer builds a vocabulary from the corpus, merging frequent sequences.

Process:

• Text → tokens.
• Tokens → embeddings.
• Input to model.

Context window—token limit (e.g., 128k). API costs and generation are charged by tokens. Word length ≠ tokens.

Key Generation Parameters

• Temperature: controls randomness. Low (0.1–0.3)—deterministic responses. High (0.8+)—creativity, but risk of hallucinations.
• Max tokens: output limit, prevents infinite generation.

Parameters are tuned for tasks: stability in production, diversity in brainstorming.

Multimodality in LLMs

Multimodal models integrate text, images, audio, video. Tokens from different modalities are projected into a shared space.

Examples: GPT-4o, Gemini, Claude 3. Capabilities:

• Image descriptions.
• Chart/table analysis.
• Responses to visual queries.

This expands LLMs into universal systems.

Practice: Running Qwen in Google Colab

For experiments, use open models. Qwen2.5-3B-Instruct—text-only, Qwen2.5-VL-3B-Instruct—multimodal. Quality is lower than proprietary ones, but sufficient for prototypes.

Text Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import textwrap

model_name = "Qwen/Qwen2.5-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = "Explain prostymi slovami, that takoe bolshaya yazykovaya model."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=0.7
)

generated_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
answer = tokenizer.decode(generated_tokens, skip_special_tokens=True)
print()
print(textwrap.fill(answer.strip(), width=80))

The output demonstrates a coherent explanation despite only 3B parameters.

Multimodal Model

Similarly, load Qwen2.5-VL-3B-Instruct and pass images as input.

Key Takeaways

• LLMs evolved from n-grams to transformers with attention.
• Scale (parameters, data) yields emergent abilities and knowledge.
• Instruct + RLHF ensure task adherence.
• Tokens define limits: context window, costs.
• Multimodality unifies modalities into a single space.

— Editorial Team