Parameter

What Parameters Are

A neural network is built from layers of mathematical operations. The inputs to those operations are your data — a sentence, an image, a prompt. But the operations themselves contain parameters: numerical values (called weights and biases) that determine what the network does with the input.

During training, these parameters are adjusted repeatedly — millions of times — until the network's outputs match the correct answers in the training data. The final set of parameter values is what makes a trained model.

When someone says a model has "7 billion parameters," they mean the network contains 7 billion individual numbers that were all tuned during training.

Why Count Matters

Parameter count is the most commonly cited measure of model size. Broadly speaking:

• More parameters → greater capacity to learn complex patterns → generally more capable on hard tasks
• Fewer parameters → faster inference, lower cost, easier to run locally

GPT-3 had 175 billion parameters. More recent models like Llama 3 come in sizes from 8 billion to 70 billion, allowing developers to choose based on their capability vs. cost tradeoff.

Larger isn't always better for a given task. A 7B model can outperform a 70B model on narrow, well-defined tasks — especially after fine-tuning.

Parameter-Efficient Fine-Tuning

Updating all parameters when fine-tuning a large model is expensive. Techniques like LoRA (Low-Rank Adaptation) update only a small fraction of parameters, dramatically reducing the compute required while preserving most of the fine-tuning benefit. This is why fine-tuning large models has become more accessible.