In their own words.
Our plain version above is ours. This is the official or expert wording we based it on.
“A second, task-specific training pass performed on a pre-trained model to refine its parameters for a specific use case.”
Related words.
Distillation
Training a smaller, cheaper model to copy the behavior of a bigger one, so it runs faster at some loss of ability.
Parameters (weights)
The numbers inside a model that are adjusted during training and hold what it has learned. Big models have hundreds of billions or more.
Training data
The text, images, code and other material a model learns from during training.
Training run
One long job, often lasting weeks to months on thousands of chips, in which a model learns from its training data. Its size is usually measured in FLOP of compute.
Compute
Computing power: the chips, and the amount of calculation they do, used to train and run AI. It is often counted in FLOP.
FLOP (floating-point operation)
One basic arithmetic step on decimal numbers. The total FLOP used to train a model (e.g. 10^25) is a standard way to say how big a training run was.