In their own words.
Our plain version above is ours. This is the official or expert wording we based it on.
“The process of reducing the size of one model (known as the teacher) into a smaller model”
Related words.
Fine-tuning
Extra training given to an already-trained model to make it better at a particular job, style or subject.
Parameters (weights)
The numbers inside a model that are adjusted during training and hold what it has learned. Big models have hundreds of billions or more.
Training data
The text, images, code and other material a model learns from during training.
Training run
One long job, often lasting weeks to months on thousands of chips, in which a model learns from its training data. Its size is usually measured in FLOP of compute.
Compute
Computing power: the chips, and the amount of calculation they do, used to train and run AI. It is often counted in FLOP.
FLOP (floating-point operation)
One basic arithmetic step on decimal numbers. The total FLOP used to train a model (e.g. 10^25) is a standard way to say how big a training run was.