A model’s training examples reflect choices about collection, selection, and representation. For a supervised task, labels also define the distinctions the model is being asked to learn.
Consider whether the examples cover the situations in which the system will be used. A collection can be large while leaving important cases underrepresented. Inconsistent labels can make the intended task unclear even when the inputs themselves are plentiful.
When evaluating a system, ask how the task and data relate to the real use case. The amount of data is one part of the story; its relevance, quality, and coverage are others. Those questions help connect model performance with the work it is expected to support.
Bring the idea into a day.
Consider a collection of sample labels that covers only easy cases. An unfamiliar input may expose a boundary the examples never explained.
Another angle on the story.
A small trial should have a clear stopping point. Decide which uncertainty the tool can help explore and what observation would answer the next question.
- Ask which situations the examples cover.
- Look for a clear labeling rule.
- Connect the training task with the intended use.
Follow a related question
Choose the details that distinguish your files.
Filenames that carry contextName the group behind the percentage.
The denominator changes the storyKeep learning
Related background to continue exploring this subject.
Google: an introduction to language models NIST: AI risk management framework