LLM engineering · Python

Distill

A classification task that runs on an LLM API costs money on every request. Here GPT-3.5 Turbo's recorded labels for IMDB reviews train two small students, TF-IDF and a sentence-embedding model, and the students are measured on accuracy, robustness to typos and dialect, speed on two CPU threads, and cost. Change the prices and volume below to see when owning the student pays.

When does it pay?Code and method

Accuracy

Test reviews the teacher never labelled for training, and 580 variants of them with typos, dialect and swapped names.

Learning curve

The embedding student trained on fewer of the teacher's labels. Each review brings its variants.

When does owning the student pay?

The teacher costs the same for every request. The student costs a machine a month, as many as the peak load needs, plus labelling its training set once, spread over a year. Defaults are the bench's: list prices when HELM ran GPT-3.5 Turbo and an AWS c7i.large on demand.