Skip to main content

3 docs tagged with "preprocessing"

View all tags

Data Preprocessing and Features

Preprocessing decides more of a model's final performance than the choice of algorithm does. It also has to be treated as part of the model, not a one-off step: whatever transformation is applied to training data must travel with the model into production and be fitted only on training data, never on validation or test data (see Train/Validation/Test Splits).

Images as Tensors

Before any model sees a pixel, an image is a grid of numbers — and the conventions for arranging and scaling those numbers disagree between nearly every library you'll touch. Almost every mysterious vision bug traces back to one of these conventions being silently wrong: a colour channel swapped, a value range mismatched, an axis order flipped.

Text Preprocessing and Tokenization

Models don't see text — they see a sequence of integers, and every choice about how those integers are assigned decides what the model can and can never learn. A tokeniser that splits "unbelievable" into three pieces lets the model reuse what it learned about "un-" and "-able" elsewhere; a tokeniser that treats it as one opaque unit has no such option.