Skip to main content

3 docs tagged with "data"

View all tags

Data Pipelines and Contracts

The model did not break. The upstream team renamed a column, and the model kept running — silently feeding a now-empty (or misaligned) feature into every prediction, with no error, no crash, and no obvious symptom until someone notices the metrics have quietly degraded.

Data Preprocessing and Features

Preprocessing decides more of a model's final performance than the choice of algorithm does. It also has to be treated as part of the model, not a one-off step: whatever transformation is applied to training data must travel with the model into production and be fitted only on training data, never on validation or test data (see Train/Validation/Test Splits).

Datasets and DataLoaders

A GPU can process a batch of data in milliseconds — and then sit idle for seconds waiting for the next batch to be loaded and preprocessed from disk. The input pipeline is where a surprising fraction of real training time actually goes, and getting it wrong silently turns an expensive GPU into an expensive way to wait for a CPU.