Training methods

Noise speedrun reports transformer training target reached in 1.6 seconds

industriaalist reported that the Noise speedrun reduced the time to pretrain a transformer from 1.76 hours to 1.6 seconds over 10 days using training without backpropagation. The account described a 4,000x speedup; cs_serdar subsequently announced a reset to 30M tokens and a forward-only record of 3.409.

industriaalist cautioned that some techniques behind the 1.6-second result would not scale, and said the dataset would expand from 10M to 30M tokens to discard approaches that overfit it. cs_serdar also announced a track allowing any architecture while retaining the requirement to avoid backpropagation.