tfixup_init_

core.nn.tfixup_init_(encoder, scale_base, v_scale_factor)

Scale a Transformer encoder’s weights in place for optimization without warmup.

Sources: Transformers without Tears https://arxiv.org/pdf/1910.05895.pdf T-Fixup http://www.cs.toronto.edu/~mvolkovs/ICML2020_tfixup.pdf

Parameters:

• encoder (TransformerEncoder) – encoder whose layers use the stock nn.TransformerEncoderLayer parameter names (linear1/linear2/self_attn).

• scale_base (float) – T-Fixup base scale, applied as scale_base * N**-0.25.

• v_scale_factor (float) – extra factor on the value weights, on top of scale.

Return type:

None