regularizationTerm

regularizationTerm, function

def const autodiff pure regularizationTerm(
  penalty: number,
  loss: number,
  alpha: number): number

Returns a penalty term derived from penalty, scaled as a fraction alpha of loss. The function uses noGrad(loss) to prevent gradients from flowing through loss.

• penalty: the penalty value to scale.
• loss: the base loss value.
• alpha: the relative weight of the penalty versus the loss.

Example

reg = regularizationTerm(0.5, 10, 0.1)
show scalar "Regularization" with reg

This outputs the following scalar:

Remarks

For small penalty values, the function behaves like:

$$\text{regularizationTerm}(penalty, loss, \alpha) = \alpha \times \text{noGrad}(loss) \times \tanh(penalty)$$

For larger penalty values, it switches to a piecewise definition that avoids vanishing gradients from tanh.

See also

• autodiff
• noGrad