autodiff

autodiff, keyword, block

The autodiff keyword introduces a block intended for a stochastic gradient descent. The logic within the autodiff block is automatically differentiated. The keyword is followed by the name of the observation table. The runtime uses Adam for parameter updates.

epochs = 500
const lr = 0.1

autodiff Scalar epochs: epochs learningRate: lr with
  params a auto
  s = a * a
  return s

show scalar "" with a // 0.00

You can override the initializer with auto(mean, stdev):

table T = with
  [| as X, as Y |]
  [| 1, 3.1 |]
  [| 2, 3.9 |]
 
autodiff T with
  params a auto(1.0, 0.1)
  params b auto(2.5, 0.01)
  Delta = a * T.X + b - T.Y
  return Delta^2

The option epochs is optional; its default value is 10. It accepts a scalar number, so it can be provided through a variable.

The option learningRate is optional; its default value is 0.01. It requires a constant scalar value, but you can assign it through a const variable.

The observation table can be Scalar. Unlike each, autodiff does not accept a scan option.

All params must belong to small tables. The return statement must provide at least one scalar loss; extra return values are treated as metrics and can be named with name: value.

Zedfunc values can be used inside an autodiff block (for example via valueAt), but parameters cannot be used to build a zedfunc (linear(a) yields no gradient). Only numeric computations that depend on params are required to be differentiable. Computations independent of all parameters may use non-differentiable operations, but the returned loss must still depend on the parameters; otherwise autodiff fails because no parameter receives a gradient.

The gradient mode

The gradient mode is intended to debug a strange behavior in an autodiff block. It returns the gradient of the parameters (not the update!) during one and only one epoch. When a parameter is utilized multiple times within an epoch, such as when it originates from a Scalar/Upstream table, the gradient is accumulated across these occurrences, and the sum of the gradients is returned.

autodiff Scalar mode:"gradient" with
  params X auto(1,0)

  Loss = X * X
  return Loss

show scalar "d(Loss)/d(X) = X * X" with X

In the script above, the parameter is not updated but stores the computed gradient through the one and only one “epoch”.

Lookups between full tables are supported inside autodiff blocks, and the gradient flows back to the looked-up vector.

table F1[kp1] = with
  [| as kp1, as k2, as X |]
  [| "a", "1", 1.0 |]
  [| "b", "3", 2.0 |]
  [| "c", "3", 3.0 |]
  [| "d", "2", 4.0 |]

table F2[kp2] = with
  [| as kp2, as mu |]
  [| "1", 1.0 |]
  [| "2", 2.0 |]
  [| "3", 3.0 |]

autodiff Scalar mode:"gradient" with
  params F2.mu
  F1.mu = F2.mu[F1.k2]
  loss = sum(F1.mu * F1.X)
  return loss

show table "Gradient" with
  F2.kp2
  F2.mu

This outputs the following table:

The parallel mode

The execution of the autodiff block can be automatically parallelized. This execution mode is intended for larger datasets. This parallelization typically incurs a modest decrease of convergence speed - counted in epochs - in exchange for a faster wall-clock execution.

autodiff Scalar epochs: 500 learningRate: 0.1 mode: "parallel" with
  params a auto
  s = a * a
  return s

show scalar "" with a // 0.00

In the script above, the mode option is set to parallel.

The batch mode

For datasets with high dissimilarities, batch processing may stabilize the learning process. It updates the model’s parameters using a batch of observations instead of an individual observation. This method may require more iterations to converge but is useful when some observations differ significantly from the others.

table Obs = extend.range(10000)

lambda = 10
Obs.Y = random.poisson(lambda into Obs)

autodiff Obs batch:100 with
  params lambdaEstimation auto(9.5 ,0)
  return -loglikelihood.poisson(lambdaEstimation, Obs.Y)
 
show summary "Regressed Poisson distribution (batch)" with lambdaEstimation // 10.02


autodiff Obs with
  params lambdaEstimation auto(9.5 ,0)

  return -loglikelihood.poisson(lambdaEstimation, Obs.Y)
 
show summary "Regressed Poisson distribution (no batch)" with lambdaEstimation // 9.90

In the script above, we estimate the poisson parameter. The autodiff block with batch mode is more accurate than the classical one.

autodiff, keyword, pure function option

The autodiff keyword indicates that the pure function can be executed inside an autodiff block

def autodiff pure mySquare(x: number) with 
  return x * x

autodiff Scalar epochs: 500 with
  params a auto
  return mySquare(a)

show scalar "" with a // 0.00

In the present reference documentation, pure functions that are part of the Envision standard library and that can be executed inside an autodiff block are marked as autodiff.