Deep Learning from Scratch to GPU - 9 - The Activation and its Derivative

Two equations of the four from Part 6, which are relevant for this article, do a similar thing. In both of these formulas, we compute the Haddamard product of the matrix that comes from the next layer (\(\nabla_a C\) and \((w^{l+1})^T \delta^{l+1}\)) and the matrix that is located in the layer at hand (\(\sigma ' (z^L)\)).

\(\delta^L = \nabla_a C \odot \sigma ' (z^L)\).

\(\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma ' (z^L)\).

First, we will handle the \(\nabla_a C\) and \(((w^{l+1})^T \delta^{l+1})\).

This is the key line of code that we'll add to our implementation of the training layer. Both equations will be summed up in this tiny one-liner.

...
(backward [_]
  (mul! (prime activ-fn z) a))
...

In this article, we'll start developing the backward method. The end result will be just a few rather intense lines of Clojure code, in the Neanedrthal fashion that you've probably started getting used to. It could be dense just a bit too much if I had just thrown that at you all at once. I am going to explain the considerations during each step in detail.

Again for reference, I repeat the inference and training layer code that we developed in the last article.

(require '[uncomplicate.commons.core
           :refer [with-release let-release Releaseable release]]
         '[uncomplicate.clojurecuda.core :as cuda
           :refer [current-context default-stream synchronize!]]
         '[uncomplicate.clojurecl.core :as opencl
           :refer [*context* *command-queue* finish!]]
         '[uncomplicate.neanderthal
           [core :refer [mrows dim raw view entry! axpy! copy! scal! transfer!
                         transfer mm! rk! view-ge vctr ge]]
           [vect-math :refer [tanh! linear-frac! sqr! mul! cosh! inv!]]
           [native :refer [fge native-float]]
           [cuda :refer [cuda-float]]
           [opencl :refer [opencl-float]]])
(import 'clojure.lang.IFn)

(defprotocol Parameters
  (weights [this])
  (bias [this]))

(defprotocol ActivationProvider
  (activation-fn [this]))

(deftype FullyConnectedInference [w b activ-fn]
  Releaseable
  (release [_]
    (release w)
    (release b))
  Parameters
  (weights [_] w)
  (bias [_] b)
  ActivationProvider
  (activation-fn [_] (activ-fn))
  IFn
  (invoke [_ x ones a]
    (activ-fn (rk! -1.0 b ones (mm! 1.0 w x 0.0 a)))))

(defn fully-connected [factory activ-fn in-dim out-dim]
  (let-release [w (ge factory out-dim in-dim)
                bias (vctr factory out-dim)]
    (->FullyConnectedInference w bias activ-fn)))

(defprotocol Backprop
  (forward [this])
  (backward [this]))

(defprotocol Transfer
  (input [this])
  (output [this])
  (ones [this]))

(deftype FullyConnectedTraining [w b a-1 z a ones-vctr activ-fn]
  Releaseable
  (release [_]
    (release w)
    (release b)
    (release a-1)
    (release z)
    (release a)
    (release ones))
  Parameters
  (weights [_] w)
  (bias [_] b)
  Transfer
  (input [_] a-1)
  (output [_] a)
  (ones [_] ones-vctr)
  Backprop
  (forward [_]
    (activ-fn (rk! -1.0 b ones (mm! 1.0 w a-1 0.0 z)) a))
  (backward [_]
    (throw (ex-info "TODO"))))

(defn training-layer
  ([inference-layer input ones-vctr]
   (let-release [w (view (weights inference-layer))
                 b (view (bias inference-layer))
                 a-1 (view input)
                 z (ge w (mrows w) (dim ones-vctr))
                 a (raw z)
                 o (view ones-vctr)]
     (->FullyConnectedTraining w b a-1 z a o (activation-fn inference-layer))))
  ([inference-layer previous-backprop]
   (training-layer inference-layer
                   (output previous-backprop)
                   (ones previous-backprop))))

From the definition of the Haddamard product, it is obvious that the dimensions of whatever is coming from the next layer (that was computed in the previous backwards step) is the same as the dimension of z.

Also note that the next layer has the view of a, it's the a-1 matrix in the next layer. The next layer uses a-1 data in the forward pass and it doesn't need it any more.

It is a rather nice coincidence that we can use this reference to receive whatever the next layer computes, \(\nabla_a C\) or \(((w^{l+1})^T \delta^{l+1})\), through this reference.

The first part is solved; we will let the next layer worry about how to compute \(\nabla_a C\) or \(((w^{l+1})^T \delta^{l+1})\) and receive it through a in the current layer when its turn comes during the backward pass.

At first look, this step seems trivial. We have the activation function \(\sigma\), we wisely saved the values of the linear outputs \(z^l\), and we just need to find out which function is the derivative \(\sigma '\) and apply it.

As you probably recall from the article Bias and Activation Function, although we could use any function that is polite enough to be differentiable in the whole domain that interests us, we typically choose one from a very narrow selection of typical ones: ReLU, sigmoid, hyperbolic tangent, and a few variants of these.

The key reason to use only a somehow standard selection is that then we can have their derivatives ready at hand. The reason to use these particular functions is that their derivatives are also easy-ish to implement.

So, given the activation function \(\sigma\), the first task is to be able to get a hold on the function that implements its derivative. We can do this in a few ways in Clojure:

• (1) Offload it to the user: when the layer is constructed, in addition to activation, the user would have to provide activation-prime.
• (2) Define a new type (using deftype) that can calculate a function, its prime, and, if necessary, pre-allocates the working memory
• (3) Provide a zero-argument implementation in the activation function itself, which returns the prime function.

I don't like the first approach, since it puts an unnecessary burden on the user. Besides, they typically will not choose any silly function as activation, but one from the handful that are tried and tested. Therefore, I'd opt for one of the other two. Let me try to implement the last option.

I'll cover sigmoid and hyperbolic tangent; you should be able to work out the details of a few more as an exercise.

(deftype FullyConnectedTraining [w b a-1 z a ones-vctr activ-fn]
  Releaseable
  (release [_]
    (release w)
    (release b)
    (release a-1)
    (release z)
    (release a)
    (release ones-vctr)
    (release activ-fn))
  Parameters
  (weights [_] w)
  (bias [_] b)
  Transfer
  (input [_] a-1)
  (output [_] a)
  (ones [_] ones-vctr)
  Backprop
  (forward [_]
    (activ activ-fn (rk! -1.0 b ones-vctr (mm! 1.0 w a-1 0.0 z)) a))
  (backward [_]
    (mul! (prime activ-fn z) a) ;; z now contains delta^l
    "TODO"))

(defn training-layer
  ([inference-layer input ones-vctr]
   (let-release [w (view (weights inference-layer))
                 b (view (bias inference-layer))
                 a-1 (view input)
                 z (ge w (mrows w) (dim ones-vctr))
                 a (raw z)
                 o (view ones-vctr)]
     (->FullyConnectedTraining w b a-1 z a o ((activation-fn inference-layer) z))))
  ([inference-layer previous-backprop]
   (training-layer inference-layer
                   (output previous-backprop)
                   (ones previous-backprop))))

Let's check it with our old friend, an arbitrary network with two hidden layers.

(with-release [x (ge native-float 2 2 [0.3 0.9 0.3 0.9])
               ones (vctr native-float 1 1)
               layer-1 (fully-connected native-float tanh 2 4)
               layer-2 (fully-connected native-float sigmoid 4 1)
               training-layer-1 (training-layer layer-1 x ones)
               training-layer-2 (training-layer layer-2 training-layer-1)]
  (transfer! [0.3 0.1 0.9 0.0 0.6 2.0 3.7 1.0] (weights layer-1))
  (transfer! [0.7 0.2 1.1 2] (bias layer-1))
  (transfer! [0.75 0.15 0.22 0.33] (weights layer-2))
  (transfer! [0.3] (bias layer-2))
  (forward training-layer-1)
  (forward training-layer-2)
  (backward training-layer-2)
  (backward training-layer-1)
  (transfer (output training-layer-2)))

nil#RealGEMatrix[float, mxn:1x2, layout:column, offset:0]
   ▥       ↓       ↓       ┓
   →       0.44    0.44
   ┗                       ┛

The output is as expected. Nice job. I hope we continue making this work.

Let's see how fast the network is. Does the backward pass, at least the part that we have implemented by now, affect performance?

If you feel that you can afford to help, and wish to donate, I even created a special Starbucks for two tier at patreon.com/draganrocks. You can do something even cooler: adopt a pet function.

So, finally, the implementation of the learning algorithm unwinds. The majority of the structure is in place. What's left for the straightforward backward pass is computing the gradients in respect to \(\nabla_b C^l\) and \(\nabla_w C^l\), \(\nabla_{a-1}\), and updating the weights and bias.

After that, we should take care of wrapping this whole infrastructure into a nice and user-friendly neural networks API. We don't expect our users to assemble all these layers by hand, right?

Then, we can play with some real-world examples, and see which optimizations we can add to our implementation so the learning itself become more robust.

We will? Yes! But, why didn't I show you the result first? Seeing what we are aiming at would be a terrific motivator! Perhaps, but I am not writing this series to serve as an easy entertainment. I want you to think about the details. I want you to imagine what you would like the end result to be. Besides, I am writing this implementation as we go :) I am yet to decide what the final result is going to be, and your suggestions are welcome!

Clojurists Together financially supported writing this series. Big thanks to all Clojurians who contribute, and thank you for reading and discussing this series.