Deep Diamond - Deep Learning in Clojure is Fast, Simpler than Keras

Need help with your custom Clojure software? I'm open to (selected) contract work.

September 5, 2020

Please share: .

These books fund my work! Please check them out.

through the direct equivalent of a fine Convolutional network example in Keras.

Good News: Deep Diamond() preview release is in Clojars, and is already quite useful! And fast! It is yet to be fully polished, but you can try it now and, I hope, you'll like it.

It now covers the functionality that is being explained from scratch in the books that I'm writing. Convolutions work, too; at the speed of Road Runner!

In accordance with my philosophy, "less talk, more walk", I introduce Deep Diamond through the direct equivalent of this fine MNIST CNN example in Keras.

Specify the network blueprint

We specify the network by plain Clojure vectors and functions, and create the blueprint. No need for special compilers and whatnot. The structure of internal parts would be picked up automatically, or we can specify these explicitly. 

(def net-spec [(convo [32] [3 3] :relu)
               (convo [64] [3 3] :relu)
               (pooling [2 2] :max)
               (dropout)
               (dense [128] :relu)
               (dropout)
               (dense [10] :softmax)])

(defonce net-bp
  (network (desc [128 1 28 28] :float :nchw)
           net-spec))

Create the network

The blueprint is a Clojure function that can instantiate the network object that holds the parameter tensors that the network should learn by using one of the built-in optimization algorithms. In this case, I'll use adaptive moments, :adam. Xavier initialization is, again, a plain function that initializes the network with appropriate weights. 

(defonce net (init! (net-bp :adam)))

That's it! The network is ready to learn.

Train the network on MNIST data (CPU)

The original MNIST data is distributed through four binary files that you can download here. To demonstrate how nice Clojure is, I'm not using any special MNIST-specific code that is magically imported from the framework's model Zoo. The complete code, from scratch, is at the end of the article (I'm just pushing it there so it doesn't steal the spotlight :). 

(time (train net train-images y-train :crossentropy 12 []))

The network learns in mini-batches of 128 images of the total of 60000, with adaptive moments, through 12 full epochs. That makes 5625 forward/backward update cycles.

The total time for that on my old 2013. i7-4790k CPU is: 368 seconds. 6 minutes for 6000 cycles. A thousand cycles per minute.

Isn't that a lot? You should try and run this in Keras with TensorFlow, and see that we got a pretty nice performance! (I'll publish some comparisons soon, and in the meantime you can try for yourself!).

Has it learned anything?

See the metrics: 

(->> (infer net test-images)
     (dec-categories)
     (classification-metrics test-labels-float)
     :metrics)

{:accuracy 0.9919,
 :f1 0.9918743606319073,
 :ba 0.9954941141884774,
 :sensitivity 0.9918944358825683,
 :specificity 0.9990937924943865,
 :precision 0.9918542861938476,
 :fall-out 9.062075056135655E-4}

Accuracy is 99.2% which is in the ballpark of what the Keras example gives.

GPU

Want to go faster? No problem, Deep Diamond supports GPU, in the same process, at the same time, with the same code! 

(defonce gpu (cudnn-factory))

(def gpu-net-bp (network gpu
                         (desc [128 1 28 28] :float :nchw)
                         net-spec))

(defonce gpu-net (init! (gpu-net-bp :adam)))

(def gpu-x-train
  (transfer! train-images (tensor gpu [60000 1 28 28] :float :nchw)))

(def gpu-y-train
 (transfer! y-train (tensor gpu [60000 10] :float :nc)))

(time (train gpu-net gpu-x-train gpu-y-train :crossentropy 12 []))

Elapsed time? 20 seconds on my Nvidia GTX 1080Ti (which is a few generations old)!

The books

Should I mention that the book Deep Learning for Programmers: An Interactive Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure teaches the nuts and bolts of neural networks and deep learning by showing you how Deep Diamond is built, from scratch? In interactive sessions. Each line of code can be executed and the results inspected in the plain Clojure REPL. The best way to master something is to build it yourself!

It' simple. But fast and powerful!

Please subscribe, read the drafts, get the full book soon, and support my work on this free open source library.

Appendix: Reading, encoding, and decoding data

The code that reads the raw image data and converts it to proper tensors should go up in the sequence of execution, but is not that interesting. 

(defonce train-images-file (random-access "data/mnist/train-images.idx3-ubyte"))
(defonce train-labels-file (random-access "data/mnist/train-labels.idx1-ubyte"))
(defonce test-images-file (random-access "data/mnist/t10k-images.idx3-ubyte"))
(defonce test-labels-file (random-access "data/mnist/t10k-labels.idx1-ubyte"))

(defonce train-images
  (map-tensor train-images-file [60000 1 28 28] :uint8 :nchw :read 16))
(defonce train-labels
  (map-tensor train-labels-file [60000] :uint8 :x :read 8))
(defonce test-images
  (map-tensor test-images-file [10000 1 28 28] :uint8 :nchw :read 16))
(defonce test-labels
 (map-tensor test-labels-file [10000] :uint8 :x :read 8))

(defn enc-categories [val-tz]
  (let [val-vector (view-vctr val-tz)]
    (let-release [cat-tz (tensor val-tz [(first (shape val-tz)) (inc (long (amax val-vector)))] :float :nc )
                  cat-matrix (view-ge (view-vctr cat-tz) (second (shape cat-tz)) (first (shape cat-tz)))]
      (dotimes [j (dim val-vector)]
        (entry! cat-matrix (entry val-vector j) j 1.0))
      cat-tz)))

(defn dec-categories [cat-tz]
  (let [cat-matrix (view-ge (view-vctr cat-tz) (second (shape cat-tz)) (first (shape cat-tz)))]
    (let-release [val-tz (tensor cat-tz [(first (shape cat-tz))] :float :x)
                  val-vector (view-vctr val-tz)]
      (dotimes [j (dim val-vector)]
        (entry! val-vector j (imax (col cat-matrix j))))
      val-tz)))

(defonce train-labels-float (transfer! train-labels (tensor [60000] :float :x)))
(defonce y-train (enc-categories train-labels-float))
(defonce test-labels-float (transfer! test-labels (tensor [10000] :float :x)))
(defonce y-test (enc-categories test-labels-float))
Deep Diamond - Deep Learning in Clojure is Fast, Simpler than Keras - September 5, 2020 - Dragan Djuric