My experience with Clojurists Together open source funding

My goal with this round is to implement Recurrent Neural Networks (RNN) support in Deep Diamond. The first month was dedicated to literature review (and other hammock-located work), exploration of OneDNN implementation of RNN layers, and implementation of RNN prototype in Clojure in Deep Diamond.

Deep Diamond currently supports general tensor operations, fully connected NN layers, and convolutional (CNN) layers, on CPU and GPU. Based on this, relatively stable, infrastructure, I started adding Vanilla RNN implementation backed by OneDNN (Intel, CPU).

OneDNN RNN implementation is, like all the low-level backends that leading DL frameworks use, very heavy, convoluted, and rather unclojure-y. So it takes some time to discover how it should be properly used, and how to best hide its complexity under a nice high level Clojure.

Specifically, this was implemented in the first month of Q1:

• a prototype of developer-friendly implementation of RNN in Clojure
• Vanilla RNN support (as the stepping stone for more serious LSTM and GRU)
• The first iteration of an extension infrastructure for various backend implementations of RNN.
• a clean low-level integration with Intel's oneDNN RNN on the CPU (for Vanilla RNN).
• TESTS. Not ideal amount of, but enough for this phase.

So far, I'm pretty satisfied with the progress, as I think I have discovered the roughest edges of Intel's implementation, and found ways to fit this into existing Deep Diamond code. I expect the following 2 months of the project to require lots of work and tests, but I feel I shouldn't expect nasty surprises. I believe that by the end of the funding period I'll be able to release the version of Deep Diamond that will have the functionality I've proposed.

My goal with this round is to implement Recurrent Neural Networks (RNN) support in Deep Diamond. The first month was dedicated to literature review (and other hammock-located work), exploration of OneDNN implementation of RNN layers, and implementation of RNN prototype in Clojure in Deep Diamond.

Now, in the second month, I managed to make the first iteration of fully functional RNN layer based on the DNNL backend (currently only Vanilla RNN implementation) that fits into the existing high-level network building code. (more details follow)

Deep Diamond currently supports general tensor operations, fully connected NN layers, and convolutional (CNN) layers, on CPU and GPU. Based on this, relatively stable, infrastructure, in the first month I added Vanilla RNN operation implementation backed by OneDNN (Intel, CPU).

Now, in the second month, I only managed to make a couple of commits, but the last one is pretty big. I couldn't break it into many smaller ones, because I had to make a couple of subtle changes that affected the internal implementation and broke existing code (_does not affect users, just me!). I didn't want to commit a completely broken code into the github repository.

Now, what does this commit do? A. It adds the first complete implementation of RNN layer that

1. technically works and pushes the numbers to the right places connected to it
2. is tested in isolation
3. is supported in high-level Deep Diamond code together with other layers type

B. I implemented a supporting "Ending" layer for feeding the time-based RNN output to other layer types (Dense, CNN, etc.) C. I improved the existing infrastructure to support this, and also made it more flexible D. I fixed bugs and wrote tests…

That is fine, but I am not completely satisfied with this milestone. I hoped to make a fully working example of a network that learns something useful, but that proved a bit harder than I expected. I couldn't make the RNN I have so far actually learn. It appears to be working, but even simple examples for RNN applications are not that simple, so I wasn't be able to make it learn the artificial one I tried it with. I am not sure whether it's due to bugs, or the example is simply pathological, and the Vanilla RNN (which is currently the only type I implemented) can't do anything with it anyway. This is something that I'll be doing the following days/weeks, and I expect to discover more when I implement more advanced LSTM and/or GRU RNN types.

Beside that, the CUDA backend is broken in the current commit since I haven't updated it to the last changes, but it's not a problem; once I nail DNNL, making CUDA compatible will be more or less routine.

My goal with this round (Q1-2022) is to implement Recurrent Neural Networks (RNN) support in Deep Diamond. The first month was dedicated to literature review (and other hammock-located work), exploration of OneDNN implementation of RNN layers, and implementation of a RNN prototype in Clojure with Deep Diamond. In the second month, I made the first iteration of fully functional vanilla RNN layer based on the DNNL backend that fits into the existing high-level network building code.

Building on this, in the third month I finally managed to crack that tough nut called "recurrent networks implementation libraries" and make:

1. A nice RNN layer implementation that seamlessly fits into the existing low-level and high level Deep Diamond infrastructure.
2. The DNNL implementation of Vanilla RNN, LSTM, and GRU layers for the CPU (fully functional, but not ideal).
3. The low-level integration of Nvidia's CUDNN GPU library, including TESTS.
4. The high-level integration of CUDNN into Deep Diamond's layers (tests show that I need to improve this).
5. A functional test that demonstrates learning of the said layers with Adam and SGD learning algorithms.
6. I released a new version of Deep Diamond 0.23.0 with these improvements (RNN support should be considered a preview, as I still need to fix and polish some parts of CUDNN GPU implementation).

Some notes about this milestone:

• Some things were smoother and easier than I expected. In general, I managed to find a way to fit RNNs into existing DD learning implementations, which was a pleasant surprise, as well as a sign of Clojure's qualities (and my ever increasing Clojure skills, if I am to stop being modest :)
• Some other things were harder than I expected. Notably, RNN implementation in both DNNL and CUDNN has many moving parts and are notoriously cumbersome to configure.
• There is total lack of simple examples of RNNs. Low level code examples of both of these libraries are practically non-existent in the wild. That slowed me a lot, since I had to discover everything by poking around.
• The documentation for DNNL and CUDNN related to RNNs is helpful, but most details still had to be discovered through trial and error. That too made me go at a snail's pace compared to my earlier work on other parts of Deep Diamond.
• Lacking simple examples, RNNs are very difficult to test. I am sure that this milestone, although usable, needs a lot more care to reach the level of usefulness of other parts of Deep Diamond.
• On the bright side, I am very satisfied with the high level API. The user virtually has three simple functions: rnn, lstm, gru, and ending, which fit into the existing infrastructure, and don't require any work from the user other than putting them at the right place; everything gets configured automatically under the hood by Deep Diamond. This part, I like a lot.

All in all, I am 80% satisfied with what I achieved in these 3 months. I hoped for a more complete implementation. On the other hand, I solved all tricky problems, and have clear idea how to fix and improve what is still not up to Uncomplicate standards, so I'll have something to work on in the following weeks/months. And it's finally time to update the book with these new goodies!

Thank you Clojurists for your continued trust in my work!