Dragan Djuric

Gemma 3 AI model in Clojure

Tue, 09 Dec 2025 23:35:00 +0100

Recently I've been working on the ONNX runtime integration into Deep Diamond, backed by the grant sponsored by the Clojurists Together Foundation. In the past few articles, we've seen how ONNX models are integrated into Deep Diamond, using only a single function onnx, with almost no need for additional configuration (which is available). I used a simple MNIST model in the demonstration. But, can we now load and run the inference on the real deal models, such as the open LLMs from the Hugging Face, for example? Let's see!

The Hugging Face model card has this to say about Gemma 3: "Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models." (etc., etc.) So, it seems to be something worth trying.

I'll try to be brief, and skip the unnecessary talk. Let's just show the code, which I've just lifted up and adapted from the Diamond's midje tests.

What we need for this? First, decide on the backend engine; this time we'll use tensors in main memory backed up by the oneDNN engine (DNNL).

(def fact (dnnl-factory))
(def neand-fact (neanderthal-factory fact))

Next, load and configure a particular flavor of Gemma 3 (a smaller one, only 1 billion parameters). The onnx function creates a generalized blueprint, which can create the actual functions when evaluated with the specific input tensors.

(def onnx-bp (onnx fact "data/gemma-3-1b-it-ONNX-GQA/onnx/model.onnx"
                   {:options (-> (options)
                           (override-dimension! "batch_size" 1)
                           (override-dimension! "sequence_length" 1)
                           (override-dimension! "past_sequence_length" 1)
                           (override-dimension! "total_sequence_length" 1))})

Gemma 3 has 63 inputs and 61 outputs. We'll need to provide these, but even here we can automate some parts with Clojure, since past-key values are pretty uniform. We only need to provide inputs, while the engine can create the outputs for us.

(def src-tz (tensor fact [1 1 28 28] :float :nchw))
(def input-ids (tensor neand-fact [1 1] :long :nc))
(def position-ids (tensor neand-fact [1 1] :long :nc))
(def attention-mask (tensor neand-fact [1 1] :long :nc))
(def past-key-values (repeatedly 60 #(tensor fact [1 3 1 64] :float :nchw)))

Next, create the executable instance model. Nothing too fancy here.

(def gemma-next! (onnx-bp (into [input-ids attention-mask position-ids] past-key-values)))

Now, these inputs need to be initialized. Normally, that would be done inside an LLM generation loop, but here we only demonstrate one step, and we transfer some mock data.

(transfer! [2] input-ids)
(transfer! [0] position-ids)
(transfer! [1] attention-mask)
(doseq [pkv past-key-values]
  (transfer! (repeat 0) pkv))

Aaaaand, we actually run the model by calling our gemma function, which provides the next token.

(gemma-next!)

Now, hold on with the celebration. This does not actually return a full answer from the LLM. This only returns the next token, but in the form of large tensor full of numbers. The information is there, but needs to be extracted from these numbers to the form of string. Also, this is only one step; a LLM would typically run this in a loop and spew tokens after tokens. There's some more work to do until we get a ready made, hands-off chatty LLM. But the main work has been done, and now it's the matter of setting it up properly, tokenizing the inputs, and calling it in a useful way! Still lots of work, but not the hardest parts :)

I've applied for Clojurists Together yearly funding in 2026. If you are a Clojurists Together member, and would like to see continued development in this area, your vote can help me keep working on this :)

My goal with this funding in 2026 is to continuously develop Clojure AI, ML, and high-performance ecosystem of Uncomplicate libraries (Neanderhal and many more), on Nvidia GPUs, Apple Silicon, and traditional PC. In this year, I will also focus on writing tutorals on my blog and creating websites for the projects involved, which is something that I wanted for years, but didn't have time to do because I spent all time on programming.

Plan for Clojure AI, ML, and high-performance Uncomplicate ecosystem in 2026

Sat, 29 Nov 2025 01:41:00 +0100

I've applied for Clojurists Together yearly funding in 2026. Here's my application. If you are a Clojurists Together member, and would like to see continued development in this area, your vote can help me keep working on this :)

How that work will benefit the Clojure community

This will highly benefit the Clojure community as this is THE AI ecosystem for Clojure, and supporting AI is arguably the main focus on probably all software platforms. Clojure has something to offer on that front, beyond just calling OpenAI API as a web service!

Uncomplicate grew to quite a few libraries (of which some are quite big; just Neanderthal is 28,000 lines of highly-condensed, aggresively macroized, and reusable code): Diamond ONNX Runtime, Neanderthal, Deep Diamond, ClojureCUDA, ClojureCPP, Apple Presets, ClojureCL, Fluokitten, Bayadera, Clojure Sound, and Commons.

Here's a word or two of how I hope to improve each of these libraries with Clojurists Together funding in 2026.

Neanderthal (Clojure's alternative to NumPy, on steroids)

In 2025, Neanderthal celebrated its 10th birthday. It started as a humble but fast matrix and vector library for Clojure, but after 10 years of relentless improvements, now it boasts a general matrix/vector/linear algebra API implemented by no less than 5(!) engines for CPUs, GPU (Nvidia CUDA), GPU (OpenCL: AMD, Intel, Nvidia), Apple Silicon (Accelerate), and general CPU (OpenBLAS). And this is not a superficial support for the sake of ticking a check box; each of these engines support much more operations on exotic structures, and configuration options, than I've seen elsewhere. It has almost everything, but it doesn't (YET!) have a METAL-based engine for Apple GPUs. Let's work on that!

Deep Diamond (the Clojure Tensor and Deep Learning library, not quite unlike PyTorch, but of different philosophy)

In 2019, I started Deep Diamond as a demo showcase for Neanderthal's capabilities as the foundation for high-performance number-crunching software. It quickly outgrew that, and became a general Tensor/Deep Learning API, implemented by several fast, native optimized, backends, that run on both CPUs and GPUs, across hardware platforms (x86_64, GPUs, arm64, Apple Silicon, you name it) and operating systems (Linux, MasOS, Windows). Of course, it does not clash with Neanderthal, but complements it, in the best manner of highly focused Clojure libraries that do one job and do it well.

Deep Diamond is quite capable, but it cries for a METAL-based engine for Apple GPUs, too.

Diamond ONNX Runtime (the Clojure library for executing AI models)

This is the latest gem in Uncomplicate's store, and I developed it thanks to Clojurists Together funding in Q3 2025. Similarly to how I started Deep Diamond mainly as a teaching showcase for Neanderthal, I started this to show Clojure programmers how close we, Clojurists, are to the latest and shiniest AI stuff that everyone's raving about. But of course, being close does not mean that we can close the gap to the multi-billion funded Python ecosystem in a few afternoons. It needs laser-focused development and knowing what to do, when, and where. Nevertheless, Clojure is there. Now we can run inference on the trained models from Hugging Face and other vibrant AI communities directly in Clojure's REPL. Does this make an effortless billion-dollar AI startup? NO. Does it bring Clojurians to the party? YES! And there's more to come.

Not only that this library is new, but the whole wider ecosystem exploded in the last year with the wide availability of open-weights model that you can run at home. So, lots of functionality is added upstream all the time, and I hope to be able to stay current and have the best and newest stuff in Clojure..

ClojureCUDA (REPL-based low-level CUDA development)

Not many Clojurians may prefer to work with GPU directly, or to write their own kernels. Neither do I. But, this library is one of the un-celebrated workhorses that enables me to implement whatever I want in Neanderthal, Deep Diamond, and Diamond ONNX Runtime, instead of just trying to wrap whatever there is in upstream C++ libraies. ClojureCUDA gives us the superpower of choice: wrap whatever works, but then implement the missing parts yourself!

As CUDA is receiving a steady stream of changes and improvements, I'd like to improve and extend ClojureCUDA to always be in top shape! It is not as easy as it seems to the casual onlooker.

ClojureCPP (the gateway to native C++ libraries)

From 20,000 feet, integrating a native library to JVM and Clojure may look straightforward. Oh, how wrong they are. Virtually every C++ library is a special kind of jungle, with its own structures, patterns and inventions. What might seems a minor technical detail might require special acrobatics to support it on the JVM. Masking that mess under the hood so that a Clojurian do not need to care might be insanely brittle if it weren't for ClojureCPP! It is not as large as Neanderthal or Deep Diamond, but it is one of the reasons that enables these upper level libraries stay on the 25,000 or 3,000 lines of code mark, instead of being 500,000 or 50,000, as many of their counterparts in other languages.

Apple Presets (native JNI bindings for various Apple libraries)

Yup. To support Apple Silicon in Neanderthal and Deep Diamond I had to make these bindings, since there weren't any to "just" wrap. And to support more Apple high performance computing apis, I'll have to create additional bindings (for example, for METAL) and only then develop the Clojure part.

Fluokitten, Bayadera, ClojureCL, Commons, Clojure Sound, etc.

These libraries will not be in focus in 2026., but will probably need some care and assorted improvements here and there.

Summary:

In short, my focus with this funding with Clojurists Together will have two main branches:

Development of new functionalities, supporting more hardware and platforms for existing functionality, and fixing issues for a dozen Uncomplicate libraries that already exist. This is what is described in the text you've just read.
Develop an unified website for Uncomplicate and stuff it with useful information in one place. Currently, some libraries have websites that I wrote many years ago, while some rely on GitHub Clojure tests, in-code documentation, tutorials on dragan.rocks and my books. There are many resources, some of which are quite detailed (2 full books!), but people without experience (which is the majority of Clojure programmers) have a hard time using all these in organized way. I hope to solve this with the unified website!

Projects directly involved:

Not One, Not Two, Not Even Three, but Four Ways to Run an ONNX AI Model on GPU with CUDA

Sun, 09 Nov 2025 18:49:00 +0100

Two weeks ago, I announced a new Clojure ML library, Diamond ONNX RT, which integrates ONNX Runtime into Deep Diamond. In that post, we explored the classic Hello World example of Neural Networks, MNIST handwritten image recognition, step-by-step. We run that example on the CPU, from main memory. The next logical step is to execute this stuff on the GPU.

You'll see that with a little help of ClojureCUDA and Deep Diamond built-in CUDA machinery, this is both easy and simple, requiring almost no effort from a curious Clojure programmer. But don't just trust me; let's fire up your REPL, and we can continue together.

Here's how you can evaluate this directly in your REPL (you can use the Hello World that is provided in the ./examples sub-folder of Diamond ONNX RT as a springboard).

Require Diamond's namespaces

First things first, we refer functions that we're going to use.

(require '[uncomplicate.commons.core :refer [with-release]]
         '[uncomplicate.neanderthal.core :refer [transfer! iamax native]]
         '[uncomplicate.diamond
           [tensor :refer [tensor with-diamond]]
           [dnn :refer [network]]
           [onnxrt :refer [onnx]]]
         '[uncomplicate.diamond.internal.dnnl.factory :refer [dnnl-factory]]
         '[uncomplicate.diamond.internal.cudnn.factory :refer [cudnn-factory]]
         '[hello-world.native :refer [input-desc input-tz mnist-onnx]])

None of the following ways to run CUDA models has preference, you use the one that best suits your needs.

Way one

One of the ways to run ONNX models on your GPU is to simply use Deep Diamond's cuDNN factory as the backend for your tensors. Then, the machinery recognizes what you need and proceeds doing everything on the GPU, using the right stream for tensors, Deep Diamond operations, and ONNX Runtime operations. This looks exactly the same as any other Deep Diamond example from this blog or the DLFP book.

(with-diamond cudnn-factory []
  (with-release [cuda-input-tz (tensor input-desc)
                 mnist (network cuda-input-tz [mnist-onnx])
                 classify! (mnist cuda-input-tz)]
    (transfer! input-tz cuda-input-tz)
    (iamax (native (classify!)))))

..it says.

Way two

As an ONNX model usually defines the whole network, you don't need to use Deep Diamond's network as a wrapper. The onnx function can create a Deep Diamond blueprint, and Deep Diamond blueprints can be used as standalone layer creators. Just like in the following code snippet.

(with-diamond cudnn-factory []
  (with-release [cuda-input-tz (tensor input-desc)
                 mnist-bp (onnx cuda-input-tz "../../data/mnist-12.onnx" nil)
                 infer-number! (mnist-bp cuda-input-tz)]
    (transfer! input-tz cuda-input-tz)
    (iamax (native (infer-number!)))))

… again.

Way three

We can even mix CUDA and CPU. Let's say your input and output tensors are in the main memory, and you'd like to process them on the CPU, but you want to take advantage of the GPU for the model processing itself. Nothing is easier, if you use Deep Diamond. Just specify an :ep (execution provider) in the onnx function configuration, and tell it that you'd like to use only CUDA. Now your network is executed on the GPU, while your input and output tensors are in the main memory, and can be easily accessed.

(with-release [mnist (network input-tz [(onnx "../../data/mnist-12.onnx" {:ep [:cuda]})])
               infer-number! (mnist input-tz)]
        (iamax (infer-number!)))

… and again the same answer.

Way four

Still need more options? No problem, onnx can create a standalone blueprint, and that blueprint recognizes the :ep configuration too.

(with-release [mnist-bp (onnx input-tz "../../data/mnist-12.onnx" {:ep [:cuda]})
               infer-number! (mnist-bp input-tz)]
        (iamax (infer-number!)))

No surprises here.

Is there anything easier?

If you've seen code in any programming language that does this in a simpler and easier way, please let me know, so we can try to make Clojure even better in the age of AI!

The books

Should I mention that the book Deep Learning for Programmers: An Interactive Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure teaches the nuts and bolts of neural networks and deep learning by showing you how Deep Diamond is built, from scratch? In interactive sessions. Each line of code can be executed and the results inspected in the plain Clojure REPL. The best way to master something is to build it yourself!

Get Ready for Clojure, GPU, and AI in 2026 with CUDA 13.0

Thu, 30 Oct 2025 17:37:00 +0100

A little anniversary

Did you know that CUDA has been available in Clojure for the last 9 years through ClojureCUDA, and GPU programming through OpenCL for more than 10? I almost forgot about these anniversaries.

Ten years ago most people liked it a lot, starred it on Github, patted me on the back, but then concluded that they don't have an Nvidia card available on their laptops, or, if they had GPUs, that they won't have time to learn to think in massive parallel algorithms, or if they have time and will, that there are no GPUs in the servers, so what would they do with their applications, even if they created them in Clojure, and so on, and so off :)

But, ClojureCUDA and ClojureCL continued living on for these 10 years, I used them in creating Neanderthal, Deep Diamond, and Diamond ML, and they proved themselves as simple and reliable tools. I still had trouble convincing Clojure programmers that they can write GPU programs that run as fast as they'd wrote them in C++, but interactively in the Cloujre REPL, without C++ hell.

But I'm not easy to shake off! If it's necessary, I'll continue for 10 more years, for I'm convinced there'd be a moment when Clojure programmers are going to say "hmmm, this is something that we can use and be good at!".

CUDA 13 is here!

I've recently released ClojureCUDA 0.25.0, with support for the latest CUDA 13.0.2!

Why not celebrate that by opening the REPL, and coding your first Hello World application on the GPU? I promise, it won't be a usual GPU carpet of text; this is ClojureCUDA, it follows the Clojure philosophy by being simple and interactive!

There's not much sense in wielding a GPU to print out "Hello World". Note that it is also not very useful to work with scalar numbers and call a GPU function to add or multiply two numbers. No. Unless you have many, many, numbers to crunch, stay by your trusty CPU. For our purposes, many, many, numbers would be two vectors of dimension 3 (hey, it's hello world; imagine it's 3 billion). Also, even when we have many, many, numbers the sheer cost of getting them to the GPU memory would destroy any gains we get in the computation speed, so we must also ensure that we want to perfom many complicated operations. Well, we will only do the simple operation of adding these two vectors, and we will pretend that this operation is extra-demanding (we're hello-worlders today, we can cheat a bit).

CUDA Hello World

First things first, we require the functions that we'll use.

(require '[uncomplicate.commons.core :refer [release]]
         '[uncomplicate.clojure-cpp :refer [float-pointer pointer-seq]]
         '[uncomplicate.clojurecuda.core
           :refer [compile! context device function grid-1d init launch! mem-alloc-driver
                   mem-alloc-pinned mem-alloc-runtime memcpy-host! module parameters program
                   synchronize! push-context!]])

If it seems to you that this list already looks too large, I agree with you. But, don't be afraid; in Uncomplicate libraries there are so many higher-level helpers that you'll rarely need to touch these. I only use them here because I want to show you that even when we program at the base CUDA level, Clojure can do it interactively and each line can be evaluated by itself, and each intermediate result can be inspected and understood.

I'll use def for poor man variables, and pop-context! so this is evaluated step-by-step. The real code would be much simpler; resources can be managed by with-release and with-context!

First we initialize CUDA and create the the CUDA context.

(init)
(def ctx (context (device)))

Unfortunately, CUDA insists on managing contexts in different threads by itself, so we have to let CUDA know that we want to use the context that we've just created. ClojureCUDA has some macros that can help with making this simpler, such as with-context, and in-context, but we'll do this hello world as real ninjas, with basic tools!

(push-context! ctx)

Many language integrations try to let you write everything in that language, both the host code that manages GPU computations, and the GPU kernels themselves. So far, they don't fare too well compared to C++ CUDA, even when it comes to simplicity of such kernel code. ClojureCUDA doesn't do that, for a reason. We are practical people. We understand that we won't be able compile kernels ourselves and be competitive with Nvidia. So, we write kernels in C++ that Nvidia uses, since they are typically not that complicated compared to host code that manages them. We can load these kernels as strings, and I find it's most convenient to not sprinkle these strings throughout my .clj source files, but to load them from .cu files, which contain C++ code that text editors recognize.

We load the kernel source.

(def kernel-source (slurp "test/cuda/examples/jnvrtc-vector-add.cu"))

Next, we compile it.

(def prog (compile! (program kernel-source)))

We create the module…

(def m (module prog))

… and load the add function that was defined in the kernel code. CUDA kernels are short functions that run on the GPU.

(def add (function m "add"))

This way we get the best of both worlds: we edit short C++ kernels, and then load them in our Clojure REPL and manage the kernels they define. If we change anything, there's no need for recompilation of everything; just the short .cu file that changed, and this is also interactive, as these tools are available as Clojure functions.

Next, this kernel needs data! We allocate some memory on the GPU. There are a few different types that CUDA offers, and even a few CUDA APIS: runtime and driver. Typically, the runtime API is what CUDA uses in the C++ code that mixes host and GPU device code, while the driver API is more geared towards tools such is ClojureCUDA. But, some CUDA libraries will expect inputs to be from the runtime API, and ClojureCUDA supports both. We can even mix them!

(def gpu-a (mem-alloc-runtime (* Float/BYTES 3)))
(def gpu-b (mem-alloc-driver (* Float/BYTES 3)))

The data needs to be transferred to the GPU memory. There are many functions in CUDA for doing that for every combination of different argument types. ClojureCUDA simplifies this a lot with protocols, and usually memcpy-host! will find the right way to transfer the data. We also need the place to keep the results, unless we want to overwrite one of the two input arrays.

(memcpy-host! (float-pointer [1 2 3]) gpu-a)
(memcpy-host! (float-pointer [2 3 4]) gpu-b)
(def gpu-result (mem-alloc-pinned (* Float/BYTES 3)))

That's a lot of work to set everything up! Let's compute it at last!

(launch! add (grid-1d 3) (parameters 3 gpu-a gpu-b gpu-result))

The kernels are launched asynchronously. That means that the launch! function returns as soon as it puts the kernel in the computation queue of the GPU, typically before the kernel has actually been executed; there might be 1000 kernels in the queue before this one. We can explicitly wait until the kernel has completed its work by calling synchronize!.

(synchronize!)

Since we now know that the new data is in the gpu-result array, we will have to move it back to the host if we wont to see it.

(pointer-seq (memcpy-host! gpu-result (float-pointer 3)))

=> (3.0 5.0 7.0)

Yeah, I know, a dozen lines of code for a simple addition of two vectors. But hear me out: the management code for more demanding kernels is not much more complicated. So, it's a dozen lines for this simple case, but it will be a dozen lines for some real crunching. Or 23 lines, but not 2000 lines.

Plus, there are so many functions in the libraries for computing vectors, matrices, and tensors, that you'll write your own kernels only occasionally, and you'll still get great speed, once you learn the basics.

So, invest some time in 2026, learn the basics of GPU computing, and enjoy the coming AI age! In Clojure!

Oh, I didn't show you the C++ kernel code. It's C++, it must be scary! No, it's not. CUDA kernels are written in a subset of C++, without the scary parts!

extern "C"
__global__ void add(int n, float *a, float *b, float *sum) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        sum[i] = a[i] + b[i];
    }
};

… and, hey, we shouldn't forget to clean up the memory! Since if we didn't use def, we could just rely on with-release to do this for us, but we did everything manually, and now we have to clean up manually.

(release gpu-result)
(release gpu-b)
(release gpu-a)
(release add)
(release m)
(release prog)
(release ctx)

Clojure Runs ONNX AI Models Now - Join the AI fun!

Sun, 26 Oct 2025 16:56:00 +0100

Hello, Clojurians! I haven't written here in a long time. Was I tired? Is anybody reading blogs anymore? Who knows. But that was not the main reason.

I've been working on several Clojure projects sponsored by the Clojurists Together Foundation. I did a ton of things, but after all this programming, I was kinda tired, and kept slugging when it comes to telling people about the work done! That's not very smart, but you know how it goes… :) But, then, if we don't tell people about awesome software that we have, nobody is going to use it, so finally I had to stop kicking this down the road, sit, and write the first post. It's been long overdue, so expect more posts soon!

ONNX Runtime in one line of Clojure

The most recent thing I'm currently working on started its life as Clojure ML (again, superthanks to Clojurists Together for sponsoring this). I proposed to create a human-friendly Clojure API for AI/DL/ML models, and back it by the first implementation, in this case based on ONNX Runtime. Of course, it should all be integrated into existing Clojure libraries, and follow the Clojure way of doing stuff as much as possible!

The idea is to get an existing, pre-trained ML model previously exported to the ONNX format from whatever technology the authors chose (which in today's world is typically Python and PyTorch), and put it into production in Clojure and JVM. It should be seamless and in-process, without any clunky interoperability, copy, translation, etc. Of course, our Clojure numerical libraries fully support GPU computing, so it goes without saying that we want that, too! Just to be clear, we do not use nor need any Python or Python interop for this, we use the ONNX Runtime's underlying C library.

Nice idea, but what parts of this well intended story can we evaluate in our REPLs right now? At least some promising demo? Are we on the trail? To access that AI goodness, we surely have to do a sophisticated dance? Are the steps hard to learn? Do we need to watch carefully for slippery floor? Is it accessible to mere mortals?

Here's the gist:

(onnx "data/mnist-12.onnx")

"Wait, what?", you'll say. One function? One tini, tiny, function, with one laughingly trivial argument? Is that an API? What does such trivial API do? "Now you confused me!", you'll scratch your head. It's just a stick.

I hope I've also intrigued you, so please keep reading to see it in action (this post is actually generated from a live REPL session, so the example is fully executable, not just interesting bits on the table, and a ton of complex boilerplate under a Persian rug).

Hello World, the MNIST image recognition model

For this recipe, you'll need the following ingredients: Deep Diamond tensors (one cup), Deep Diamond network (one slice), one Neanderthal transfer! function for moving data around for demo purposes, and that's it! Oh, yes, don't forget the new onnx function. We load the native namespace, and the right Deep Diamond engine is set up for our system (yes, even on Mac OS, thanks to Clojurists Together!).

(require '[uncomplicate.neanderthal.core :refer [transfer! iamax]]
         '[uncomplicate.diamond
           [tensor :refer [tensor desc]]
           [dnn :refer [network]]
           [onnxrt :refer [onnx]]]
         '[uncomplicate.diamond.native])

The ONNX model

We evaluate the onnx function, and it loads the model.

(def mnist-onnx (onnx "../../data/mnist-12.onnx"))

#'user/mnist-onnx

Sure, that's easy, but how is that useful? Well, the result is a function. This function has just been set up with ONNX internals, so now it can create Deep Diamond network layers and fit in with the rest of the Tensor-y stuff that DD already provides.

The ONNX Runtime model revolves around environment, session, input and output tensors, type info, and a lot of other stuff and brittle ceremony. Sure, sometimes you need to reach these internals, and diamond-onnxrt provides clojurized internals API even for that. However, it can sing the main song, and set all the right arguments at the right places for you. Even the onnx function supports option map, where you can tell what you like, and it will take care to configure ONNX to do the right thing, but this is a story for another article.

The rest is the usual Deep Diamond stuff, which is simple as beans!

The MNIST dataset specifies images of hand-written digits, in just one grayscale channel, each $28\times28$ pixels a challenging task for 1989's USPO and the tecnology from back then, but a hello world level stuff for today's accelerated libraries (still, keep in mind that if you tried to code even this easy example without such libraries, you'll be surprised how slow that can be!).

We create a tensor descriptor for such input (this step can be left out, but I'm being pedantic to accommodate beginners):

(def input-desc (desc [1 1 28 28] :float :nchw))

#'user/input-desc

Next, we create a reusable abstract network blueprint, that can then create concrete networks tailored for training, or optimized for inference, that is classifying MNIST images. Normally, we would have to train these networks, or load the parameters from somewhere, but in this case it contains only of the onnx model, which had already been trained and already knows all the right weights, so no training is needed (nor available with ONNX Runtime yet; it's main job is inference in production).

(def mnist (network input-desc [mnist-onnx]))

#'user/mnist

Note that all these things so far look and behave just as ordinary Clojure objects. You can use them even outside this specific structure. Full flexibility that I hope will spark your creativity.

We'll also need a place for the actual image that we'd like to classify. This particular network that I downloaded from ONNX Runtime examples specifies exactly one image at input, to classify one at a time. Typically, if we have many images, it's better to compute them in batches, but it's just a hello-world, after all, we won't be too demanding.

(def input-tz (tensor input-desc))

#'user/input-tz

A blueprint (mnist in this case) is a function that can create networks optimized for inference with concrete tensors, adequate internal tensors, and parameters. The following line is the moment when the network is actually created from the abstract descriptors contained in its blueprint, to the actual engines, operation primitives, and tensors in memory.

(def classify! (mnist input-tz))

#'user/classify!

True to the Clojure philosophy, mnist is a function, which, given the specification for desired input, (mnist input-tz) produces classify!, which is a function, too, but for actual inference! It might sound cumbersome when it's written out, but the code shows it's elegance. No need for complex APIs. Each thing does exactly one thing, and does it in the most simple way, by just evaluating with one or two parameters!

Now we got a function that classifies images

This is how you would typically use this

Step one: classify! is now a typical Clojure function! Evaluate it:

(classify!)

{:shape [1 10], :data-type :float, :layout [10 1]} (-0.04485602676868439 0.007791661191731691 0.06810081750154495 0.02999374084174633 -0.1264096349477768 0.14021874964237213 -0.055284902453422546 -0.04938381537795067 0.08432205021381378 -0.05454041436314583)

The result is a ten-element tensor, each element represents the possibility that the category at its index is the right one. So we should just find which element contains the highest value, and that'd be our category, which in the MNIST example is, very conveniently a digit 0 to 9 that is equal to that index.

However, you can see that the current values are just random small numbers. This is because we never loaded any image data to the input tensor! It just classified random noise as not very likely to be an image of any digit.

We need step zero: place the image data in network's input somehow. This could be done in many different ways (for example, by memory-mapping the image data on disk), but we'll keep it simple, and we'll just transfer it naively from an in-place Clojure sequence. (this is a hello-world :)

The following sequence is copied from the actual MNIST data, but I just took the data of the first image, and scaled it to 0-1 range instead of 0-255.

(transfer! (map #(float (/ % 255)) [0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 84.0 185.0 159.0 151.0 60.0 36.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 222.0 254.0 254.0 254.0 254.0 241.0 198.0 198.0 198.0 198.0 198.0 198.0 198.0 198.0 170.0 52.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 67.0 114.0 72.0 114.0 163.0 227.0 254.0 225.0 254.0 254.0 254.0 250.0 229.0 254.0 254.0 140.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17.0 66.0 14.0 67.0 67.0 67.0 59.0 21.0 236.0 254.0 106.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 83.0 253.0 209.0 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 22.0 233.0 255.0 83.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 129.0 254.0 238.0 44.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 59.0 249.0 254.0 62.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.0 254.0 187.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 9.0 205.0 248.0 58.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.0 254.0 182.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 75.0 251.0 240.0 57.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 19.0 221.0 254.0 166.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 203.0 254.0 219.0 35.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 38.0 254.0 254.0 77.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 31.0 224.0 254.0 115.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.0 254.0 254.0 52.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 61.0 242.0 254.0 254.0 52.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 121.0 254.0 254.0 219.0 40.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 121.0 254.0 207.0 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0])
           input-tz)

{:shape [1 1 28 28], :data-type :float, :layout [784 784 28 1]} (0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0)

(classify!)

{:shape [1 10], :data-type :float, :layout [10 1]} (-1.2567189931869507 0.6275832653045654 8.642718315124512 9.428943634033203 -13.740066528320312 -6.045698642730713 -23.486745834350586 28.3399658203125 -6.7914958000183105 3.941998243331909)

Now, we see some better looking results, but are we the one who need to look at a bunch of numbers and compare them?

No, the machine should do that. Luckily, Neanderthal has just the right function for this!

(iamax (classify!))

And this is the kind of answer that we can show our clients! What's on this image? Easy, it's 7!

Can you tell me the main point of this, in one paragraph?

Yes. Clojure programmers typically write functions. Functions are things that take something at the input, compute stuff internally, and return an output, which is hopefully useful downstream. The funcion transforms the input into the output according to the logic that we programmers wrote in code, following some algorithm that we designed for the purpose. Now, sometimes the problem is so convoluted that we don't have the slightest idea how to write that transformation in code, but what we (or someone else) do have is lots of data, and in many such cases we can train a general machinery (neural networks for example) to find out a good enough transformation. Sometimes someone else have already done the hard part by training the network, exporting it to a standard format (ONNX) and gave it to you! Now, you can load it in Clojure and use it as a Clojure function. You don't even need to know how it works internally, but it does the thing that you need, it transforms the input tensors that you have into just the right output tensors. What you do with these outputs is up to you :)

Who is this for?

Do you need to be an AI researcher to find this useful? Absolutely not! This can appeal to any Clojure engineer.

AI researchers try to find novel AI models, or to push their model by 0.1% on an artificial benchmark. Recently, they don't necessarily even do that, some of them found the way to chase funding at crazy evaluations, and catch it. Some of them don't necessarily write code but work with mathematical models trying to figure a way to do some abstract thing. Or, if they are PhD students, they spend endless nights fiddling with Python and PyTorch trying to figure this or that task assigned by their laboratory, or they just try to catch a bit of sleep while a GPU cluster crunches some tiny step in an endless training cycle.

There's nothing wrong with that, but if you're a Clojure programmer, you probably don't have time, opportunity, experience, or even interest to work on that stuff. But, even if you don't want (or can't) understand AI internals, you can still be very creative with the applications. Now there are many, many, published ML models that work, many of them are even exported to ONNX, and quite usable. You don't need to invent a new OpenAI competitor, there are many more mundane problems that can be solved by taking an already existing model and applying it in a niche context, in a domain that you know well. You don't even need to understand exactly what or how the model does what it does, you can treat it as a black-box function that transforms inputs and outputs, and that function just need a bit more care to work than a regular Clojure four-liner that you'd normally write and be proud of.

Although (sadly) Clojure has not find its way in the big guns AI arena, Clojure is a very capable capable language and Clojure programmers very knowledgeable people when it comes to integrating stuff into real-world applications! So, here it is, now you don't have to make compromises; you can got to Hugging Face, or some other AI related community, find ONNX models that other people already prepared, and join the AI fun, directly from Clojure.

Apple silicon support in Clojure's Neanderthal Fast Matrix Library

Tue, 19 Nov 2024 22:15:00 +0100

I've applied for Clojurists Together funding to explore and hopefully implement a Neanderthal Apple silicon backend. If you are a Clojurists Together member, and you think that this functionality is important, please have that in mind when voting. Here's the proposal that I've sent. Of course, any suggestions are welcome and highly appreciated!

The proposal

My goal with this funding in 2025 is to support Apple silicon (M cpus) in Neanderthal (and other Uncomplicate libraries where that makes sense and where it's possible).

This will hugely streamline user experience regarding high performance computing in Clojure for Apple macOS users, which is a considerable chunk of Clojure community. They ask for it all the time, and I am always sorry to tell them that I still don't have a solution. Once upon a time, Neanderthal worked on Mac too, but then Apple did one of their complete turnarounds with M1… This basically broke all number crunching software on macs, and the world is slow to catch up. Several Clojure developers started exploring high performance computing on Apple, but didn't get too far; it's LOTS of functionality. So, having Neandeathal support Apple would enable several Clojure data processing communities to leapfrog the whole milestone and concentrate on more high-level tasks.

This affects Neanderthal (matrices & vectors), and Deep Diamond (tensors & deep learning) the most. Nvidia's CUDA is not physically available on Macs at all, while OpenCL is discontinued in favor of Apple's proprietary Metal (and who knows what else they've came up with since).

Neanderthal is a Clojure library for fast matrix computations based on the highly optimized native libraries and computation routines for both CPU and GPU. It is a lean, high performance, infrastructure for working with vectors and matrices in Clojure, which is the foundation for implementing high performance computing related tasks, including, but not limited to, machine learning and artificial intelligence. Deep Diamond is a tensor and deep learning Clojure library that uses Neanderthal and ClojureCUDA under the hood (among other things).

So, what are the missing parts for Apple silicon?

A good C++/native interop. That is more or less solved by JavaCPP, but their ARM support is still in its infancy, especially regarding distribution. But it is improving.
A good BLAS/LAPACK alternative. There's OpenBLAS, and there's Apple's Accelerate. Both support only a part of Intel's MKL functionality. But, if we don't insist on 100% coverage (we're not) and are willing to accept missing operations to be slower, I could implement the most important ones in Clojure if nothing else is available.
A good GPU computing alternative. CUDA is not supported on Apple, and OpenCL has been discontinued by Apple. So that leaves us with Apple's Metal, which is a mess (or so I hear). So I wouldn't put too much hope on GPU, at the moment. Maybe much, much, later, with much, much, more experience…
Assorted auxiliary operations that are not in BLAS/LAPACK/Apple Accelerate, which are usually programmed in C++ in native-land. I'd have to see how many appear, and what I have to do with them.
Explore what's the situation related to tensors and deep learning on Apple. I doubt that Intel's DNNL can cover this, but who knows. Also, Apple certainly supports something, but how compatible it is with cuDNN and DNNL, is a complete unknown to me…
Who knows which roadblocks can pop up.

So, there's a lots of functionality to be implemented, and there's a lots of unknowns.

I propose to * Implement an Apple M engine for Neanderthal.* This involves:

buying an Apple M2/3 Mac (the cheapest M3 in Serbia is almost 3000 USD (with VAT).
learning enough macOS tools (Xcode was terrible back in the days) to be able to do anything.
exploring JavaCPP support for ARM and macOS.
exploring relevant libraries (OpenBLAS may even work through JavaCPP).
exploring Apple Accelerate.
learning enough JavaCPP tooling to be able to see whether it is realistic that I build Accelerate wrapper (and if I can't, at least to know how much I don't know).
I forgot even little C/C++ that I did know back in the day. This may also give me some headaches, as I'll have to quickly pick up whatever is needed.
writing articles about relevant topics so Clojurians can pick this functionality as it arrives.

It may include implementing Tensor & Deep Learning support for Apple in Deep Diamond, but that depends on how far I get with Neanderthal. I hope that I can do it, but can't promise it.

By the end of 2025, I am fairly sure that I can provide Apple support for Neanderthal (and ClojureCPP) and I hope that I can even add it for Deep Diamond.

Projects directly involved: https://github.com/uncomplicate/neanderthal https://github.com/uncomplicate/deep-diamond https://github.com/uncomplicate/clojure-cpp

Recurrent Networks Hello World in Clojure with new Deep Diamond RNN support on CPU and GPU

Fri, 26 Aug 2022 16:25:00 +0200

I've been busy in the last period working on new major features in Deep Diamond, one of which is the support for Recurrent Neural Networks (RNN). It's not been an easy ride, but I can finally show you some results! Big thanks for everyone that's helping me with this by buying my books (or subscribing to the upcoming editions), and Clojurists Together, who generously funded me in the past year to work on this.

I know that most of you probably don't have much more than passing familiarity with deep learning, let alone recurrent neural networks, and that's why I'll try to show a very simple example on the level of Hello World that anyone interested in machine learning and programming can understand and try.

So, enough talk, let's get down to business.

What are we doing

We are demonstrating a hammer: recurrent neural networks. Just kidding; we would like to create a (software) device that can learn to predict the next data point in a series. Depending on the data, this can be done in a number of ways (even by convolutional neural networks (CNN) that Deep Diamond already supported), one of which is RNN. So, we are creating a recurrent network, and training it with a set of data for this task.

An example of data that fits this task would be temperature at some place, stock prices, and any other (possibly infinite) sequence of numbers in one of more dimensions that have an ordinal relation, that is, has an abstract notion of time attached to it. Then, we are trying to forecast one or more values that are happening in the future (temperature the next day, or the closing price of a stock, etc.).

Since Hello World has to be dead simple, a real example would have too many opaque numbers, so we'll solve an artificially trivial task of teaching our network to predict the next number in the series for obvious series such as 1, 2, 3, 4, 5. Of course, in real life we rarely need to solve that exact task with such a bazooka as RNN, but if this is your first contact with time series prediction with deep learning, I guess it'd be just what works best.

Let's get down to code

First, we'll require a bunch of Clojure namespaces that we need.

(require '[uncomplicate.commons.core :refer [with-release let-release release]]
         '[uncomplicate.neanderthal
           [core :refer [ge dim amax submatrix subvector mrows trans transfer transfer! view-vctr
                         native view-ge cols mv! rk! raw col row nrm2 scal! ncols dim rows axpby!]]
           [native :refer [fge native-float fv iv]]]
         '[uncomplicate.diamond
           [tensor :refer [*diamond-factory* tensor offset! connector transformer
                           desc revert shape input output view-tz batcher]]
           [dnn :refer [rnn infer sum activation inner-product dense
                        network init! train cost train train-shuffle ending]]
           [native :refer []]])

All the AI in the world would be useless to us if we hadn't measured some data from the real world to feed it. Suppose (for the sake of Hello World), that I "measured" a narrow range of whole number domain by generating it from thin air. Does this data tell us anything about stock market? Of course not, and please does not expect any model that we train on this data to magically be informed on anything that could not be learned from the data it has seen.

(def simple-sequence (range -100 100))

(-100 -99 -98 -97 -96 -95 -94 -93 -92 -91 -90 -89 -88 -87 -86 -85 ...)

Now, I'll create a blueprint for an arbitrary recurrent neural network (RNN). This network has 3 recurrent layers (Gated Recurrent Unit cells), an abbreviation to one timestep, and two dense layers at the end. Please note that I used a completely arbitrary architecture. Neither this layer structure, nor the number of hidden neurons are optimal for this data; we are not even sure it's any good. If anything, it's probably a huge overkill. I chose it simply to show you how it's easy to construct with Deep Diamond(), as it will practically do everything on its own if you specify the bare minimum, that is "what you want". And we hope it'll at least learn to work well at the end, as non-optimal as it is.

(def net-bp (network (desc [5 32 1] :float :tnc)
                     [(rnn [128] :gru)
                      (rnn 2)
                      (abbreviate)
                      (dense [128] :relu)
                      (dense [1] :linear)]))

There is no place here to explain what RNN is and how it works internally, other than pointing that recurrent layers have an ability to handle sequential relations of its input by "memorizing" the signals that pass through it. The upcoming version of my book Deep Learning for Programmers (2.0) discusses RNNs in more detail.

Formatting the input data

The input of this network differs from fully connected or convolutional networks by explicitly modeling the time dimension, "t" in the ":tnc" format. Technically, you can feed it with any 3D tensor that matches its [5 32 1] shape, but for that data to be in context, it has to actually be arranged as 5 timesteps of a minibatch of 32 samples of 1-dimensional data.

We do have 1-dimensional data, but how do we fit our (range -100 100) sequence to its input? We do have more than 5 timesteps (we have 400), and we are far from 32 samples, since we only have one! We could try to just cram the sequence as-is by doing (transfer! simple-sequence (input net)) but this would be the "garbage in" part of "garbage in - garbage out". No. The solution is, as always in machine learning, to actually think what our data represents, and matching it with our knowledge of how the actual model intends to process its input.

What we do need is a bunch of 5-long sequences, such as [1 2 3 4 5] and the output that we would deem correct. In this case, I choose that the goal is to teach the network to output 6 to this input (or a number sufficiently close to it). So, the training data should be input sequences such as [3 4 5 6 7] and [-12 -11 -10 -9 -8], and target outputs such as [8] and [-7]. I hope you see how a bunch of these sequences, almost 400, and their respective target outputs could be extracted from simple-sequence.

The following function employs some stock Neanderthal () matrix functions to process the data and pack it into input and target output tensors [x-train] and [y-train]. I don't have time to explain each step, which is not trivial, but this is fairly standard vector/matrix/tensor stuff, well explained in both books from my Interactive Programming for Artificial Intelligence book series.

(defn split-series [fact s ^long t]
  (let [n (- (ncols s) t)
        c (mrows s)]
    (let-release [x-tz (tensor fact [t n c] :float :tnc)
                  y-tz (tensor fact [n c] :float :nc)
                  x-ge (trans (view-ge (view-vctr x-tz) (* n c) t))
                  s-vctr (view-vctr s)]
      (transfer! (submatrix s 0 t c n) (view-ge (view-vctr y-tz) c n))
      (dotimes [j t]
        (transfer! (subvector s-vctr (* j c) (* c n)) (row x-ge j)))
      [x-tz y-tz])))

Here's how the output looks like on an ever simpler example of 2-step sample sequences produced from 5 element long full sequence.

(def dummy (fge 1 5 (range 5)))

#RealGEMatrix[float, mxn:1x5, layout:column, offset:0]
   ▥       ↓       ↓       ↓       ↓       ↓       ┓
   →       0.00    1.00    2.00    3.00    4.00
   ┗                                               ┛

(def dummy-split (split-series *diamond-factory* dummy 2))

[{:shape [2 3 1], :data-type :float, :layout [3 1 1]} (0.0 1.0 2.0 1.0 2.0 3.0)
 {:shape [3 1], :data-type :float, :layout [1 1]} (2.0 3.0 4.0)]

This split produces 3 samples for training, each sample has 2 entries, and for each sample there is a desired output. The tensor printout does not show dimensions, which would be super hard to make sense anyway due to large dimensionality and enormous number of entries in any tensor of any use. We can extract a matrix view, in cases when it makes sense.

(view-ge (view-vctr (dummy-split 0)) 3 2)

#RealGEMatrix[float, mxn:3x2, layout:column, offset:0]
   ▥       ↓       ↓       ┓
   →       0.00    1.00
   →       1.00    2.00
   →       2.00    3.00
   ┗                       ┛

So, inputs are arranged in rows: [0 1], [1 2], and [2 3]. That's because tensor's default layout is :tnc, meaning that innermost grouping is channels ($C=1$), (mini)batch size ($N=3$) and time ($T=2).

Ok, so, finally, we transform our own data so that the network can learn from it.

(def full-series (fge 1 200 simple-sequence))

#RealGEMatrix[float, mxn:1x200, layout:column, offset:0]
   ▥       ↓       ↓       ↓       ↓       ↓       ┓
   →    -100.00  -99.00    ⁙      98.00   99.00
   ┗                                               ┛

(def train-data (split-series *diamond-factory* full-series 5))

[{:shape [5 195 1], :data-type :float, :layout [195 1 1]} (-100.0 -99.0 -98.0 -97.0 -96.0 -95.0 -94.0 -93.0 -92.0 -91.0 -90.0 -89.0 -88.0 -87.0 -86.0 -85.0)
 {:shape [195 1], :data-type :float, :layout [1 1]} (-95.0 -94.0 -93.0 -92.0 -91.0 -90.0 -89.0 -88.0 -87.0 -86.0 -85.0 -84.0 -83.0 -82.0 -81.0 -80.0)]

The printouts of long tensors show only a subset of the content of the tensor.

The actual network

The blueprint that we've created at the beginning of the article can be simplified for later reuse. Please note that it's not some super-opaque magical compiler. The network architecture is defined as a simple Clojure vector of straight Clojure functions!

(def architecture [(rnn [128] :gru)
                   (rnn 2)
                   (abbreviate)
                   (dense [128] :relu)
                   (dense [1] :linear)])

[#function[uncomplicate.diamond.dnn/rnn/fn--43604]
 #function[uncomplicate.diamond.dnn/rnn/fn--43604]
 #function[uncomplicate.diamond.dnn/abbreviate/fn--43609]
 #function[uncomplicate.diamond.dnn/fully-connected/fn--43493]
 #function[uncomplicate.diamond.dnn/fully-connected/fn--43493]]

This architecture is independent from the specific input dimensions. We create a blueprint by specifying the dimensions and the architecture.

(def net-bp (network (desc [5 32 1] :float :tnc)
                     architecture))

This blueprint is independent of the learning algorithm. It is a function that creates the actual network training program. In this case, I will use gradient descent with adaptive moments (:adam). Before we start learning, we initialize the network with random weights, automatically chosen by Deep Diamond() according to good practices, but you are free to use another initialization function that is more to your liking. Everything in Deep Diamond is modular and implemented in Clojure fashion of "assemble your own if you wish". If you are content with my choices, then everything is automatic!

(def net (init! (net-bp :adam)))

Here's the network printout, in case you

=======================================================================
#SequentialNetwork[train, input:[5 32 1], layers:5, workspace:1]
-----------------------------------------------------------------------
#Adam[topology::gru, shape:[5 32 128], activation: :identity]
 parameters: [{:shape [1 1 129 3 128], :data-type :float, :layout [49536 49536 384 128 1]} (2.1676771211787127E-5 -5.19975537827122E-6 5.7575953178456984E-6 2.380384103162214E-5 -2.894530371122528E-5 -2.4803119913485716E-7 -1.6729745766497217E-5 -2.902976348195807E-6 4.530028309090994E-5 5.464621608552989E-6 -5.7127394939016085E-6 1.593221713847015E-5 -1.1793279554694891E-5 -7.507987476174094E-8 -1.1438844921940472E-5 5.965541276964359E-6) {:shape [1 1 3 128], :data-type :float, :layout [384 384 128 1]} (0.0035437364131212234 -0.001584756188094616 0.0013698132243007421 -1.2431709910742939E-4 1.7048766312655061E-4 0.002398068318143487 0.00372034078463912 -0.0021170536056160927 8.799560600891709E-4 -0.0023348634131252766 8.223060867749155E-4 -1.1841517698485404E-4 -0.004045043606311083 9.459585417062044E-4 -0.0037800846621394157 -0.002804018324241042)]
-----------------------------------------------------------------------
#Adam[topology::gru, shape:[5 32 128], activation: :identity]
 parameters: [{:shape [2 1 256 3 128], :data-type :float, :layout [98304 98304 384 128 1]} (2.276021717761978E-7 3.804875632340554E-6 1.058972247847123E-5 2.8063212198503606E-7 1.8309128790860996E-5 -9.416969078301918E-6 -7.987003414200444E-8 1.0779360309243202E-5 2.222931470896583E-6 -1.2704362234217115E-5 1.5140467439778149E-5 -2.0407780539244413E-5 -1.5779027080498054E-6 -1.0661310625437181E-5 -3.834541985270334E-6 -1.0737002412497532E-5) {:shape [2 1 3 128], :data-type :float, :layout [384 384 128 1]} (8.910018368624151E-4 0.003690000157803297 -4.8378523206338286E-4 -0.0034620240330696106 0.0031146425753831863 6.610305863432586E-4 0.00204548635520041 0.001728582545183599 -0.0027434946969151497 0.007643579971045256 0.00425624568015337 -0.00295245717279613 1.8937387721962295E-5 -0.0027048818301409483 -0.0012806318700313568 -0.0028582836966961622)]
-----------------------------------------------------------------------
{:shape [32 128], :topology :abbreviate}
#Adam[topology::dense, shape:[32 128], activation: :relu]
 parameters: [{:shape [128 128], :data-type :float, :layout [8192 1024]} (-0.004568892996758223 -0.004355841316282749 0.005139997228980064 -0.005750759970396757 -0.004274052567780018 0.005599929019808769 -0.003351157531142235 -0.008299742825329304 -0.0031023912597447634 0.0013810413656756282 0.002118719043210149 0.0023180078715085983 -0.005323362536728382 -0.013326002284884453 7.393552223220468E-4 -0.013735005632042885) {:shape [128], :data-type :float, :layout [1]} (0.5049463510513306 -0.47153180837631226 -1.3509427309036255 -1.3017100095748901 -0.39814749360084534 0.8303372263908386 0.6530964970588684 -0.22249580919742584 -1.326366901397705 0.16360507905483246 -0.022157425060868263 2.0535836219787598 1.8076190948486328 -0.0799550786614418 -1.6791125535964966 -0.7451670169830322)]
-----------------------------------------------------------------------
#Adam[topology::dense, shape:[32 1], activation: :linear]
 parameters: [{:shape [1 128], :data-type :float, :layout [1 1]} (0.003621552372351289 -0.004416522569954395 2.548426273278892E-4 0.006995228119194508 0.0013199367094784975 -0.0018220240017399192 0.009454095736145973 0.003091101534664631 -0.01203352864831686 -0.014204473234713078 -0.007159397471696138 0.0039085038006305695 0.0029486482962965965 -0.009481357410550117 0.009158425033092499 -0.004999339580535889) {:shape [1], :data-type :float, :layout [1]} (-0.09112684428691864)]
=======================================================================

Training, finally!

Hey, I promised you a Hello World, and I've been beating the bush for half an hour formatting the data. And we didn't even touched the biggest obstacle: actually training the network. Is it hard? Yes, but not for the user. Now it's time for Deep Diamond() to beat the hell out of your CPU or GPU. But it will at leas do this on its own :)

Since this is a Hello World, we'll start with 50 epochs and see how it's going.

(time (train-shuffle net (train-data 0) (train-data 1) :quadratic 50 [0.005]))

"Elapsed time: 769.986155 msecs"
3238.298712769756

Hmmmm. The cost of 3000 and change does not look very good. Would more training help?

(time (train-shuffle net (train-data 0) (train-data 1) :quadratic 50 [0.005]))

"Elapsed time: 744.456035 msecs"
708.0862449675478

Still bad, but it's improving! For the sake of experinmenting, I've run this ten(ish) times, and the cost has been steadily decreasing, to the point that it looks good now.

(time (train-shuffle net (train-data 0) (train-data 1) :quadratic 50 [0.005]))

"Elapsed time: 720.427332 msecs"
0.13662090173881225

We don't have to stick to 50-epoch runs. Let's do 500 at once.

(time (train-shuffle net (train-data 0) (train-data 1) :quadratic 500 [0.005]))

"Elapsed time: 6325.427605 msecs"
0.008672867995529465

This looks nice. A thousand epochs might seem a lot, but considering the large network size, recurrent layers, and the scarceness of the training data, it might actually be appropriate. On the other hand, Deep Diamond did it blazingly fast, in a mere dozen seconds! In the world of machine learning, this is nothing.

(def question (tensor [5 1 1] :float :tnc))
(transfer! [1 2 3 4 5] question)

{:shape [5 1 1], :data-type :float, :layout [1 1 1]} (1.0 2.0 3.0 4.0 5.0)

(infer net question)

Dragan says: Requested subtensor is outside of bounds.
{:src-index -31, :src-cnt 1, :dst-index 0, :dst-cnt 32, :mb-size 1}

Using the network for inference

Now that we have our super useful network, we can stop and think: but how do we use it? First, we need data that could be interpreted as a "question". Our network needs a sequence of five numbers, and when fed, will answer with one number. Obviously, these should be provided as tensors of appropriate dimensions.

(def question (tensor [5 1 1] :float :tnc))
(transfer! [1 2 3 4 5] question)

However, invoking inference would cause the complaint from the network.

(infer net question)

Execution error (ExceptionInfo) at uncomplicate.commons.utils/dragan-says-ex (utils.clj:105).
Dragan says: Requested subtensor is outside of bounds.

Our network's input requires a minibatch of 32 samples. The infer function can handle more, and do the inference in mini batches of 32, but it can't handle fewer samples.

One of the solutions is to create another blueprint with the same structure, and transfer learned weights to the new network.

(def net1-bp (network (desc [5 1 1] :float :tnc)
                      architecture))

While we're at it, we don't need to create a complex network capable of learning (:adam or :sgd). We can create a much cheaper inference network that takes fewer resources.

(def net1-infer (net1-bp))
(transfer! net net1-infer)

Now, finally, give us the answer, network!

(infer net1-infer question)

{:shape [1 1], :data-type :float, :layout [1 1]} (6.016137599945068)

So, being asked what is the next element in the sequence of [1.0 2.0 3.0 4.0 5.0] (remember, we specified data type :float), our network answers [6.0161] which is close enough to the actual target value that we can mark this as correct. But, it's not a great achievement, since our network already saw this sequence in training. A hash map would have done much better job at guessing this. Let's try a previously unseen sequence.

(transfer! [10 12 14 16 18] question)
(infer net1-infer question)

{:shape [1 1], :data-type :float, :layout [1 1]} (17.750324249267578)

Not that off, but one would expect 19.0. What went wrong? We trained our network with ducks $(x+1)$ and asked it about geese $(x+2)$. What about griffons?

(transfer! [10 1 100 16 34] question)
(infer net1-infer question)

{:shape [1 1], :data-type :float, :layout [1 1]} (-6.679039001464844)

Now the answer does not make any sense, but would you be able to come with a better answer?

All, right, let's try with a sequence that is similar to the one we used in training, but is out of the range of data that the network has seen.

(transfer! [1000 1001 1002 1003 1004] question)
(infer net1-infer question)

{:shape [1 1], :data-type :float, :layout [1 1]} (96.41997528076172)

Nope, not much success, but we should not have expected any. The network can not answer question outside its domain of expertise.

Let's try with previously unseen data, but inside the domain that we used for training (floats from -100.0 to 1000).

(transfer! [37.4 38.4 39.4 40.4 41.4] question)
(infer net1-infer question)

{:shape [1 1], :data-type :float, :layout [1 1]} (42.39200210571289)

This is actually pretty close!

What about GPU?

Sure!

(def nvidia (cudnn-factory))

(def gpu-net-bp (network nvidia
                         (desc [5 32 1] :float :tnc)
                         architecture))

(def gpu-net (init! (gpu-net-bp :adam)))

(def gpu-train-data (split-series nvidia full-series 5))

Let's hit it with 1000 epochs right away.

(time (train-shuffle gpu-net (gpu-train-data 0) (gpu-train-data 1) :quadratic 1000 [0.005]))

"Elapsed time: 5734.688665 msecs"
4.561427662266859E-4

Clojure Sound - 5 - Double Click with Foot Control

Sat, 30 Jul 2022 17:45:00 +0200

In the last article we created a receiver function that listened to signals from our foot controller and started or stopped playback on consecutive clicks. The trouble with our player it that it only works until the end of the track. The start! function does not automatically rewind the playback. Neither stop! does that. Our program is responsible for detecting that the track should be rewound, perhaps by detecting that we reached the end of track (but even that is not fool proof, since audio infrastructure is not that precise). Now we have to think about a foot user interface that is useful and simple at the same time - there's not many different precise actions that a foot can do.

My idea at this time is the following: distinguish three gestures:

Button (un)pressed (value 0)
Button pressed (value 127)
Button pressed twice in a short time span (doubleclick). This can have two varieties:
- First pressed (value 127), then (un)pressed (value 0)
- (Un)pressed (value 0), then pressed (value 127).
- The values 0 and 127 always interchange, there's no way to send the same value twice in a row (with the MIDI controller I have).

This gives us 4 different signals from one button that we can work with, which doesn't seem that much, but on the other hand, we can't assume that the feet of our user that is doing all this stomping is able or eager for much more complicated stuff.

So, if I assume that the particular button is dedicated to a particular clip playback, I see the following actions:

If the button is (un)pressed (value 0), the clip should stop, regardless of the previous state. If the clip has not been playing, nothing changes.
If the button is pressed (value 127), the clip should start, regardless of the previous state. If the clip has been playing, it just keeps playing.
If the button is clicked twice in a row, the things might get complicated!

It's not exactly rocket science level complicated, but we still need to think what to do. In our desktop user interface toolkits, we are accustomed to working with a very high level API. We write functions that react to gestures (click, left click, right click, double click, wheel scroll, touch, etc.) but we are not responsible for detecting these gestures from the signals that our Magic Trackpad sends; the drivers and the operating system take care of that complexity. We just write actions that happen onDoubleclick.

But here, we have to take care of the whole stream of raw signals. How should we distinguish between an action that happens when the button is clicked once, and an action that should register that first click, and wait for the second one, which may never come? And, we would like the solution to be simple!

I drew a simple state transition diagram (with a pen on a napkin) and concluded that for our limited foot control it can be simplified to this:

As soon as the button is pressed, evaluate either start! (value 127), or stop! (value 0) function.
When the button is pressed shortly after it has been pressed the previous time (say, 600 milliseconds), treat it as a double click, and initiate the alternative action.

This leaves us with two tasks:

Detecting doubleclick
Deciding on a simple decision process for the alternative action (remember, the user should anticipate the action while playing the guitar).

The usual imports:

(ns my-midi)
(require '[uncomplicate.commons.core :refer [close! info]]
         '[uncomplicate.clojure-sound
           [core :refer :all]
           [midi :refer :all]
           [sampled :refer :all]])

Detecting doubleclick

If you went back to one of the previous articles and connect the println as a receiver for the controller you have, you'd see that each message contains its timestamps in microseconds. We can detect whether a "click" message comes shortly after the previous "unclick" simply by subtracting the previous message timestamp. The only trouble is that we don't have access to message history.

If we saved the last message, we have all the information we need; not only that we know both timestamps, we can even recognized whether the last message is of the same kind, and coming from the same button! Fast pressing different buttons does not count as "doubleclick". Clojure gives us several elegant facilities for state management. In this case, the state is internal, so we don't have to care about synchronization. We don't need what refs, agents, or atoms offer. I think that the simple volatile! can serve our needs rather well.

(defn play [line]
  (let [previous-message (volatile! (short-message :control-change 0 0))
        previous-timestamp (volatile! 0)]
    (fn [message timestamp]
      (future
        (case (command-type-key (status message))
          :control-change (let [doubleclick (and (= (data1 @previous-message) (data1 message))
                                                 (< (- timestamp @previous-timestamp) 600000))]
                            (vreset! previous-timestamp timestamp)
                            (vreset! previous-message message)
                            (case (data1 message)
                              80 (play-control (data2 message) repeated)
                              true))
          :program-change "We'll do something with other buttons later!"
          :default)))))

Building on the play function from the last article, I've added closures via let to keep track of the previous message and its timestamp, and a simple case to distinguish :control-change vs :program-change messages. I've also delegated the action logic to an extracted function play-control.

Alternative action

The play-control function receives the value and whether the click is double. What should it do? Of course, the first click starts or stops the clip. But, what happens with doubleclick when value is 0, and what when value is 127? The first detailed state transition chart was not that trivial. It all depends whether the clip position was 0, maximum value, or something in between. My idea is to enable rewind, so the main theme is "doubleclick should rewind the clip". But, is there a difference whether the clip was stopped or not?

For example, let's say that the clip was in the middle, and stopped. The first click starts it, and the second is detected after 300 milliseconds. We can rewind the clip, and it's logical to me that the clip should stop, instead of instantly emitting sound. After I analyzed all other situations, it turned out that the simplest logic does quite a logical thing.

I left out the state transition diagram and my analysis on purpose. I hope that this inspire you to think about your own applications that use similar controllers. It's unlikely that you practice guitar at this moment, and it's equally unlikely that you have the same foot controller. Or you do! Anyway, here's very simple logic of my play-control:

(defn play-control [clip ^long v rewind]
    (if (< 64 v)
      (start! clip)
      (stop! clip))
    (when (or rewind (< 0 (frame-length clip) (+ (frame-position clip) 1000000)))
      (frame-position! clip 0))
  clip)

Here's what it does. At the start, the clip is not playing, and the button 80 is either in on (value 127) or off state (value 0), depending on the state you left it when you used it prior to that. If the button is in on state, the first click stops the clip, effectively doing nothing that changes what you hear, since the clip was stopped anyway. That was needed because on my controller, there is a red led light that indicates the on/off state, and I want it to match the playback.

Then, when you press the button again, the clip starts playing. Whenever you press it again, it starts or stops the playback, matching the red light. But, you might want to rewind it for whatever reason in the middle of the clip. If the clip was playing, the first click will stop it, and the subsequent (double) click will start it again, and immediately move its position to the beginning. If the clip was stopped, the first click will start it, but the second will stop it immediately, and rewind it, so it's ready to play when you click it the third time. Effectively, if you press the button twice in a short timeframe in the middle of the clip, it will just continue what it did (play or wait) but from the beginning.

If the clip reaches the end, it will be silent, but the red light will still be bright (there's no way I know of that I can direct my controller to change that, as it seems to not listen to MIDI commands). The first click will issue the stop command, and the second will start the playback. Now, I decided to relax the requirement for rewind in that case. Even if these two commands were not issued quickly, there's no reason for the clip to stay at the end, because there's no sound there whatever you do, which is not very useful (in this use case).

If the clip somehow ended in end of clip, but stopped state, with the led light off, the first click will start it, and immediately rewind it to the beginning.

I have also noticed that the clip that is very close to its end, but not exactly there, is a good candidate for rewinding. So, everything under a second until the end is treated as "rewind whatever happens" (in this use case!).

At the end, the playback logic turned out to be much simpler than I anticipated when I analyzed all states where clip and the controller (with its red LED light) can be. It is an easy trap to overcomplicate things and create sophisticated universal solutions. In this case, even if we had a more high level API, it would need to be configured, and we would miss the special case when the clip is near the end. Sometimes the humble code is the right stuff. Clojure and simple go well together!

Clojure Sound - 4 - Ctrl-Left-Pedal

Wed, 13 Jul 2022 20:45:00 +0200

In the last article we managed to connect a MIDI controller and receive updates whenever something happens to its knobs and buttons. So what? As it is, nothing. Printing out messages is not of much use, beyond perhaps getting informed about how these messages look like, and what kind of data they typically contain. On another thought, exactly that is often valuable, because how else can we debug what's happening and learn how to use these devices. They typically don't come with terrific manuals.

Luckily, basic use revolves around receiving one of a few kind of standard messages, that typically contain only a handful of bytes, more or less conforming to the MIDI standard, so it's up to us to assign the meaning according to our use case. If we care to read the standard, we could try to pick the closest meaning in our application, but if that's not possible, or practical, well… you live only once. Let's celebrate creativity and be silly.

What I would like

Cool, so, at first, I have an audio recording of a few chords interposed with some talk and noise, just a single wav file. I'd like to discover how to play only selected segments (i.e. the chords I'm interested in guessing), with arbitrary repetitions. The purpose of this is not to create a perfect looper or learning device, at least not yet. At first, I'm just interested in creating the crudest audio player, and discovering how to connect the player to react to the input from the MIDI controller that we connected last week.

After that, it would be nice to see whether the foot pedal works in the same way, or it needs more poking.

As we discovered last week, the more modern Faderfox MX12 connects through USB and is listed as one of the connected MIDI devices. The Roland FC-300 foot pedal, does not support USB, so I connect it via a 5-pin MIDI cable to my USB sound card. It will probably be displayed as a MIDI in/out of that sound card (more on that later).

MIDI Controller (hands)

The usual imports:

(ns my-midi)
(require '[uncomplicate.commons.core :refer [close! info]]
         '[uncomplicate.clojure-sound
           [core :refer :all]
           [midi :refer :all]
           [sampled :refer :all]])

The following code is what we've done previously: find out the Fiderfox input device and open it.

(device-info)

| :description | Software MIDI Synthesizer                    | :name | Gervill             | :vendor | OpenJDK                            | :version | 1.0             |
| :description | Software sequencer                           | :name | Real Time Sequencer | :vendor | Oracle Corporation                 | :version | Version 1.0     |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [default]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [hw:0,0,0]     | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB | :name | USB [hw:1,0,0]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [default]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [hw:0,0,0]     | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB | :name | USB [hw:1,0,0]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |

(def mx12 (map device (filter #(clojure.string/includes? (info % :name) "MX12") (device-info))))
(def mx12in (first (filter transmitter? mx12)))
(open! mx12in)

| {:class "MidiInDevice", :status :open, :micro-position 464, :description "Faderfox MX12, USB MIDI, Faderfox MX12", :name "MX12 [default]", :vendor "ALSA (http://www.alsa-project.org)", :version "5.18.10-arch1-1"} |

Receiver

Now, we should receive the input from MX12, but instead of printing it out to the REPL, we would like to control something. The first thing that comes to my mind is to start and stop the playback. Nothing more. Let's say that I choose one of the buttons, for example the leftmost gray one. When it's pressed, and its led light is on, my sound recording should be playing. When it's off, my clip should stop.

The Clip

But how do I access the recording at all? In Clojure Sound, the simplest (and the most straightforward) way of playing a pre-recorded sound is as a clip. I can load the actual wav as a (Java/Clojure) resource, or URL, or any number of supported methods, create an audio-input-stream, and then create a clip out of it.

(def ssr (audio-input-stream (clojure.java.io/resource "justin/ssr-1.2.wav")))
(def ssr-clip (line (line-info :clip (audio-format ssr))))

Now the clip can be manipulated by Clojure Sound functions whose names speak for themselves, such as open!, start!, stop!, frame-position!, etc. Let's try it!

(open! ssr-clip ssr)
(start! ssr-clip)

| {:class "Clip", :status :open, :level -1.0, :active false, :running false} |
| {:class "Clip", :status :open, :level -1.0, :active true, :running true}   |

The clip that you provided (obviously, different than the one I'm using) should start playing. We could wait until it reaches the end, or we can stop it at any time.

(stop! ssr-clip)

| :class | Clip | :status | :closed | :level | -1.0 | :active | false | :running | false |

Note that wen you start it again, it picks up where it stopped, instead of starting all over from the beginning.

(start! ssr-clip)

| :class | Clip | :status | :closed | :level | -1.0 | :active | false | :running | false |

Should we want it to jump to any position, including the beginning, we can call the frame-position!, or the =tick-position! function. The following code rewinds the clip to the very beginning.

(frame-position! ssr-clip 0)

| :class | Clip | :status | :open | :level | -1.0 | :active | true | :running | true |

Receiver, again

Now we can return to the receiver function. One thing that comes to my mind is that I can dedicate a button to toggle between starting and stopping the playback. Let's say that I dedicate the first gray button from the left on the MX12 for that task. How do I know how to access its events? One way is to read that in the documentation, but the problem is that the documentation is often cryptic or nonexistent. The other is to poke at the device, and see the event log through the printing function that we've already used. That's how I've discovered that the gray button I'm interested has a :controller id 37, and sends :value 0 for off, and value 127 for the on state, in accordance to the MIDI standard. The on/off state is indicated on the device by a red LED light. A basic receiver would then just check for controller 37, and start! or stop! the clip (line) that we provide, depending on its value.

(defn play [line]
  (fn [message _]
    (let [data (decode message)]
      (when (and (map? data) (= 37 (:controller data)))
        (if (< 64 (:value data))
          (start! line)
          (stop! line))))))

The play function assumes that we're going to receive a Control Change MIDI message, and expects that the decoded data identifies the controller and value. We will talk more about different kinds of MIDI messages and how to handle the bytes that they deliver. Fortunately, Clojure Sound helps with decoding these bytes according to the message type, but it couldn't help if we'd asked for nonexistent data in a wrong format. That's why play is not a universal, do-it-all, function. The programmer (you!) has to put some thinking in designing it to fit the use case.

Now we connect this receiver function to MX12.

(connect! mx12in (receiver (play ssr-clip)))

| :class | MidiInTransmitter | :id | 1772000663 |

Now I'll press the button and I expect the clip to start playing….

Yes, I hear the sound. When I press the button again, the sound stops. Press it again, it continues.

Controlling playback with Roland FC-300 pedalboard

Now that I've connected my clip with my Faderfox MX12, I wonder how difficult it can be to control it with the pedal. The first challenge is connection: FC-300 does not support USB. Fortunately, my external USB sound card does have 5-pin MIDI ports, so I connected the pedal through this.

(def scarlett (map device (filter #(clojure.string/includes? (info % :description) "Scarlett") (device-info))))
(def scarlett-in (first (filter transmitter? scarlett)))
(open! scarlett-in)

Now I'm going to press some buttons on the pedal, and spy the output.

(connect! scarlett-in (receiver (partial println "Hello FC-300")))

The output looks something like this.

Hello FC-300 {:channel 14, :command nil, :data nil} 80025801
Hello FC-300 {:channel 14, :command nil, :data nil} 80226589
Hello FC-300 {:channel 14, :command nil, :data nil} 80426603
Hello FC-300 {:channel 0, :command :control-change, :controller 80, :value 127, :control :gp-control-5} 80517901
Hello FC-300 {:channel 14, :command nil, :data nil} 80626739
Hello FC-300 {:channel 14, :command nil, :data nil} 80826209

Note that there is an empty message at channel 14 every 200 milliseconds regardless of me pressing any of the buttons. I am not sure why this occurs, nor I found the explanation in the (70 page long) manual, or on Internet forums. I suspect it's some signal for time synchronization, but that's just me, an inexperienced MIDI beginner, guessing. On the other hand, the rest of the output is familiar.

There are two stomps (foot buttons?) dedicated to general purpose control, that fit my need. As per MIDI standard, they send controller ID 80. I can change that in the pedal settings to 37, and re-use the same function I've used for MX12, but this is cumbersome, might break what standard programs expect from this pedal, and 37 was arbitrary in my use case to begin with. I think, in general, this hardware is rather cumbersome to program, so it might be good idea to just accept what they send by default, and then do any re-mapping and re-routing in Clojure-world, where automation is much easier!

(defn pedal-play [line]
  (fn [message _]
    (let [{controller :controller
           value :value} (decode message)]
      (case controller
        80 (start! line)
        81 (stop! line)
        true))))

I varied the setup a bit, and now one button is dedicated for start, and the other to stop command.

(close! scarlett-in)
(connect! scarlett-in (receiver (pedal-play ssr-clip)))
(open! scarlett-in)

Does it work? Yes, it does!

Note that I haven't used any fancy GUI reactive library. I wanted to show a simple example. But we can guess that this forest of message codes and filtering can quickly become unwieldy. Maybe we need a state machine to handle this. Or we might use some atoms and refs? Or core.async channels? Sure, and you are free to use any of these in specific use cases. Maybe I'll discover some nice patterns that work elegantly. "Hardware" user interface and the challenges it brings is not that dissimilar from graphical user interface.

Two-way communication

Regardless of whether I press control buttons on MX12 or FC-300, the clip acts accordingly.

The only glitch in this setup is the LED indication for these buttons: if I change the state of the clip from one controller, the other is unaware of that change. For example, if I start the clip from the MX12, its LED will flash, but then when I stop it from the FC-300, it just keeps flashing. When I press the button 37 on MX12 it initiate stopping, but the clip has already been stopped, so this achieves nothing.

What we really need is a way to notify controller of this change. You might guess that we should send the same control change message to the controller, and it will update its hardware. Easy!

However, when I did this, nothing changes. It seems that the controller is built only to control other things, not to be controlled. I can't find any reference in the manuals that would indicate that these two controllers can react to appropriate messages to do what I need here. Information I've found on the Internet is very obscure, and seems to confirm my suspicions here, for FC-300. In general, it seems that most MIDI controllers are built to control other things, not to be updated themselves. When I think about this, it's expected: if the pedal is in position X, what is controller to do when I try to update? Turn on a motor and bring the pedal in the appropriate physical position? I guess that would be too much trouble for little gain. How much would such pedal cost?

Clojure Sound 3 - Hello MIDI Controller

Thu, 30 Jun 2022 23:59:00 +0200

You may not know that music instrument were connected devices even in the Stone Age, that is, many decades ago. In the 80's that even resulted in the standard that facilitated connectivity of heterogeneous devices. Typically, you'd have a keyboard, a synthesizer, some external knobs, maybe effect boxes, and many of them could be connected by cable and talk to each other, even though that synth might be 27 years old, and the keyboard 2 years old.

That's the power of standard. MIDI might not be technically impressive. Especially for 2022, its transfer rate is unbelievably slow. On the other hand, a random device that you'd pick up at the store, on a yard sale, e-bay, or even at the dumpster, is almost guaranteed to have a MIDI connector.

So, even though you'd have a hard time connecting your iPhone with your Linux laptop, my guitar can talk with my Arch Linux desktop just fine (I'm not kidding, it really does!).

Making that guitar send a meaningful data to my computer is another pair of shoes. MIDI is very basic, and it's up to the user program to make sense of the data it receives. It is likely that you'll want to use a random exotic device in a very specific way; after all, music and art are all about originality. But, to walk on the Moon, we first have to get up from our bed. That's today's objective: taking input from a MIDI device, and controlling something on our computer by twisting knobs.

The software

As for the software, we need nothing more than Clojure Sound(), of course!

You can read how to prepare a fairly basic Clojure project in the first article in Clojure Sound tutorial series.

(require '[uncomplicate.commons.core :refer [close! info]]
         '[uncomplicate.clojure-sound
           [core :refer :all]
           [midi :refer :all]
           [sampled :refer :all]])

The hardware

It is unlikely that you'll have the same controller that I do, but so what? Any MIDI device that has a button or a knob on it will do!

At this moment, I'm using Faderfox MX12, not because it's a great device (which it is!), but because it conveniently sat next to my table. Also, being a newer device, it supports MIDI-over-USB connectivity (otherwise, I'd have to use an old-school MIDI cable, and plug it into an external USB sound card that also has a MIDI jack, which I do have, but you likely do not).

If you have any device at home that is remotely related to music, please check it for MIDI support. If you don't, and you'll like to follow along, you may find a cheap old controller, any working MIDI device will do, as long as you can connect it and as long as it has at least one area to press or stick to twist!

What I would like

In the past several months, I've been learning to play guitar, which includes some basic of music theory. There is a nice ear training exercise where you listen to a chord (several notes played at the same time) and try to guess the exact chord that you've heard. If you have a training buddy, that is easy: one of you can play the chord, and the other could try to guess. Most of us do this alone, though. If I play the chord, I spoil the guessing part, which is the whole point of the exercise.

So, I thought of a solution. What if I could record a bunch of chords, and then somehow randomly play them? I would like this. There is a challenge, though: to record chords, I'd have to move fingers away from my guitar, and put them on my keyboard. Then, when guessing, I'd have to press keys for start, repeat, or end. But ear training likely requires that I actively strum my guitar to help my guessing by comparing with the recorded sample. That leaves the keyboard out.

Ideally, I'd use a foot controller, such as this Roland FC-300 (short of Foot Controller, I suppose with 300 feet :)

I would press one of stomps 1-5 to record 5 different chord examples, and then press buttons 1 and 2 to play and stop random samples, pressing stomps 2-6 to enter my answers. The software, Clojure program running in REPL, might display results of my guessing, or accumulate them for later scoring. There are many possibilities, but the first step that we need to solve is: how does our program talk with these controllers in the first place?

Roland FC-300 has an additional obstacle of not supporting USB, so I'd have to use additional MIDI-to-USB sound card. I'll leave that for later, and in this article we will talk to the Faderfox.

MIDI device

First, thing, we make sure the device is connected to our computer. Then, we list all MIDI devices that our computer know of with (device-info).

(device-info)

:description	Software MIDI Synthesizer	:name	Gervill	:vendor	OpenJDK	:version	1.0
:description	Software sequencer	:name	Real Time Sequencer	:vendor	Oracle Corporation	:version	Version 1.0
:description	Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB	:name	USB [hw:2,0,0]	:vendor	ALSA (http://www.alsa-project.org)	:version	5.18.7-arch1-1
:description	Faderfox MX12, USB MIDI, Faderfox MX12	:name	MX12 [hw:3,0,0]	:vendor	ALSA (http://www.alsa-project.org)	:version	5.18.7-arch1-1
:description	Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB	:name	USB [hw:2,0,0]	:vendor	ALSA (http://www.alsa-project.org)	:version	5.18.7-arch1-1
:description	Faderfox MX12, USB MIDI, Faderfox MX12	:name	MX12 [hw:3,0,0]	:vendor	ALSA (http://www.alsa-project.org)	:version	5.18.7-arch1-1

We see two devices that we already recognize from earlier articles, two appearances of my Scarlett 2i4 external audio card, and two appearance of Faderfox MX12. Why do these appear twice?

(def mx12 (map device (filter #(clojure.string/includes? (info % :name) "MX12") (device-info))))

#'user/mx12

Here's the first one.

(first mx12)

:class

MidiOutDevice

:status

:closed

:micro-position

-1

:description

Faderfox MX12, USB MIDI, Faderfox MX12

:name

MX12 [hw:3,0,0]

:vendor

ALSA (http://www.alsa-project.org)

:version

5.18.7-arch1-1

Note the :class MidiOutDevice. This instance can only send MIDI messages from computer to the device.

(second mx12)

:class

MidiInDevice

:status

:open

:micro-position

13220527818

:description

Faderfox MX12, USB MIDI, Faderfox MX12

:name

MX12 [hw:3,0,0]

:vendor

ALSA (http://www.alsa-project.org)

:version

5.18.7-arch1-1

The second one is MidiInDevice, which receives data, but doesn't know anything about sending.

In this case, the system sees the device through these two lenses. In other cases, one object might be capable of both sending and receiving, as in the case of the "Real Time Sequencer".

But how would our program know this, without us deciphering this from the :class output? The program could check for transmitter and receiver capabilities by calling appropriate predicates, and filter devices that match our requirements. In this case, we're interested in detecting button presses and knob turnings from the device, so we are looking for the transmitter.

(def mx12in (first (filter transmitter? mx12)))

#'user/mx12in

mx12in

:class

MidiInDevice

:status

:open

:micro-position

13220813146

:description

Faderfox MX12, USB MIDI, Faderfox MX12

:name

MX12 [hw:3,0,0]

:vendor

ALSA (http://www.alsa-project.org)

:version

5.18.7-arch1-1

Receiver

All right, we have a device object that sends MIDI messages. But how do we access these messages? We need to implement a receiver to our liking (or use an existing one in unlikely case that someone already implemented our custom creative requirements; remember, this is music, not accounting).

Usually, it's a PITA. Clojure Sound makes it easy by:

Decoding raw byte MIDI data into nice Clojure maps and keywords, and integers, and floats.
Helping plain functions to listen for MIDI events.

We'll create a trivial listener that just prints the messages to the REPL output, and connect it to mx12in. First, we have to open! the device, or it will refuse connections.

(open! mx12in)
(connect! mx12in (receiver (partial println "Hello Faderfox")))

{:class "MidiInDevice", :status :open, :micro-position 13220961811, :description "Faderfox MX12, USB MIDI, Faderfox MX12", :name "MX12 [hw:3,0,0]", :vendor "ALSA (http://www.alsa-project.org)", :version "5.18.7-arch1-1"}

{:class "MidiInTransmitter", :id 2000879663}

Here's what I get when I start turning a knob.

Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 1, :control :effect-control-2} 12732693869
Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 1, :control :effect-control-2} 12732693869
Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 2, :control :effect-control-2} 12732704461
Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 2, :control :effect-control-2} 12732704461
Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 3, :control :effect-control-2} 12732718154
Hello Faderfox {:channel 0, :command :cc, :controller 13, :value 3, :control :effect-control-2} 12732718154

Nice! My computer understands my controller!

As this article is getting big, and it's 2 AM, I'll consider this a milestone, publish it, play guitar for a short time, and go to sleep. I deserve sleep, too :) In the next article, we'll create a more useful receiver that will use the information from these messages to control these chords I've been singing praises for!