Clojure Sound - 4 - Ctrl-Left-Pedal

Cool, so, at first, I have an audio recording of a few chords interposed with some talk and noise, just a single wav file. I'd like to discover how to play only selected segments (i.e. the chords I'm interested in guessing), with arbitrary repetitions. The purpose of this is not to create a perfect looper or learning device, at least not yet. At first, I'm just interested in creating the crudest audio player, and discovering how to connect the player to react to the input from the MIDI controller that we connected last week.

After that, it would be nice to see whether the foot pedal works in the same way, or it needs more poking.

As we discovered last week, the more modern Faderfox MX12 connects through USB and is listed as one of the connected MIDI devices. The Roland FC-300 foot pedal, does not support USB, so I connect it via a 5-pin MIDI cable to my USB sound card. It will probably be displayed as a MIDI in/out of that sound card (more on that later).

The usual imports:

(ns my-midi)
(require '[uncomplicate.commons.core :refer [close! info]]
         '[uncomplicate.clojure-sound
           [core :refer :all]
           [midi :refer :all]
           [sampled :refer :all]])

The following code is what we've done previously: find out the Fiderfox input device and open it.

(device-info)

| :description | Software MIDI Synthesizer                    | :name | Gervill             | :vendor | OpenJDK                            | :version | 1.0             |
| :description | Software sequencer                           | :name | Real Time Sequencer | :vendor | Oracle Corporation                 | :version | Version 1.0     |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [default]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [hw:0,0,0]     | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB | :name | USB [hw:1,0,0]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [default]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Faderfox MX12, USB MIDI, Faderfox MX12       | :name | MX12 [hw:0,0,0]     | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |
| :description | Scarlett 2i4 USB, USB MIDI, Scarlett 2i4 USB | :name | USB [hw:1,0,0]      | :vendor | ALSA (http://www.alsa-project.org) | :version | 5.18.10-arch1-1 |

(def mx12 (map device (filter #(clojure.string/includes? (info % :name) "MX12") (device-info))))
(def mx12in (first (filter transmitter? mx12)))
(open! mx12in)

| {:class "MidiInDevice", :status :open, :micro-position 464, :description "Faderfox MX12, USB MIDI, Faderfox MX12", :name "MX12 [default]", :vendor "ALSA (http://www.alsa-project.org)", :version "5.18.10-arch1-1"} |

Now, we should receive the input from MX12, but instead of printing it out to the REPL, we would like to control something. The first thing that comes to my mind is to start and stop the playback. Nothing more. Let's say that I choose one of the buttons, for example the leftmost gray one. When it's pressed, and its led light is on, my sound recording should be playing. When it's off, my clip should stop.

But how do I access the recording at all? In Clojure Sound, the simplest (and the most straightforward) way of playing a pre-recorded sound is as a clip. I can load the actual wav as a (Java/Clojure) resource, or URL, or any number of supported methods, create an audio-input-stream, and then create a clip out of it.

(def ssr (audio-input-stream (clojure.java.io/resource "justin/ssr-1.2.wav")))
(def ssr-clip (line (line-info :clip (audio-format ssr))))

Now the clip can be manipulated by Clojure Sound functions whose names speak for themselves, such as open!, start!, stop!, frame-position!, etc. Let's try it!

(open! ssr-clip ssr)
(start! ssr-clip)

| {:class "Clip", :status :open, :level -1.0, :active false, :running false} |
| {:class "Clip", :status :open, :level -1.0, :active true, :running true}   |

The clip that you provided (obviously, different than the one I'm using) should start playing. We could wait until it reaches the end, or we can stop it at any time.

(stop! ssr-clip)

| :class | Clip | :status | :closed | :level | -1.0 | :active | false | :running | false |

Note that wen you start it again, it picks up where it stopped, instead of starting all over from the beginning.

(start! ssr-clip)

| :class | Clip | :status | :closed | :level | -1.0 | :active | false | :running | false |

Should we want it to jump to any position, including the beginning, we can call the frame-position!, or the =tick-position! function. The following code rewinds the clip to the very beginning.

(frame-position! ssr-clip 0)

| :class | Clip | :status | :open | :level | -1.0 | :active | true | :running | true |

Now we can return to the receiver function. One thing that comes to my mind is that I can dedicate a button to toggle between starting and stopping the playback. Let's say that I dedicate the first gray button from the left on the MX12 for that task. How do I know how to access its events? One way is to read that in the documentation, but the problem is that the documentation is often cryptic or nonexistent. The other is to poke at the device, and see the event log through the printing function that we've already used. That's how I've discovered that the gray button I'm interested has a :controller id 37, and sends :value 0 for off, and value 127 for the on state, in accordance to the MIDI standard. The on/off state is indicated on the device by a red LED light. A basic receiver would then just check for controller 37, and start! or stop! the clip (line) that we provide, depending on its value.

(defn play [line]
  (fn [message _]
    (let [data (decode message)]
      (when (and (map? data) (= 37 (:controller data)))
        (if (< 64 (:value data))
          (start! line)
          (stop! line))))))

The play function assumes that we're going to receive a Control Change MIDI message, and expects that the decoded data identifies the controller and value. We will talk more about different kinds of MIDI messages and how to handle the bytes that they deliver. Fortunately, Clojure Sound helps with decoding these bytes according to the message type, but it couldn't help if we'd asked for nonexistent data in a wrong format. That's why play is not a universal, do-it-all, function. The programmer (you!) has to put some thinking in designing it to fit the use case.

Now we connect this receiver function to MX12.

(connect! mx12in (receiver (play ssr-clip)))

| :class | MidiInTransmitter | :id | 1772000663 |

Now I'll press the button and I expect the clip to start playing….

Yes, I hear the sound. When I press the button again, the sound stops. Press it again, it continues.

Now that I've connected my clip with my Faderfox MX12, I wonder how difficult it can be to control it with the pedal. The first challenge is connection: FC-300 does not support USB. Fortunately, my external USB sound card does have 5-pin MIDI ports, so I connected the pedal through this.

(def scarlett (map device (filter #(clojure.string/includes? (info % :description) "Scarlett") (device-info))))
(def scarlett-in (first (filter transmitter? scarlett)))
(open! scarlett-in)

Now I'm going to press some buttons on the pedal, and spy the output.

(connect! scarlett-in (receiver (partial println "Hello FC-300")))

The output looks something like this.

Hello FC-300 {:channel 14, :command nil, :data nil} 80025801
Hello FC-300 {:channel 14, :command nil, :data nil} 80226589
Hello FC-300 {:channel 14, :command nil, :data nil} 80426603
Hello FC-300 {:channel 0, :command :control-change, :controller 80, :value 127, :control :gp-control-5} 80517901
Hello FC-300 {:channel 14, :command nil, :data nil} 80626739
Hello FC-300 {:channel 14, :command nil, :data nil} 80826209

Note that there is an empty message at channel 14 every 200 milliseconds regardless of me pressing any of the buttons. I am not sure why this occurs, nor I found the explanation in the (70 page long) manual, or on Internet forums. I suspect it's some signal for time synchronization, but that's just me, an inexperienced MIDI beginner, guessing. On the other hand, the rest of the output is familiar.

There are two stomps (foot buttons?) dedicated to general purpose control, that fit my need. As per MIDI standard, they send controller ID 80. I can change that in the pedal settings to 37, and re-use the same function I've used for MX12, but this is cumbersome, might break what standard programs expect from this pedal, and 37 was arbitrary in my use case to begin with. I think, in general, this hardware is rather cumbersome to program, so it might be good idea to just accept what they send by default, and then do any re-mapping and re-routing in Clojure-world, where automation is much easier!

(defn pedal-play [line]
  (fn [message _]
    (let [{controller :controller
           value :value} (decode message)]
      (case controller
        80 (start! line)
        81 (stop! line)
        true))))

I varied the setup a bit, and now one button is dedicated for start, and the other to stop command.

(close! scarlett-in)
(connect! scarlett-in (receiver (pedal-play ssr-clip)))
(open! scarlett-in)

Does it work? Yes, it does!

Note that I haven't used any fancy GUI reactive library. I wanted to show a simple example. But we can guess that this forest of message codes and filtering can quickly become unwieldy. Maybe we need a state machine to handle this. Or we might use some atoms and refs? Or core.async channels? Sure, and you are free to use any of these in specific use cases. Maybe I'll discover some nice patterns that work elegantly. "Hardware" user interface and the challenges it brings is not that dissimilar from graphical user interface.

Regardless of whether I press control buttons on MX12 or FC-300, the clip acts accordingly.

The only glitch in this setup is the LED indication for these buttons: if I change the state of the clip from one controller, the other is unaware of that change. For example, if I start the clip from the MX12, its LED will flash, but then when I stop it from the FC-300, it just keeps flashing. When I press the button 37 on MX12 it initiate stopping, but the clip has already been stopped, so this achieves nothing.

What we really need is a way to notify controller of this change. You might guess that we should send the same control change message to the controller, and it will update its hardware. Easy!

However, when I did this, nothing changes. It seems that the controller is built only to control other things, not to be controlled. I can't find any reference in the manuals that would indicate that these two controllers can react to appropriate messages to do what I need here. Information I've found on the Internet is very obscure, and seems to confirm my suspicions here, for FC-300. In general, it seems that most MIDI controllers are built to control other things, not to be updated themselves. When I think about this, it's expected: if the pedal is in position X, what is controller to do when I try to update? Turn on a motor and bring the pedal in the appropriate physical position? I guess that would be too much trouble for little gain. How much would such pedal cost?