During Intel's 2018 CEO Keynote, I was part of the pre-show team that developed 15 minutes of technology facilitated performance. This project was part of Act 2 starting at 6:45 in the video. This was using Intel's Movidius Neurocomputestick, Jazz C Blues RNN models and custom Max MSP interface, the show's music director Kevin was able to play a call and response piece with the two AI musicians.
The system was set up to take live input from Kevin, via a midi-capturing keyboard (using Ableton) in addition to some control input using a MAX-based control panel running on an iPad. The notes were captured and sent to a server that had a neural compute stick installed. This server was running the AI model on the NCS which took in the midi notes and responded with a response sequence of notes. The response sequence was then sent back to the MAX panel which distributed it to a Unity server that rendered the response of the avatars as well as back through Ableton which rendered the music. This was all specified to run with a latency of less than 30 ms end to end. I also developed the Python, Max MSP and Ableton real time interface for the project, as well as operated the system during the live pre-show performance.
The AI model was based on a sequence-to-sequence conversion recurrent neural network. In effect it was trained to convert a sequence of input notes to a sequence of output notes, both of which were prepared for the training set by the musical director. The sequences were constrained such that the tempo, scale of the notes, and sequence lengths were fixed. We trained and deployed models for both bass and guitar. Models were designed, trained, and validated using TensorFlow. The NCS did not have the support for RNN’s at that time, so we had to write our own small script which converts our model to the blob file to run inference on the NCS.
I worked on developing, testing and training the music dataset and model with the team over the course of the project, using tools like tensorboard to visualize and tune hyperparameters. Currently, I am developing new encodings for other types of musical instruments as well as musical expression.