Unity and libpd

pd-unity-screen

We are big fans of Pure Data at Two Big Ears. It makes it quick and easy to prototype ideas, or even treat our prototypes as ‘shippable’ products. Our most recent project (which you should hear about in the next week or so) involved a lot of work in Unity. There have been many attempts to get Pure Data (libpd) and Unity working. Patrick Sébastien had done amazing work in getting libpd to work with Unity on Windows (libpd-4-unity). In my spare time I took it on myself to extend support to OSX, Android and iOS.

Success? Yes (almost).

You will find some differences in the way it is all setup, if you have used libpd-4-unity in the past. The project structure needed streamlining to maintain cross-platform compatibility. Thankfully there is a whole library of helper APIs in Unity to make this possible. Let me drill into the specifics:

General:

libpd works *within* Unity. Unlike Kalimba (which is a great alternative for iOS/Android), it is a plugin for Unity and runs on the audio thread. This does mean that libpd works only with Unity Pro (although, there are workarounds to this that I haven’t tested or tried).

The ‘heart’ of the libpd setup in Unity is a single script: LibPdFilterRead.cs. It includes basic methods for initialising libpd and opening and closing patches. It automatically queries the sample rate and buffer size and sets things up depending on the platform/system.

Asset Hierarchy:

All Pd assets (patches, audio files, text files, externals) must be placed in Assets > StreamingAssets > PdAssets. libpd will not be able to find any of the resources in a build if the assets are placed in another folder.

asset-hierarchy

Setup:

It is recommended that you add LibPdFilterRead.cs to the camera or an empty game object in the scene. It is not recommended to have multiple active instances of LibPdFilterRead.cs in a scene (this could be enforced programmatically, but hasn’t been done).  The scene 01_LibPd_Basic.unity included in the project demonstrates this.

LibPdFilterRead.cs uses the onAudioFilterRead callback in Unity. You can think of it as a plugin insert on the DSP chain. This means that you can also send audio data into Pd from Unity (audio playback within Unity or microphone input). The scene 02_LibPd_ADC.unity included in the project demonstrates this.

The project includes two other scripts: GUIToggleScript.cs and GUITextScript.cs that demonstrate sending and receiving messages from Pd.

Creating your own project:

To use libpd-4-unity in your own project, copy the ‘Plugins’ folder , the ‘LibPD’ folder (this includes the C# bindings) and create the ‘PdAssets’ folder under StreamingAssets (create this folder too if you don’t have it already). Add LibPdFilterRead.cs to a game object, specify the patch name and you should be good to go!

filterread

OSX:

There doesn’t seem to be any major problems on OSX — I’ve tried building multiple projects and they all work fine. Externals work fine too! The full libpd API hasn’t been tested, but this will happen at some point in the near future.

The Unity Editor (previewing your game within Unity) at times fails to open a patch (SIGILL for those interested). I haven’t had the chance to track down this issue and it can be temporarily fixed by restarting Unity. This doesn’t happen in a build though.

This said, unity-4-libpd hasn’t been extensively tested so if you come across any problems do let us know!

Android:

Oh the joys of Android development!

The good news: it runs fine. No issues if you have a patch with no dependencies (audio files, text files, externals). The problem is that any data within an Android APK isn’t easily accessible by Unity/libpd. The workaround is to run a few lines of code and copy the data from the APK to the SD card (this is what a lot of apps do). Currently it only copies the Pd patch you’d like to open in Unity and nothing else. Ideally, it should be copying across the whole PdAssets folder to the SD card (or could borrow a trick or two from how libpd-android is setup). This is high on my priority list and will be fixed as soon as I get the time. If you have ideas for workarounds let me know!

iOS:

This is pending and is on the to-do list. In time!

Windows:

I haven’t tested the current build on Windows yet, but Patrick’s previous work seemed to have run fine. Feel free to let us know. I’ll test it as soon as I have my windows development pipeline working again.

Documentation:

The documentation/readme/Wiki needs work. I will add information on building the plugins for OSX and Android soon.

To conclude:

There is work pending, but the project in its current form should allow you to get started on creating cool procedural audio/music within your game! If you have ideas for further development, please fork and create a pull-request.

Thanks again to Patrick Sébastien and everyone else involved in the libpd/Pd project for all their hard work.

Go get it here.

Give us a shout if this helped or made life tougher for you. Follow us on Twitter or on Facebook.