Interview

(Pd) is one of the most powerful and commonly used open source softwares for music and sound creation. Spearheaded by , it is completely free. About three years ago libpd was released - a library of Pd Vanilla (the basic version of Pd) that makes it possible to run Pd pretty much anywhere. Since then there have been an explosion of applications for PC, OSX, iOS, Android, Unity 3D and others powered by libpd. With libpd it is quite easy to use a regular Pd 'patch' to drive an experience - whether it is a musical app, sound design tool, game or installation.



leads the development of libpd and is supported by a wider community who are mentioned through the interview. It is quite amazing to see such a great (and stable) product being extended and used in an array of , all for free. More recently he has released the code for a new project that makes it possible to share audio between Android apps (something like what Audiobus does on iOS) - .




DS: Peter, thanks for doing this! It would be great to start with your background.



PB: I'm a mathematician by training. I used to be a college professor, doing work in algebra and geometry but I have always been interested in software engineering - I had been writing a lot of scientific software on the side and then at some point I left academia and became a software engineer. Right now I am working in speech recognition and libpd is a hobby.



DS: What got you started with libpd then?



PB: It was basically a coincidence. In the summer of 2010 I was leaving my academic job and was supposed to start a new job in the software industry and then it turned out that my academic H1B visa didn't translate to the business world so I had to wait three months to get a new visa. I found myself with three months on my hands and not much to do which was an incredible luxury. Then it just happened that I was at some kind of party for who is one of the leading people in the Pd world. I was chatting with him and he suggested that I should try to create a port of Pd to Android. That actually seemed like a great idea. At that time I was thinking that I needed to learn mobile development, wanted to learn git which I had never used before, and I guess the difference between commercial software development and academic hacking is that when you do commercial development you spend a lot of time reading somebody else's code and working with legacy code and so on. I hadn't done much of that in the past so I figured I could do with a project that involved staring at someone else's code just to get into the habit. Porting Pd to android seemed to fit the bill across the board so I started on that. So it was basically a learning project.



DS: That is quite amazing for a learning project then!



PB: Coincidences can be amazing..



DS: Any stumbling blocks when you started, what was the process like?



PB: I started by looking at existing code. Hans Christoph-Steiner, and had been working on an Android port of Pd and they had gotten to a point where they could basically launch Pd within an android app.. but it wasn't working basically. There were timing problems and it wasn't producing any sound. The good news was that there was existing code for me to look at and I am extremely grateful to them for setting up all the scaffolding. I had a good foundation to work with. As I was looking at the code I realised that the usual way of porting Pd to a new platform wouldn't work in this context because Pd expects to be in charge of things and the way Android apps are structured that kind of setup wouldn't work. It was pretty clear that the Android OS would have to be in charge and Pd would have to live somewhere inside it somehow. Getting this structure setup turned out to be surprisingly hard - Pd is not meant to work in that way. I went looking for something like an audio processing function in Pd - in a platonic kind of way it had to be there so the only question was where it was and how to get it out of there. I spent a few days staring at the source code of Pd until I got an idea of where the signal processing was happening and where the buffers were and so on. Once I had a sense of the general shape of Pd I was able to isolate the calls and repackage them in a way that made the audio synthesis available as an audio processing callback. That was the hard part, as soon as I had an audio processing callback the rest was pretty routine engineering.



DS: It's been three years now and there's been a mushrooming of apps using libpd. What do you feel about spearheading such a project?



PB: It's been a humbling experience and I am very happy to contribute to this, not sure what else to say! I'm having a lot of fun working on this and it is tremendously satisfying to see what people are doing.



DS: There have also been a lot of derivates from it - libpd with , . Do you see yourself taking libpd beyond iOS and Android?



PB: Two related aspects that I am rather excited about are the integration of libpd into openFrameworks, something that has done and also the integration of libpd into Processing that is something Peter Kirn and I have been working on and that opens up a lot of possibilities[Ed: Peter wrote to us after the interview asking to specifically mention Rich Eakin'swith audio in Cinder and libpd for Cinder]. In particular it extends what we already have for iOS and Android because you can create apps in Processing and openFrameworks and deploy them to those mobile settings as well. So tighter integration of libpd into creative coding frameworks is something that is very much going on right now and I have high hopes for.



DS: Mobile devices have their own challenges - both from a creative and technical standpoint. Do you think the ease of use of Pd (compared to 'real' coding languages) oversimplifies the whole process?



PB: I don't think the ease of use has anything to do with this. Once Pd has translated the graphical representation of the signal processing chain that the sound designer would create, Pd will execute that code natively. At that point you don't have any overhead from the user interface or the parsing of the patch anymore. It is not the user-friendliness per se that slows it down. What is slowing us down a bit at this point is that all the code is entirely portable to any system that has a reasonable C compiler on it. The consequence of that is that something that is portable everywhere is optimised nowhere. On some processors you might have vector and signal processing operations and you might have all kind of optimisations that can really improve performance that Pd doesn't currently doesn't take advantage of just because the code is generic. So the question is what to do about this and the related question is who should be doing this? A number of people have tried to integrate vector optimisations in Pd and I'm not quite sure to what extent they have been successful. The reason I have stayed away from this so far is that I don't want to create a fork of Pd. My goal has always been to touch the code base as little as possible and to create something that sits on top of Pd. Optimising Pd for vector operations would require changing the code base of Pd itself and at that point there would be the risk of creating a fork and that is one of the reasons why I have stayed away from it so far. Another reason is that it takes time which I don't have right now!



DS: This is something you had blogged about - the limitations of what Pd is and where it can go. Do you think because it is a community effort and because it is used for a variety of applications that some point in the future it would need an overhaul to match up to mobility?



PB: Possibly. It is something I have been discussing with Miller Puckette and other members of the Pd community. The other question I ask is - what is Pd exactly? Is it the existing code or is it a specification that happens to be implemented by the existing code and regardless of what the answer to it is right now, the question is what the answer should be. My belief is that we should really have a general definition of Pd and then we should have implementations of it - more than one. In a sense we are already on our way. In addition to Pd itself there is - he's been working on a new implementation of Pd and there's lots of interesting work in there. Also has been working on WebPd [Ed: now managed by ], essentially a Java Script version of Pd. Regardless of what Pd is, it is already turning into a specification that is being implemented in different ways and I think that is the right way to go. What we really need is someone who is willing to take the time and actually make this specification official. Right now it is informal - it is Pd if it kind of works like Miller's original code. I think we need a formal definition of what Pd is. It might be a master's project or a semester project for an interested student. There are ways to do this.



DS: Recently you rolled out support for Midi and Bluetooth and you have been working with openSL on Android to overcome issues there. Where do you see libpd in its current state going?



PB: So libpd consists of many moving pieces. There is the core library and that has hardly changed in a long time. As long as Pd itself doesn't change much I think the core library in C is done. Also, the Objective C components for iOS haven't changed much in a year and a half or so. I think that is a really good sign and means we haven't had real bug reports. So it looks like all those parts are fairly stable. What still needs to happen is more around the edges - integration with other packages like Processing, openFrameworks and Unity 3D. I've been thinking of best practices for Android development and one of the recent results of that is that I have an improved audio streaming layer for Android. That sits on top of openSL and is being used by libpd. It is more stable than what we had before and the roundtrip input to output latency is improved because I came up with a better heuristic for synchronising the input and output. Also a bigger change that needs to happen, but is mostly out of my hands, is that we need support for multiple instances of Pd. Right now Pd creates a global state and you can have only one instance of Pd in one process. That is unfortunate because that basically means that very desirable applications like Pd based VST plugins are not possible right now because you would need multiple instances.



DS: Is this on the roadmap?



PB: All the maintainers of Pd are aware of it and basically everybody agrees that it needs to be done. It is a question of time and to a certain extent a question of aesthetics. There's one obvious solution but it is just so unattractive that nobody has the stomach to implement it. If you go through the Pd developers list we kick around some new ideas for making this happen. So far we always conclude that the ugly solution looks like the right solution. The ugliness is one of the reasons why nobody has done it yet (laughs). I hope it will happen because it really is a significant limitation. With this limitation gone we would instantly see many new applications of libpd.



DS: Do you have any other pet projects other than libpd?



BP: If you look at my Github repository you will see a fairly recent major revision of the openSL compatibility layer in there. That actually uses ideas from the recent talk at Google I/O. That gave me a lot to think about. Once the openSL library was done, I had a few discussions with colleagues about what to do next. With their help, I was able to put together a layer on top of openslstream that provided support for multiprocessing, i.e., running multiple audio modules in parallel. Once that was done, I realised that I only needed to add a little bit of shared memory magic to get inter-app audio routing to work, and that's how the Patchfield project was born.



[Ed: This interview happened before the announcement of and Peter got back to us with the above less-secretive-answer. You can find out more about it and with the video below]
Full Post