Paul and I recently finished up our tutorial on Using Visualizations for Music Discovery. The fast paced, 3 hour long presentation covered nearly every instance of music corpus visualization that we had found in the literature, or in casual “fan” based renditions.  This included visualizations of artist connections generated from millions of playlists by high powered supercomputers, all the way down to personal hand-drawn sketches on the back of a notebook.

I’ve embedded the slides from the tutorial above.  However, the format of the online slides do not include any of the video segments that we included, nor does it include the demo visualizations that we generated live on our laptops.  While our demo code (as well as the rest of the tutorial materials) is available online, I also prepared some back up video versions of my demos, just in case something went horribly wrong during the presentation.  I thought it would be a good idea to post these videos online for posterity.

The demo focused on exploring the “acoustic space” of music involving the lute and the folk guitar. Both instruments can be similar acoustically, but generally take part in culturally distinct genres. In the realm of audio-based genre classification, renaissance music that features the lute can get confused for folk music songs that feature the guitar. I wanted to focus on representative lute and folk music for the demo, and see if visualization helps to show the confusion that a classification might run into, and perhaps lead to ways of resolving it.

Rather than show all of the acoustic features at once, I chose to look at them one at a time. The first feature that I wanted to focus on was pitch information, or a general representation of what pitches, keys, etc. were present in the music.  For this and all demo videos, the blue nodes corresponds to lute music, while the red nodes correspond to folk music.  However, there are some “mislabeled” songs, as we will discover later:

The visualization shows how adjacent songs do in fact have similar pitched melodies, but can have very different styles or genre. This makes sense, since most popular genres use the same western arrangement of major and minor chords, etc. So, pitch information is essentially useless to separate the lute and folk music in our data.

The next acoustic data I looked at was timbre, which is a representation of the song’s tone or color.  This type of data is typically the “best” information to use for genre classification, and I was curious to see how the timbre separated these genres:

Interestingly enough, the timbre space “almost” separates the music clearly.  The softer tones of certain lute music are distinct from the rest of the folk music, which has harsher or brassier coloring.  However, the lute has a variety of timbres that it is able to create, and once the lutist uses this harsher plucking action, it sounds more like conventional folk guitar music.  At this point, it may be possible to make a “pretty good” separation using a hyperplane, but as indicated by the visualization, there would most likely be some significant error using timbre alone.

Luckily, our database of music included non-acoustic features, such as tags. The tags were simple terms or phrases that were applied to songs by human listeners.  Using appropriate text retrieval techniques allows us to see how the music was separated according to how it was tagged:

Here we see the two styles of music are clearly separated.  However, there is a problem… I had originally used the tag data to identify/label lute and folk music in the first place.  I quickly found out that some of this music was mislabeled… i.e. classical lute music was labeled as folk, etc.  So, even though the separation was clean, we now have exposed a new form of error.

The final approach that I tried was to mix the term and the timbre data in a “hybrid” representation:

This had the best result of any of the approaches that I had tried. It is no longer possible to separate the music by color (some of the red folk music is included among the blue lute cluster), I show that these red dots were actually mislabeled. In this fashion, the combined term + timbre features were able to “correct” each other, and a more valid representation of the genres were presented.

The code for this is all available on the website.  The code is written in R, and requires the packages rgl, RGtk2, as well as a GUI interface package I wrote called sculpt3d.

Also, if you’ve made it this far, you should check out  (the co-presenter) Paul’s blog.  He’ll be live-blogging the rest of the conference as it continues through the week.

Researchers in academia and the industry consistently use visualizations to better understand their data.  The standard two dimension scatter-plot is a staple of many exploratory data analyses.  However, there are many cases where two dimensions are not enough. For instance in the plot below, I’ve set up a pairwise plot between three distributions (x, y, and z).  The plots are duplicated on opposite sides of the diagonal (showing x vs. y …or… y vs. x).   I find it extraordinarily difficult to perceive the three dimensional structure of this data solely with the series of two dimensional plots shown here.

Luckily, R has a package called rgl that lets me plot in three dimensions using OpenGL.  It even allows for simple interactions with the mouse, like rotation and zooming.

This allows for a much better sense of the underlying data… but It’s still a bit limiting.  Many times, there is “interesting structure” in the three dimensional representation that I would like to focus in on, or perhaps there’s just a lot of junk that I want to get rid of.  Many times this structure is not isolated in one or two dimensions, but only becomes apparent through rotating and zooming the three dimensional display.

RGL has a useful method called “select3d” that involves selecting points in the plot, but it’s a bit tricky to use.  Using select3d involves typing into the R console, then clicking and dragging on the plot.  This produces output from the select3d function.  However, the output of the function is… another function!  It is then necessary to apply this produced function on the original data to determine which data points fall inside the selection range…. and then it’s completely up to you what you do at that point… do you filter them out?  Crop them? Color them differently than the others?  It becomes necessary to keep a further supply of methods at hand that can perform these routines on the output of select3d.

This is still all a bit too much to keep in my head for extended periods of time.  It’s also a pain to constantly switch back and forth between the three dimensional plot environment and the console in order to pare down the data.

So, I’ve cooked up a little GUI tool bar that makes selecting, labeling, cropping, and deleting points in the RGL plot much simpler.  It’s called “sculpt3d”, since it’s focused on altering and shaping the underlying data.  In order to launch the toolbar, you first install a small library, and then make a method call:   sculpt3d(x,y,z) where x, y, and z are the three dimensions you are interested in plotting.

I put together a small video that shows me playing around with it:

In the video, you can see how I choose a selection color (a mint green color that I thought would be noticeable against the bright rainbow colored points).  Then I crop the data on this selection.  This is a bit jarring because the crop function automatically zooms in on the selected points, and reverts them back to their original color.  The delete function works in a similar way, and is a little easier to follow.

Once I have a smaller collection of datapoints, I might be interesting in labelling them (assuming I passed labels as an argument to sculpt3d).  I can toggle the labels with a click of the “Label” button.

If I’m interested in saving the results of the pruning and cropping, I can access the currently selected points by calling the function sculpt3d.selected(), or the currently visible scene by sculpt3d.current(). This returns the same sort of logic vector that select3d does, so I can now use it to filter my dataset, save it under a new name, and come back to it later on.  Furthermore… since it’s in R, and uses a cross-platform gui (GTK+ via RGtk2 and Glade), it’s possible to use this tool and the data on any platform I want to.

Currently, I only have this tool on a local webserver, and it’s still a bit rough.  However, I thought I’d make it available.  To try it out, enter the following in R’s console:

install.packages(rgl)
install.packages(RGtk2)
rep='http://ethos.informatics.indiana.edu/~jjdonald/r'
install.packages(sculpt3d)

It’s currently source-only, so if you’re on windows, you’ll need to install RTools in order to compile everything.  You’ll also need the GTK+ framework and Glade (which should get installed automatically with RGtk2).

However, once that is done, you can check the demo out by entering:

library(sculpt3d)
demo(sculpt3d)

Thanks to Daniel Adler and Duncan Murdoch for rgl, and Michael Lawrence and Duncan Temple Lang for RGtk2.  Between the four of them, they’ve produced a lot of great stuff for R.

In the interest of self disclosure, I didn’t read the book, because I tend to be more interested in what people do than what they say to do. The cliff notes versions I’ve seen on line seem to imply that he has rediscovered the Pareto Principle, or the incredibly well worn 80/20 rule, along with a host of other efficiency principles that seem incredibly hackneyed.  I’m not sure what his unique twist is, maybe he’s good at telling stories.

It turns out Tim’s claim to fame is that he’s gotten very good at finding success in niche activities on technicalities, and then marketing the hell out of them.

1. He’s a Chinese Kickboxing champion, which he achieved by shoving the smaller opponents out of the ring, and winning through their disqualification.
2. He’s an expert at Argentine tango, where he has set a record for spinning.  This isn’t really a technical feat, it was more of a novelty act for the Kelly Ripa show.
3. He lectures at Princeton’s electrical engineering group (on entrepreneurship).

I think that very few people can actually benefit from what he talks about, mainly because he proposes “outsourcing” minute details of daily planning (to personal assistants in India… seriously!).  I don’t know if he’s thought about marketing his book in India (who would they outsource to?).  He also is against liberal use of cell phones and e-mail.

I want to mark this up as another “the secret to success is selling secrets to success” type of situations.  I could care less about yet another productivity guide, I’d rather hear about how he was able to network and promote himself.  That seems to be something he actually knows a thing or two about.  From what I hear, his self promotion drive irked a lot of people, and certainly took much more than 4 hours a week.  However, I have to hand it to him, it seems to have been very successful.

public function (it:Iterable<Iterable<T>>) : …{}

will not work in the intended way.  In fact, trying to pass a List<List<T>> or [[1,2,3],[4,5,6]] for “it” will cause errors.

This is a limitation for functional programming routines, especially functions that handle collections of collections like chain, zip, etc.

In order to handle these situations, it’s necessary to come up with a work-around.  For relevant methods, I have a function header that looks like:

function(it:Iterable<Dynamic> , ?nonIterableBehavior<Dynamic>->Iterator<Dynamic>):…

The “it” parameter is a simple single collection of any Type (Dynamic).  This will accept Iterable<Iterable<T>>, but also Iterable<T>, or a mixed Iterable of both iterable/non-iterable elements.  I’m mainly interested in the first case, but I’ll have to handle the latter two cases as well.

In order to handle the all of the cases, it’s necessary to detect if an individual element T is actually an Iterable<T2> (T2 equals some other type).  The straightforward method would be to use Reflect.hasField(“field”, iterator).  However, this will not detect the iterator() in Arrays.  Also, since we’re using a Dynamic type, we’ll have to test for null.  In the end, this should work for an Iterable tester:

public static function isIterable(d:Dynamic):Bool{
return (d != null && (Reflect.hasField(d,’iterator’) || Std.is(d, Array)));
}

As long as the object has an “iterator” field function that behaves in a consistent way, this will work for it.

Now that we can test for Iterables, we can handle non-iterables.  In relevant functions I use a function called nonIterableBehavior that transforms these non-iterables into Iterators, and then pass that resulting iterator along to chain as if it were a proper Iterator.  In my own classes, nonIterableBehavior defaults to something I thought would be appropriate.  For instance, in chain(), the default nonIterable function applies IterTools.repeat(“x”,1) to the non-iterable element.  In zip(), the default nonIterable function instead skips any nonIterable elements  (skipping is assumed by a function that returns null).

So… the net effect is that I can now call something like

ListTools.chain([[1,2,3,4],5,[6,7,8]])

and get back a List<Dynamic>:

{1,2,3,4,5,6,7,8}

From here, if I need to do anything further with the elements in the list, I’ll have to type check everything with Reflect/Std.is methods like Std.is(x,Float), etc.  However, in most of my cases, just printing out a list is useful.

One reason that linked-lists are more useful is that web developers are usually not coding with heavily optimized, low-level languages like C.  Many developers will code for virtual-machine platforms that do not allocate and handle arrays optimally for speed or efficiency.  Adobe’s AVM2 is a good example of this, and several people have noticed that linked-lists can outperform arrays for a lot of relevant tasks.  While there are workarounds for working efficiently in arrays, these can introduce a lot of cognitive overhead, especially when working with complex objects (such as XML/JSON data types).

On the server side, linked-lists are useful because it’s not always possible to store a simple index of items (people, pages, clicks, etc.) on one machine.  This is one reason that Google has started to work more with Lambda type operations such as MapReduce, and to spread the calculation of PageRank scores over multiple machines in a flexible and more fail-safe environment.  Lambda and linked list data types go together quite well because Lambda methods are normally intended to be “stateless.”  For each step, a typical Lambda method usually only concerns itself with an individual element, a function, and some output element.  For this reason, they can work on successive individual elements of a linked list without having to keep the whole list or index in memory.  The drawback to linked lists are that you don’t have random access to any of the elements (you have to go through them in order), and you incur a significant amount of memory overhead for each element (each element has to contain a pointer to the next element, which essentially is a memory location… i.e. for a 32 bit machine, it’ll take at least 4 bytes).  So, if you’re only storing (common) Integers in a linked-list, then the space required to store the pointers is going to be equal to the data itself.  However, keeping these limitations in mind, they still can be very useful in appropriate contexts.

One of the reasons that I’ve been interested in haXe, is that it includes a Lambda class as part of its API.  This fact, combined with its concise type-checking facilities, means that you can create very powerful Lambda type operations, and implement them with strong type checking.  This is great for performance and debugging, and haXe’s target options for client side and server side could make this type of programming extremely useful.

So to summarize, array’s performance benefits can be lost when dealing with client/server side virtual machines, or in large scale server side information processing.  I think it’s worth checking into some functional programming techniques to improve performance or scalability for some common web site tasks, and I think haXe is a wonderful place to start.

To this end, I’ve started to put together a library that extends the basic functionality of the haXe Lambda class, and offered it as part of an open source haXe library called “sugar-hx“.  The library includes functions taken from different Lambda method implementations in other languages.  It currently includes scan, unfold, zip, unique, bifurcate, as well as several “grouping” functions.  If you use it, or have suggestions, let me know!

I’m at the 2008 ISMIR conference on music information retrieval, presenting a poster paper with my co-author, Claudio Baccigalupo.  I talked a bit about the paper in a previous post.

I was surprised at how prominently “social” sources for musical understanding were featured in the projects and demos, so our work on using playlists/streams to understand relationships between genres was right at home.  There were also some fantastic demos, including the “Aura Music Explaura” which allows for “steerable recommendations” by using tags.

The way that Francois and Paul (the project leads) implemented the tag interface was particularly brilliant.  By leveraging the common visual motif of a tag cloud, the interface communicates the frequencies of the tag labels applied to the music.  However, the tags themselves are interactive.  By dragging the tag term boundaries, one can change the “weight” of the tag.  This also increases or decreases its relative visual size.  Doing so will alter the importance of the tag in selecting similar artists.  So, hypothetically someone could alter the amount of “emo” they want in their music results.  Paul gave me a quick demo, but I really hope this ends up publicly accessible soon, it’s killer.

The first incident is with Adobe Flash 10 (Astro).  While there are a few very nice features that Astro brings (more useful text rendering/layout methods, and support for shaders), it was a let down on several fronts.  The 3D capabilities that were teased at through early demo movies turned out to be extremely limited. It quickly became evident that it would be very convoluted to manage some common 3D methods.  Furthermore, the advanced 3D “Pixel Blending” won’t even use the graphics card.  However, at the very least, the new version of Flash will fix some memory handling problems.

To Adobe’s credit, they have been instrumental in leading the charge for Javascript 2.0, which was basically going to be the new version of their Actionscript language.  Brendan Eich (the creator of Javascript and a Mozilla Lead) recently sat on a standards body that was to determine the new language.  Also present were Google and Microsoft.  A recent post by Eich hints at the frustration the standards body met in agreeing upon a standard… presumably because none of the commercial entities wanted to grant the other any sort of advantage (as might be the case between Adobe and Microsoft).

As web developers, it’s disheartening to think that there’s really no new development environment or language on the horizon to look forward to.  As such, I would argue that it’s a good time to give the haXe language a try.  It’s been one of the few bright points I’ve seen lately in the realm of web development.

Claudio Baccigalupo and I have a paper at ISMIR entitled Uncovering affinity of artists to multiple genres from social behaviour data.  The paper details a project we worked on for the past year or so involving popular music listening activity from a pool of MusicStrands (MyStrands) users.

We provide not only the paper, but also the dataset and the code used in our analysis.  All of this is available at the website we have set up for the project.

The main contribution of the project is an analysis and illustration of genres as “fuzzy sets” rather than boolean labels.  Through a co-occurence analysis of hundreds of thousands of user playlists, a frequency based “affinity” metric is formed between artists and genres.  This affinity metric is a more detailed expression of the style of a given artist’s music.  The idea and awareness of predominant genres are a trivial part of any person’s understanding of the vast corpus of popular music.  However, genres typically are used as boolean categorical labels.  I.e. an artist is understood to be associated with only one given genre.

By expressing a connection to multiple genres through our affinity metric, a more detailed picture of the artist emerges.  We give a lot more examples in the website, so be sure to check it out.