WebSonify

Nick Violi

@nvioli / https://nvioli.github.io/websonify/app/

Like many of you, one of the tools we use to measure our traffic is Google Analytics, and we can use a dashboard like this to see information about how many people are using our site and what they’re doing there in real time.

As great as information visualization is, I think there are a few drawbacks to this approach for interpreting live data. The most important one is that because sight is our most vivid sense -- which is the very reason that information visualization is such a great tool for displaying complex data -- we’re really only able to process one thing at a time; if I want to understand (in real time) how my website’s usage is changing over the course of the day, I pretty much have to watch this dashboard constantly for eight hours.

That’s what I’m trying to avoid; the motivation for this project was to see whether I could create an auditory interface to my site’s traffic data, and to learn how well it would function in communicating the data to me.

Biologically, we do different things with our ears than our eyes, so when we come to understanding structure in the world, [...] we can use our ears to understand something about it which is different to what we can understand with our eyes.

-David Worrall

If the visual sense is already highly loaded, sonification can be a suited alternative to provide additional information

-Thomas Hermann

Here’s the current state of the project. For the first version, it’s very simple: the only data being encoded is simply the number of visitors on the site: each note you’re hearing right now represents one person currently browsing the site. The visitors are separated into three groups by the Google Analytics goals that they are meeting, and each group is assigned an instrument.

The tool is set up to monitor GlobalGiving’s donor activity, though in a few minutes, anyone could connect it to their own Google Analytics account and sonify the data that’s being generated by their site, in real time. Currently, the cello represents the visitors furthest away from completing their donation, this could represent our nonprofit partners working in the backend, or anyone looking at the informational parts of our site. The violins represent visitors who are viewing one of our project pages. (If we were a retail site, the analogy is that these pages would be like the products we were selling, so these people represent potential donors, or browsing shoppers, who have not yet started the checkout process.) And finally the flute represents visitors in some part of the checkout process: they’re either on the cart and billing page, where they'll hopefully check out soon, or the thank-you page meaning they've just donated. Since these visitors are the most important to me, they get an additional auditory flourish called a musical cadence.

The goal classifications, as well as the instrument representing them are editable by the user, so that the tool can be modified to suit the goals and preferences of different people within the organization.

The result is an procedurally generated song, that’s theoretically infinite, changing over time to mirror the changes in the site’s traffic patterns.

One of my goals with this project was to create something that can exist around you, ambiently informing you about something you care about, while not monopolizing your attention. I always envisioned it as being played at a volume just around the level of cognition; I think it would be interesting to play very quietly in a lobby, lounge, or common area of the company whose data it’s representing, where it would blend into the architecture and become something of a data-enabled soundtrack.

            
function liveQuery() {
  gapi.client.analytics.data.realtime.get({
    ids: `ga:${constants.VIEW_ID}`,
    metrics: "rt:activeUsers",
    dimensions: "rt:goalId"
    // more metrics and dimensions could be added in the future
  }).then(response => {
    // response is ~ 743B!
    // response.result.rows: [["(not set)","90"],["1","2"],["11","2"],["3","2"],["4","40"]]

    const groups = mapGoalsToGroups(response.result.rows);
    // {groupA: 92, groupB: 40, groupC: 4}

    scheduleNotes(groups);

    // recurse
    setTimeout(liveQuery,60000);
  })
}

Let’s talk now about some of the implementation details, and I think I’ll keep that playing unless you're having trouble hearing me. The scheduling algorithm starts with a call to the Google Analytics Real Time API. We use their javascript library to pass the site’s id, and specify that we want the metric of “active users” grouped by the dimension of “goal ID”.

The response that’s returned is around three quarters of a kilobyte, and once the instruments are initially loaded, that’s all the network traffic needed, less than a kilobyte per minute. I don’t have to tell you all this, but it feels so powerful to me to have this entire library running on the client. It’s really amazing what we can accomplish with minimal server access, as compared to the more traditional ways to play sound or music for a user: streaming the entire sound file.

The important part of the response is reproduced here: an array of pairs which relate the goal ID to the number of users meeting that goal. First we run a short function to group the goals according to the user’s selections into the three buckets represented by the three instruments, then pass the aggregated totals to the scheduling function. Finally we set a timeout to recurse after another minute.


                    0   0   0   0   0   0   0
                    |   |   |   |   |   |   |
                    |   |   |   |   |   |   ⤷ 2^0 =  1
                    |   |   |   |   |   |
                    |   |   |   |   |   ⤷---- 2^1 =  2
                    |   |   |   |   |
                    |   |   |   |   ⤷-------- 2^2 =  4
                    |   |   |   |
                    |   |   |   ⤷------------ 2^3 =  8
                    |   |   |
                    |   |   ⤷---------------- 2^4 = 16
                    |   |
                    |   ⤷-------------------- 2^5 = 32
                    |
                    ⤷------------------------ 2^6 = 64


               92 = 1   0   1   1   1   0   0
                    |   |   |   |   |   |   |
                    |   |   |   |   |   |   ⤷ 2^0 =  1 * 0 =  0
                    |   |   |   |   |   |
                    |   |   |   |   |   ⤷---- 2^1 =  2 * 0 =  0
                    |   |   |   |   |
                    |   |   |   |   ⤷-------- 2^2 =  4 * 1 =  4
                    |   |   |   |
                    |   |   |   ⤷------------ 2^3 =  8 * 1 =  8
                    |   |   |
                    |   |   ⤷---------------- 2^4 = 16 * 1 = 16
                    |   |
                    |   ⤷-------------------- 2^5 = 32 * 1 =  0
                    |
                    ⤷------------------------ 2^6 = 64 * 1 = 64
                                                             ___

                                                              92

Here’s what that looks like. The song is written at 64bpm, so there are 16 four-beat measures every minute, and I should note that in all the visualizations we’re only seeing four measures (only because that's what seemed to fit comfortably on the screen), so this is 15 seconds of data if you want to think about it that way. The green row at the top, then, is playing eighth notes repeating on the quarter note -- which works out to 64 notes in a minute, the purplish one is playing a half note every measure, so 16 notes in a minute, the green one is playing a whole note every other measures, so eight notes, and the pink one is playing a whole note every four measures, or just four notes in the minute.

Now that we have a series of loops which will generate the correct number of notes, we make one adjustment: if we started all the loops at the same time, each minute would start with one note from each loop played at the same time. You can see they’re all stacked here at the left. To avoid this unpleasant chord (which also makes it hard to interpret the data-- it’s hard for your ear to discern whether you’re hearing three notes, or five notes, or more playing all at the same time). To deal with this issue, we introduce an offset variable which offsets the start of each loop by some amount related to the length of the note being played.

The idea to use the binary encoding seems to produce a good level of variability. In this case, all numbers between 64 and 127 will have those same quarter notes, but different numbers within that range will create different combinations of the other loops, so it’s a good balance between stability and variation.

Back to Demo

I’m going to switch back to the demo to point out that, because the numbers I’ve been using so far have been relatively small, I added this function to change the number of visitors represented, so the cadence can be sped up or slowed down, depending on the scale of the site’s traffic. If your site has 1000 unique visitors at any given point in time, or 10000, we can scale down the number of notes played by some factor.

I’ll use this to tell a quick story: earlier in the year when I was working on this, some days it would be working and some it wouldn’t; I was refining the algorithm. I was listening to it at work, as I often did, to get a sense of how it was working, and I had forgotten that we sent a large corporate gift card email and from one minute to the next, when the traffic went from this … (it’s a relatively slow time of the year right now), and then in the next minute when the emails went out and people started opening them and clicking through to our site, and it started sounding more like this (switch to 5 notes per visitor) … That’s when I knew I was onto something and started feeling good about this project because I was just passively listening to the traffic on my site and my ears told me something about the traffic that I didn’t know before.

Future Work

I’ll switch back to the presentation now and move to the final section where I’d like to discuss some of the future work that I’m interested in exploring in this project.

The implementation I’ve outlined so far is pretty simplistic, as it only encodes one dimension of data. There’s obviously much more data available: we could encode visitors’ geographic location, age, whether they’re a new or returning user, the amount of their donation, anything else that Google Analytics can provide us that would be interesting to us. To accommodate these additional dimensions of data, we certainly have many more attributes of the song that could be used, such as they key, note selection, note duration, note volume and envelope, to say nothing of the possibilities for adding filters and effects which could convey meaning. All of those elements are chosen somewhat idiosyncratically, and none of them have any representation at the data level.

However I’m wary of trying to encode too much data, as I might get to the point where understanding the represented data sonically requires as much attention and focus as the visualizations that I am trying to replace.

Ambient music must be able to accommodate many levels of listening attention without enforcing one in particular; it must be as ignorable as it is interesting.

-Brian Eno

[Y]ou only need to have three legs on a table. After two, three meant many, and that was it, you don't have to go any further than that: the three components of songwriting, the three chords of rock 'n' roll or the blues--that always seemed to be the number.

-Jack White

The other possible avenue for extending the current implementation is by adding additional instruments. There’s no special reason I’ve limited it to three, besides perhaps to honor Jack White’s fascination with the number three. But again, the more voices I add to the ensemble, the more difficult it will be for our ears to understand the nuances of the visitor patterns being represented.

For both possibilities, I’d be interested in experimenting with what limitations and possibilities exist.

Finally, it might also be interesting to give more creative control of the song to the user, so that more surprising and unexpected variations could be uncovered. Since the notes’ pitches and durations do not currently convey any meaning, these could be made editable in the UI.

WebSonify...

Is an infinite, procedurally-generated song, written in real time in response to live, streaming data.
Noticeably changes depending on current traffic patterns.
Sounds engaging and interesting enough to listen to for long time periods (though I'm biased)
Contains a pleasing balance between stability and variation; shows progression without ever changing dramatically.

As I wrap up, I want to recap some of the successes I feel this project has achieved. WebSonify is an infinite, procedurally-generated song, written in real time in response to live, streaming data. By listening to it at different times of the day and week, with different traffic patterns, the ability to represent these unique patterns sonically is obvious. I’m biased of course in considering the aesthetics of the song, but I find it engaging and interesting enough to merit continued listening over many hours. The method of using binary representation seems to create a good balance between stability and variation; each minute’s data is different from the last, so the song shows some progression.

Music makes you feel something [...] I don't remember the last time I was moved by looking at a line chart. I think it's the right medium for human data, a medium that has this very primal, visceral quality to it.

-Brian Foo, Data-Driven DJ

Thanks.

Built with tone.js, svelte, glitch.com.
Slides with reveal.js.