THE READER

The RFID reader is working! It took a while though. I did some search and ended up taking arduino codes from http://blog.formatlos.de/2008/12/08/arduino-id-12/ featuring the AF_SoftSerial library. It’s very well written and comprehensive. The scripts added a kicking off step and a disconnection detection as well. Great work! A simple processing sketch was made to display the identifier code.

The tag needs to be really close to the reader to be recognized, which is fine in my project. But since I probably needs to get reading from both ends of a miniature trumpet/horn, that might not be good enough. I’m trying to build some antenna for the reader and see if it could extend the accessibility.

THE TAG

The tags arrived too. I ordered 300 raw EM4001 RFID tags (meaning that they come with exposed coil antennae) and unfortunately a lot of them broke. A lot of soldering is ahead! But I got enough to start anyway, like a hundred.

THE SOUND

I got the wave sound shield from Adafruit and finished assembling while procrastinating. But I haven’t tried to use them because it doesn’t record to the card and I decided to fall back to use a computer in this phrase.

So now it works like this. Whenever a card/tag/sticker is seen/read, according to the mode an audio clip associated to the tag ID would be recorded or played, by the computer. All sound files are stored on the computer instead of any hardware/memory card as I previously planned. The processing sketch was adapted to keep track of a known card and pick the corresponding sound file to play/record.

[Videos to be uploaded soon]

THE NAME

Any thoughts on a name for the toy?

For the midterm, Li and I teamed up and decided to create an answering bot which you can trash-talk to when you really cannot sleep at night and don’t wanna bother your friends. Some apparent challenges are real-time level detection, smart (or vague) response behavior (to make sure it’s not too robotic), and the uncertainty that the monitor application might lock the access to the audio file that it’s writing into.

DIAL PLAN AND AGI

We started by wrapping up everything within a single php file keeping everything through AGI, and our extension dial plan looked as simple as:

The dial plan itself didn’t do much except for initializing some of the global variables, clearing up some of the tmp folders, starting the monitor recording and eventually move the recorded file to another place.

We decided to use Monitor instead of using continuous Record because we realized we cannot record while there’s an audio file being played back. So far I haven’t find a good way to bypass this synchronous behavior in AGI. (Probably need to involve more complicated shell scripts? )

Ideally I would use a time stamp as part of the folder name in combination with the caller id, but I’m having two issues here.

• I first tried to start the monitor within the php file using AGI, but with no luck. I did not spend too much time on that but for now it seems still doable by including the asterisk manager script and using the Monitor function. Will try next time.
• Due to the failure above, I just thought I could initiate the Monitor outside the php script in the dial plan, and pass the path to the monitor recording file to php script for further processing. However this is not that straightforward. AGI() application in dialplan does not take more arguments than the php script file name. I guess it could be a workaround to call it by passing the param in a querystring and use $_GET in php to retrieve that. I cannot remember whether that worked for me or not but anyway I didn’t go that direction.

So, I didn’t want to implement another protocal between dial plan and php either, so I just defined the arbitary naming convention for the monitor recording to be stored under ~/tmp/[callerid].

CUT THE FILE

The second step turned out to be creating a loop in the php script, to constantly monitor and analyse the recorded audio as an user input every ten seconds to provide reasonble feedback. But we were not that far yet. We did some research to find possible ways to analyse only the short clip of the audio file that we’re interested (last 10 seconds). Several feasible solutions:

• shntool, pretty easy to use and to embed in the php code, which is what we used for this project. We called shntool to cut the file on demand in each iteration by following command:
  

  shntool split -O always -f [cue file name] -d [output folder] [input file name]

  The cue file was generated automatically according to the total elapsed time and last interested time period, in following format:

  INDEX 01 00:10
  INDEX 02 00:20
  …

  Detailed documentation of shntool.

• sox, we just realized after we finished almost everything that sox can do this even simpler, by using trim effect as following:
  

  sox [source file name] [target file name] trim [SECOND TO START] [SECONDS DURATION]

  We haven’t tried this anyway, just throwing it here for documentation.

GET THE STAT

Another exciting feature in sox that we found was to do DFT on the sound data. The command is also simple:

sox [sound file name] -e stat -freq

Using the stat effect without the -freq option will give brief info which is actually good enough for basic level detection. Adding the -freq would print out the DFT data. One more thing is it’s printing to stderr by default, so if you want it be directed to a log file would need to call:

sox [sound file name] -e stat -freq -v 2> [log file name]

We were not sure about the DFT data we got, and solely looking at the numbers was not helpful enough for us to determine the thresholds for volumes. We would like to know the typical output of murmuring, normal talking and probably yelling, so we built a simple processing sketch to verify these outputs.

As shown in the screenshot, the processing sketch is polling the stat file created for each short chunk of the audio file. We failed to get a perfect realtime output reading in the processing since each file needs a little bit time to be trimmed and calculated. Also we were getting some weird problem which I cannot describe in English now that the AGI scripts would get halt if it’s trying to access a just-generated file too hasty. Anyway the shortest latency we got was a 5-second-late visualization of the telephone conversation.

MOODS

The processing sketch proved the stat info from sox was valid, and within acceptable range of delay, so we moved on to define different “mood” status for the answering bot. We put down the sound effects that we would like to create and broke them into over ten categories, including 3 different greeting mode, 3 levels of “interested” mode, 3 levels of “idle” mode and a frustrated mode which hangs up the phone — we called it a BOT HANGUP.

The actual recording of these sound effects didn’t take very long and was pretty fun to do. We ended up having 58 different responses according to different contexts. Li recorded almost all the sounds, hopefully he would enjoy talking to himself!

STATE MACHINE

The transfer between moods was implement in a simple state machine, by looking into current mood states and current (and some of the history) context, which contains the duration of the previous recorded audio piece, the average, minimum and maximum amplitude. There’s whole bunches of more information that are made available through the DFT however we didn’t have enough knowledge about sounds to push it further.

The basic idea is the harder the caller talks, the more interested the bot is supposed to be. If the caller stops talking, the bot falls into idle mode and would eventually go to frustrated mode and hangup. I guess we did pretty well in some of the transitions, but it did happen that the caller might be stuck in one of these states. Sometimes the splitted audio file just could not be updated and so was the stat file. We haven’t figured out why.

TESTS

We tried to trash talk to the bot, read poems and random news to “him”. He got mad pretty easily, or sometimes went retarded. We tweaked the numbers in the state machine, and we read more news to him. We stopped while he was not getting too excited about poems any more. You know these kind of stuff can take forever.

We asked some friends to call. Our friends on the floor were pretty cool and tolerated its stupidness. Some of the others didn’t feel comfortable with it and would not like to talk. One of my friend hated it and told me it’s the dumbest thing ever and I guess this was my fault.

CONCLUSION

I’m happy with what we did in this week and am excited that we made this framework work. The bot sounds pathetic sometimes, but probably I do want to call him while I cannot sleep. To make him smarter seems to be a pretty clear direction to go, we also got some really interesting ideas from Jorge that we might introduce some vague quotes that will fit any conversation, or let him talk about any sports or whatever suggestive topics that might help the conversation. Also we would like to build a web interface for people to retrieve their recordings the following day. The possibility of application is endless, but I will definitely apply this to one of my previous project, which is a dream sharing community website, to make submission via phone possible. I really love this class so far. Asterisk rocks!

If you would like to talk to the bot, or if you were so bored to read my blog at midnight, call:

1.212.796.0961 x 193

Here comes the visualization for the famous pink song:

First I changed the audio source to line-in so that I could play live with whatever I want. And I do love to snap to my laptop and see it react. *snap* *snap*

Also it’s fun to whistle at the program to see what’s happening.

The primary factor for visualization in this piece is the shape of diamond. I do not really want to mess up with 3D transforming and matrix things…so to make life easier I came up with following specs:

* Frequency – mapped to color of the diamond. Firstly the full spectrum was mapped to full Hue scope in a linear manner. That is to map 0 – 22050Hz to 0-360. However for music and vocal the frequency gathers around below 2000Hz, so I trimmed the sample frequency range to 0 – 16000Hz. Not much difference in the result though because it’s not enough. But at least it’s more colorful than before the change, and actually I don’t really wish to lose the sampling of higher frequency.

* Amplitude – mapped to saturation and size of the diamond. A little randomness was also added.

* History – I would rather create an overall feeling of the piece of the sound than to analyze its features at any timeslot. So I made the previous visualization of the sound fade into the background. I’m thinking of a better solution though… maybe after I get comfortable with the 3D stuff.

* Keyboard Controls – several commands could be made through keyboard shortcuts:

1) l (as in Leopard) – toggle log/debug mode (which shows a waveform of spectrum)

2) c – clear screen

3) + – increase threshold for amplitude, which filters frequency with amplitude above threshold to sample

4) – - decrease threshold for amplitude

After getting tired of snapping, I used miniAudicle to generate sounds for testing, which looks pretty good from the result’s aspect.

Link for applet (using pink.mp3)

http://leejayxia.com/processing/v5s_week2/

To play it with microphone you need to download Executable for Mac OSX

http://leejayxia.com/processing/v5s_week2/week2.zip

I played around with the functions the FFT provides and decided to use both the Forward Transform to create full spectrum of the piece of music, and the Averages to see what it would be different than the full scale. I put them oppose from bottom and the top of the screen so that I could compare the amplitutes of different frequences and the aggregated/average values while groups of frequences are combined.

As can be seen from the screenshot the “average” group has fewer bands.  However the shape doesn’t look right compared to the full spectrum, although the most loud parts of the sound gather around lower frequencies, the average shows the most heavy part in the middle. This is because the average was initiated and calculated as Log-Average in stead of Linear. To use a linear average just change the initializing statement from:

fftLog.logAverages(22, 3);

to:

fftLog.linAverages(30);

It turned out to be what’s expected. Still not making much sense though.

Another important factor I thought was important for music visualization is the beat. So I made splash-like effect to the canvas whenever a beat was detected. This is pretty straight-forward and not doing well with the sample music. The sound energy mode tracks just “significant spikes in the level” so it might not be accurate for all samples.

TO-DOs for week2

* Make visualization more comprehensive;

* Add dimension of time;

* Allow interaction for exploration into the music

* Try Frequency Energy mode for beat detecting, which is much for flexible with custom parameters in isRange() functionality;

applet link here: http://leejayxia.com/processing/v5s_week1/