I can't use WebRTC yet because I want to have support for IE and especially Safari & Mobile Safari. I want to add audio for an iOS App[1] that's using jsmpeg[2] for video streaming.
IE doesn't support Web Audio either. The project I linked above does contain a Flash shim for IE, which you may find helpful.
Any proper realtime audio stack (which ogv.js is not: it targets moderate latency streaming) is going to require a jitter buffer to deal with network/processing latency variation. Writing one is a non-trivial exercise (though not impossibly so).
In any case, I'd recommend using Opus (the audio codec used by WebRTC) over Speex.
Just curious, if your goal is realtime, why aren't you using WebRTC?