Depending on the number of people in the room, this could easily turn into insane waits between when you sent your message and when everyone hears it. This would probably work best in a conference call sort of situation where you have people mostly speaking one at a time rather than just a normal IRC room where there's a dozen conversations going on simultaneously.
You'd need some really great moderators for this kind of conversation. No conversation is perfectly linear - new ideas lead to forks, and a large group needs to somehow choose whether or not to take one. But if there's a queue of five folks waiting to speak, that's four opportunities for the tangent to either be taken or not taken, and exponentially many opportunities for the wrong tangent to be replied to.
It would be fun to design ways to manage the latency. I could imagine a modified push-to-talk where you hold down a key and the system gives you an audio cue to start speaking when the backlog of audio has dropped to an acceptable length. The system could also load balance speakers that way, giving people who make shorter or less frequent statements priority in the meeting.
