> In 1973, he recalled, he persuaded Shannon to give the first annual Shannon lecture at the International Information Theory Symposium, but Shannon almost backed out at the last minute. “I never saw a guy with so much stage fright,” Berlekamp said. “In this crowd, he was viewed as a godlike figure, and I guess he was worried he wouldn’t live up to his reputation.”
> In 1985 he made an unexpected appearance at the International Information Theory Symposium in Brighton, England. The meeting was proceeding smoothly, if uneventfully, when news raced through the halls and lecture rooms that the snowy-haired man with the shy grin who was wandering in and out of the sessions was none other than Claude Shannon. Some of those at the conference had not even known he was still alive.
> At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. The audience cheered and lined up for autographs.
> At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling.
When Shannon first derived his famous formula for information, he asked von Neumann what he should call it and von Neumann replied “You should call it entropy for two reasons: first because that is what the formula is in statistical mechanises but second and more important, as nobody knows what entropy is, whenever you use the term you will always be at an advantage!
Yeah, I would characterize it as a mathematical theory of -symbols- and that information has different definitions. However, the argument can be made that symbolic representation is a principle of reality.
I don’t think you can meaningly disentangle “information” and its representation without doing violence to the underlying theory. Nor is a discussion of “symbols” coherent without a reference to the information encoded. Grounding the idea of information in the logical and physical limitations of representation is part of the genius of the theory, not an unfortunate epiphenomenon.
From what I understand, before Shannon showed up no one knew how to calculate the Channel Capacity of a medium and how to deal with noise in the medium.
So people just yelled louder or yelled repeatedly to get things across with better reliability. And when channel capacity was exceeded with everyone yelling too much, no one knew it had exceeded and lot of energy and time was wasted with increasing errors in the system.
Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things.
> Doesn't that feel like a repeat with whats happening in social media and news media these days? Or is it just different things.
Maybe. It can feel like this, but applying mathematical models to social phenomena is tricky. You have to map formal parameters to fuzzy, poorly understood factors. How do you define, precisely, what is a "channel" on a social network? What is "signal", and what is "noise"? People can and do prove anything by using slightly different (or inconsistent) definitions here, getting the numbers to line up just like they want them to.
My understanding is that a better way to approach such mapping would be to shove in probability distributions in place of hard-to-map exact parameters - abstracting away choices and measurements lets you see the wider context here. This pushes the problem to defining the appropriate distributions, but I think that's more tamper-proof. Unfortunately, the results may come out next to useless - e.g. probability distributions so wide you could sail a carrier strike group through them.
I'd love to know what's considered the correct, robust approach to such problems.
I think OP has a good point. Although language if fuzzy, so are "binary" signals once you drill down to the physical layer. The fuzziness of language goes away if you allow yourself to assume an idealized language embedding model. Similar ideas are in an epsilon ball around the embedding of some concrete linguistic instantiation of the idea.
An example of a binary signal: "It is unambiguously true that a lab leak did not occur in Wuhan." The spread of this binary signal could be measured using the technique described above. A BERT embedding may suffice.
The channels are just tweets, DMs, and Vox articles.
Your analogy doesn't sound far off from what information theory describes. Still, we need to be cautious of applying information theory to areas in which it might be over extended and may not be suitable. Claude Shannon himself was aware of this trend and wrote his essay "The Bandwagon" [1] urging caution and diligence in applying information theory to different fields.
One key insight of Shannon's is that to study information as he did, you need to remove "meaning". You could measure the information content of anything and it doesn't matter what the symbols mean. Paradoxically, a random sequence of characters of the same length as this message would contain more information than this message that I am writing.
---
The connection you draw to social media is cute, and I think it could help as an analogy in some cases. The Bit Player (2018 movie) makes a similar cute analogy to robust communication in marriage, that rather than saying the same message louder (yelling) or repeating the message (nagging), you should find many different ways to say the same message. And you could add an error-correcting code like "I love you".
I personally think it's a cute (not disparaging) way to think about communication among humans, but note that we still need to rely heavily on meaning, to decide what it even means to say the same message with different words.
What goes on with social media can be seen through lenses such as decision theory, and game theory, and graph theory, ..., and I think those fields have more to offer from the get-go than information theory. In any case, I personally would lean heavily on social science to try to get a better understanding of social situations, and to consider possible "interventions".
In particular, I do not think the kinds of communication errors that occur between humans are the same as e.g. bit flips. And to go back to the original point, it's tempting to conflate "information" (as in information theory) with "knowledge", or "understanding", or "truth".
Now we need an analog of Shannon in social sciences to offer a Theory of Meaning. Then perhaps a century or so where there are still incompatibilities with the theory of information that prevent a bigger picture from emerging, and eventually an all-encompassing theory ushering humanity into an Era of Meaning.
Ted Chiang could probably write a plausible future with that as a writing prompt.
> Now we need an analog of Shannon in social sciences to offer a Theory of Meaning.
That's called semiotics and it was first developed in the late 19th century by Ferdinand de Saussure.
> Saussure approaches theory of language from two different perspectives. On the one hand, language is a system of signs. That is, a semiotic system; or a semiological system as he himself calls it. On the other hand, a language is also a social phenomenon: a product of the language community.
> One of Saussure's key contributions to semiotics lies in what he called semiology, the concept of the bilateral (two-sided) sign which consists of 'the signifier' (a linguistic form, e.g. a word) and 'the signified' (the meaning of the form). Saussure supported the argument for the arbitrariness of the sign although he did not deny the fact that some words are onomatopoeic, or claim that picture-like symbols are fully arbitrary. Saussure also did not consider the linguistic sign as random, but as historically cemented.[a] All in all, he did not invent the philosophy of arbitrariness, but made a very influential contribution to it.
> After his death, structural and functional linguists applied Saussure's concept to the analysis of the linguistic form as motivated by meaning. The opposite direction of the linguistic expressions as giving rise to the conceptual system, on the other hand, became the foundation of the post-Second World War structuralists who adopted Saussure's concept of structural linguistics as the model for all human sciences as the study of how language shapes our concepts of the world. Thus, Saussure's model became important not only for linguistics, but for humanities and social sciences as a whole.
I think it’s a different thing but an interesting analog. In conventional information theory you can add another channel. I guess the social media metaphor is like crowded wifi or if I’m being clever alohanet... question is, how do we add another channel here?if you’re talking about social discourse rather than a physical medium are we talking about the necessity of class hierarchy? Is this what the Tower of Babel was about?
I think the social media issue has more to do with the heavy application of ML curation to feeds and ads creating feedback loops where people are really talking to themselves using a computer rather than other people who might confront them.
Many know Shannon as the author of "A Mathematical Theory of Communication" and the founder of information theory, as well as the brain behind "Communication Theory of Secrecy Systems"
Both papers derive from a technical report, "A Mathematical Theory of Cryptography"[0], written by Shannon in 1945 — 3 years before his breakthrough papers in the Bell Labs Technical Journal.
Cautionary Tales is my favourite podcast. Most episodes concern either interesting (mis)use of statistics or cognitive biases, both of which are topics I find inherently interesting, but Tim Harford and team do a brilliant job of the storytelling too.
Tim Harford's other podcasts are great too - More or Less, and 50 Things Which Made the Modern Economy.
Does anyone have any recommendations along similar lines?
I recommend reading "A Mind at Play", a biography about Shannon. I found it to be a decent read. Funnily enough, I knew about him more through the unicycling community than I did through computer science. It wasn't until a few years ago when I realised what sort of impact he had on the world.
I had forgotten (or never fully appreciated) what Shannon did and what other early pioneers did in computing. I have a better appreciation for all of them now.
One other highlight was Shannon's early childhood and using electrified fencing to communicate with others in his farmland neighborhood.
A really good book is "The Idea Factory" by Jon Gertner. It describes Bell Labs in this period, and through the book basically tells the story of where modern life was born: everything from growing crystals to transistors to cell phones to radar was created at Bell Labs.
Claude Shannon was also an early pioneer in artificial intelligence, the minimax algorithm applied to game trees was one of his inventions. He built several machines to play games and demonstrate his ideas, some of which can be seen in the article photos.
I compiled a list of his game gadgets, most of which still exist in the care of the MIT museum.
Shannon was awesome. It seemed like after he wrote his groundbreaking information theory paper he did whatever the hell he wanted for the rest of his life in Cambridge. Made all kinds of contraptions and devices.
An interesting tidbit from Hamming in The Art of Doing Science and Engineering [1][2] on the name "Information Theory"
> Information Theory was created by C.E.Shannon in the late 1940s. The management of Bell Telephone Labs wanted him to call it “Communication Theory” as that is a far more accurate name, but for obvious publicity reasons “Information Theory” has a much greater impact—thus Shannon chose and so it is known to this day.
Here's a very so-so promo site for The Bit Player. I saw it about a year ago (Curiosity Stream) and thought it was pretty decent. I somewhat sheepishly admit that, for a while, I thought that they had somehow interviewed the real Claude Shannon.
Although not devoted to Claude Shannon, Jim Al-Khalili's documentary "The Story of Information" does discuss Shannon's significant contributions and is how I first became aware of Claude Shannon.
Shannon was cursed by being the inventor of information theory. It seems like this wonderful thing; who wouldn’t want to invent information theory? But as an inventor, you’ll never top it. You’ll forever live in your own shadow. And for someone that enjoys inventing things, that’s... not a happy thing.
Nothing I've read about Shannon suggests that he felt "cursed" in his later life. Is there some reason to think otherwise, or is this just assuming that anybody who does something big early feels sad about it later?
In Shannon's case, given how fundamental his 1937 thesis was (you can use Boolean Algebra to design digital circuits), most other people might have been afraid of not being able to top that. My impression was that he was always looking forward to bigger things through his own work and through his students.
> In 1985 he made an unexpected appearance at the International Information Theory Symposium in Brighton, England. The meeting was proceeding smoothly, if uneventfully, when news raced through the halls and lecture rooms that the snowy-haired man with the shy grin who was wandering in and out of the sessions was none other than Claude Shannon. Some of those at the conference had not even known he was still alive.
> At the banquet, the meeting’s organizers somehow persuaded Shannon to address the audience. He spoke for a few minutes and then—fearing that he was boring his audience, he recalled later—pulled three balls out of his pockets and began juggling. The audience cheered and lined up for autographs.