This assumes that all information is stored physically.
Let's say you get a telegram. The telegram itself can contain maybe a couple of paragraphs of text. You might think the bandwidth of the channel is at most a kilobyte or so. But there's plenty of other aspects that can raise the amount of information conveyed. Say you get one saying "Short Dow." Just going by the content, you'd have no idea what it was saying. Is there a guy named Dow somewhere that's short? But if you're a stockbroker, all of a sudden there's a whole lot more info there. When you allow for context, information density can approach infinity.
'If you can store infinite information outside of a system, then you can achieve infinite information density inside the system if you use outside the system as 'context'. I hope that paraphasing makes it clear enough the problem with this line of thinking. You have to include all the information when calculating absolute density. In your case, the context has to be stored somewhere too.
It's worse than that. There are a finite number of states of a message, and you can only reference that number of states of the outside system. If you think of the message as a pointer containing n bits, then you can only reference the first 2^n positions in the dictionary.
If the outside system is a probability distribution over output messages (a more general case of the dictionary you described) then the problem is synonymous with compression.
If all information has to be accounted for and stored somewhere, and context is part of the information, then you can't store any information without storing all information, everywhere. Because every bit of information exists inside the context of the entire universe.
You're getting very metaphysical here, but reality remains the same even if you expand these principles to the universe. The rules of physics still apply.
I think he has a point, although it's more about the semantics of the term 'information' density.
Shannon information is always measured relative to a receiving context in which it the symbols are understood, and information content is related to the inverse of the probability of observing a particular signal as assessed by the receiver. So from that perspective vinceguidry has a reasonable point.
However really the question being asked is something more like 'data density' and that is generally what people are talking about when the term 'information density' is invoked.
Edit: I see that the original article does indeed refer to Data Density, and that HN title is just wrong.
You are Dr. Who. You have a Tardis. It will convert your language in to any other language telepathically. Why not create a word that means the summation of everything you know, and say it to a person. They create a word that means the summation of everything they know and everything you know and say it to the next person...
Even better - attribute a meaning, in that language, to refusing to say anything to the next person (remaining silent). Then you can put the summation of everything they know and everything you know into the case where you don't say anything (in that language).
This is similar to making a version of gcc that outputs a Tetris program everytime it's asked to compile a 0-byte file, or maybe outputs all of Wikipedia. I mean, sure, you can hide Tetris, or the whole of human knowledge, in 0 bytes this way. But it's not a very useful exercise.
Remaining silent is as good as communicating one bit of information.
Even a 0-byte file has metadata. Even if it is 0-byte long, GCC knows that it is reading an input file. How did it come to know? Because you communicated some amount of information by initiating the compilation.
> When you allow for context, information density can approach infinity.
I absolutely agree with you. You are my hero.
The amount of information stored in an object (which is capable of storing at least one bit of information in traditional sense) depends on the size of the context.
If the object is not even capable of storing one bit of information in traditional sense, then the amount of information that can be stored is zero.
And for all objects that can store one bit or more of information in traditional sense, the total amount of information that can be stored in it = the number of bits it can store + the number of bits that can be stored in rest of the universe (context). So any one bit object can store the same amount of information that can be stored in the entire universe.
And if it turns out that our universe is enclosed in yet another larger universe, then you have to include that as part of the context as well.
If you were a stockbroker, that message would contain less information - and you probably would know it already. It's useful information to a stockbroker, but it would be more informative to almost anyone else, who doesn't encounter such messages daily. As another example, the information in the Rosetta stone was diminished - if more accessible - to its carvers, and a 700 MB disk holds 700 MB regardless of what those bits are.
What you're referring to is compression. High information density is indistinguishable from noise to any recipient who cannot translate it, which is why compression contests regularly require the extractor to be included in the message.
Let's say you get a telegram. The telegram itself can contain maybe a couple of paragraphs of text. You might think the bandwidth of the channel is at most a kilobyte or so. But there's plenty of other aspects that can raise the amount of information conveyed. Say you get one saying "Short Dow." Just going by the content, you'd have no idea what it was saying. Is there a guy named Dow somewhere that's short? But if you're a stockbroker, all of a sudden there's a whole lot more info there. When you allow for context, information density can approach infinity.