This is a 1-dimensional cellular automaton. Apparently John Dee was fascinated by the chaotic patterns such automata are famous for. It reminds me of Stephen Wolfram's love of Rule 30.
Conway's Game of Life similarly intrigued hackers in the 20th century, but without the mystical subtext.
I can't help thinking that these books might have been keys for encrypting messages. Key-based substitution ciphers, such as the Vigenère cipher[1], were already known in the 1500s. If two people each had a copy of this book, they could send a key to each other by referring to a sequence of characters such as "the third row on page 25". The messenger who carried this message wouldn't be able to reconstruct the key without a copy of the book. If you wanted to give the code to somebody else, you wouldn't have to send them the book: just tell them the "seed" words and the algorithm for generating the pages, which could be sent separately, for security.
Using this as a keypad (modular addition) would be much better than the Vigenère cipher, since there aren't such obvious correlations between key characters-- a good cellular automata is closer to a proper PRF.
I haven't delved into the whole logics, but it seems from the wikipedias (http://en.wikipedia.org/wiki/Book_of_Soyga) that there's much more to be decoded from the tables. Given their likely magical or esoteric nature, the contents themselves of course are likely to be much less interesting to HN than their encoding... but a challenge remains.
Conway's Game of Life similarly intrigued hackers in the 20th century, but without the mystical subtext.