Bush's Memex and Engelbart's mother of all demos hint at something much more powerful than HTML and the premature optimization that we got as a result.
In reading the pdf linked by Jtsummers, I recall how some of the sales people I supported had 20+ gigabytes of email in their inboxes... which drove me nuts (Exchange Server really didn't cope well with it back then), and now I know why, they were building a hyperlinked database (well, threaded) that they could search through.
Here in HN, we do the same thing over time, building a tree structured database that preserves context and is searchable. The main limitation here is there's only one possible tree, and it's the privileged view.
There's at least a billion dollars of knowledge captured here in HN. Yet, it takes Dang and a strong community to keep that one view rational, due to the corrupting forces of human nature and profit motives from spam and scams.
I think Google, GPT-3, co-pilot, simple tricks like a 'wordcloud' and even regurgitation/review communities like HN, reddit and good reads show that the presentation format of the information in a given system can be transformed to many diverse representations.
Books and movies have long made plots off new ways to help humans learn things and form connections by either 1) changing the presentation format or 2) downloading all the information into the human brain to let it do the sorting. For example, overlying data on a map in 3d using glasses or wiring a human brain into a computer's serial ports.
The main problem moving forward is making the data available. I have a fear that in the future only governments and large companies in bed with governments will have access to this data and use it in unethical ways.
Meanwhile, startups and individuals will be bared from creating 'unlicensed' presentations or accessing the underlying data itself (i.e. running a search engine)
>I have a fear that in the future only governments and large companies in bed with governments will have access to this data and use it in unethical ways.
I'd look back to the Memex for the solution to that problem, it stored everything locally, thus links couldn't be broken. Output included all the linked items. It's possible to build a system that stores everything you've seen on the web locally, to prevent things from disappearing.
I think local storage is highly undervalued, and the pendulum will shift back.
Yes. The opportunity to increase our functional representations of 2D knowledge bases is going to be a game changer that will compliment the Cambrian Explosion currently underway with the PKM/TfT use of symantec Graph connections & AI LLMs. Some efforts ongoing towards this [1] already and imho there is so much you can do to interact with data once you open up the presentation model to spatial computing inside a 3D embodied user interface like VR today, Passthrough AR tomorrow, and full AR a little down the line.
Autoencode it into the latent space of the collective conceptual understanding. (See OpenAI’s latest paper on concepts arising due to “multimodal neurons” [1]). Then use all of hacker news to “fine tune” GPT-x. That will preserve the knowledge in a highly flexible manner that can be queried at will to generate any representation you like.
> The main limitation here is there's only one possible tree, and it's the privileged view.
Mainly because inferring a different "view" over gigabytes or even terabytes of data (current collections of publicly available "general purpose" datasets can run in the terabytes) is really, really hard and computationally expensive.
And our current data models don't even include satisfactory ways of representing the kind of "common sense" data that would be reflected in a HN comment, an e-mail thread or really any kind of textual source. You can kinda get there by marking "named entities" of sorts but that's the tip of the iceberg, all the structure and semantics is still missing. It's something that's very much being worked on, and has been for decades (with "frame structures" and the like). But it's still hard.
There is more knowledge in your local library. But it requires discipline and time to process. Generating discipline and time at large scale is non trivial.
Despite ”design for knowledge based organizations” being the title of the article in the PDF, this is perhaps more about approach and capabilities for organizing flow from capture to use to pruning of knowledge, more than the organizations working on knowledge.
There is one aside about the organization itself:
> Engelbart sees every organization as a collection of interacting knowledge domains. He has focused his research on designing support structures for knowledge collection and refinementwithin and across these knowledge domains.
But then that jumps down into a given domain, and goes on to propose a knowledge management approach within that (and each) domain.
This is where it gets interestingly predictive of the recent coalescence in knowledge management tools emerging after the two dark age decades of style over semantics.
The first idea is a concept of bringing knowledge in, working it, and keeping it. He calls this Concurrent Development, Integration, and Application of Knowledge (CODIAK) process.
To make this workable he proposes a time-relevance layered approach very similar to the (perhaps easier to action) “PARA” method adopted by many KM tool users today:
After this, he goes into functional implications, and describes almost to a T capabilities in the latest round of knowledge management tooling “systems” such as, say, Obsidian.md, built on standards (markdown, wikilinks, front matter) and extensibility enabling journaling, querying, views, as first class citizens.
When hypertext markup iterated into a page style description advertorial tool rather than linked semantic knowledge structure, that left an opening for the tools we’re seeing now. Today, Obsidian.md is close to the mark, except, of course, for approachability by casuals. For casuals, consider e.g. Craft.do (https://www.craft.do/).
If you’re doing a comparison before jumping in, it’s worth diffing the tools against Englebart’s take.
Obsidian's been helpful, even game-changing for my working life. But my obsidian knowledge is siloed to my computer. Is there something similarly good for working with teams/organizations? Confluence and Notion seem to check some of the boxes, but neither tool feels as fluent to use as Obsidian or Roam.
I like craft.do for teams with casuals on them, it allows you to make team spaces. It’s markdown based and also has incredibly frictionless online sharing.
For technical teams, you can hook your obsidian markdown folders to a multi-party savvy file sync system, e.g. git, there are extensions for this. Then it plays nice with, for example, team members using VS Code as well. You can match the markdown, linking, tagging, etc., using VSCode plugins, or by carefully matching options between them, you can have some people in Obsidian and some people in VSCode + Foam.
Using obsidian+git is definitely possible, but it doesn't really get to the seamless collaborative feel that I'm looking for. Looking into Notion again, it seems like they've imported more ideas from Roam et al. and might be worth another look.
There are similarities but I'd disagree with this:
> Despite ”design for knowledge based organizations” being the title of the article in the PDF, this is perhaps more about approach and capabilities for organizing flow from capture to use to pruning of knowledge, more than the organizations working on knowledge.
I would agree that without the tools, an organization wouldn't be able to collaborate effectively in the way he advocates for, but I think unlocking group productivity should very much be more the goal here than single-person productivity. From the second page: "Englebert realized that they key to dealing with increasing complexity was human collaboration."
No amount of methodology will let a single person keep up with a team using a similar methodology.
Another key line from me: "Englebert is not advocating that we perform artificial acts with documents by superimposing structure on them. Instead, he advocates that we capture the inherent structure in all forms of human expression in order to make them easier for people to navigate through, view in different ways, and hyperlink."
As you note, there are a lot of similarities in there to Roam Research and those similar systems, especially the cross-linking/blocks type stuff vs just having a single hierarchical document structure. But don't just think of it for your own navigation and recollection, but for others to learn and get up to speed and then offer their perspectives.
But I think it's a big disservice to think about it only from a single-person's POV vs a way of capturing knowledge inside a group.
Think of "git" for software dev too, which is very along the lines of the logging/journaling of changes discussed in the article, but which differs from some previous iterations by being far more multi-user friendly (remember fighting over SVN locks?).
This presumably is the point where one should consider that long before Foam, there were things called "wikis". One of them tries to build a network of human knowledge. It's called Wikipedia, you might have heard of it ;)
Foam is simply the next evolution - the shared tool is well-understood, we're now getting an augmented local version. And after that, we'll get something that's far more multi-user friendly, too. It's a pendulum that seems to swing back and forth in our industry.
The thing that I hope Foam & Friends will generate is a much better understanding of knowledge "operations", so we can understand the fundamental structures underneath better. (That's what happened with git & signed DAGs as version log. That's what happened with text & CRDTs as a model of shared edits)
In terms of the commenter here talking about "tools emerging after the two dark age decades of style over semantics" wikis don't count as new development. ;) Even Wikipedia is more than 20 years old.
And I think wikis are still quite limited in that bit I quoted: "Englebert is not advocating that we perform artificial acts with documents by superimposing structure on them. Instead, he advocates that we capture the inherent structure in all forms of human expression in order to make them easier for people to navigate through, view in different ways, and hyperlink."
They have hyperlinks, sure, but otherwise most pages in wikis have only those views that can be created through operations on a hierarchical table of contents. The structure is imposed by the tech, there, rather than the expression of knowledge itself. The blocks in Roam - https://thinkstack.club/how-to-use-roam-block-reference-menu... - seems like a major iteration on that to me in terms of moving towards units of knowledge as the fundamental element instead of "pages." (It's also worth noting that Foam doesn't do this - https://github.com/foambubble/foam/issues/294 - though apparently Obsidian does.)
Absolutely. But the blocks, which are the one notable improvement, are tech that isn't well-used enough that we could make sense of it in a multiuser environment. That's the reason it's single-player right now. (And hence my comment about better understanding underlying structures)
And yes, it's closer to Engelbert's vision. We've been getting closer to that (and the Memex) for a long time, and we'll continue to do so. They are distant visions what a future could look like, while we barely understand knowledge work. I mean, the term itself is already an improvement of clarity on the ideas of the memex.
If you compare knowledge work to Mathematics, I'd guess we're shortly before Elements. We're at best about to understand the basic building blocks. Which I find tremendously exciting - we're still at the very beginning of exploration.
Maybe I'm too software-development-oriented but I feel like the blocks are tantalizingly close to being ready for multi-user. EDIT: To the point of frustration with how things like that are still silo'd away. Seems like so much work has been done around like "concurrent editing" (Notion seems like a big improvement over Confluence here, but still seemed to be the same page-based world just with a better editor) but so little around what documentation are we actually building for teams?
In my head they're akin methods, and so in software that idea of "call this from lots of different places, edit it in a centralized way" concept is pretty well established.
In terms of "a single document flowing with a single voice" you can't necessarily do that from assembling an arbitrary collection of blocks like that, but for "good enough for effective in-corp collaboriation"? Sure!
Maybe. The problem with multi-user blocks is the same as with methods in software - Hyrum's law. People will rely on them staying as-implemented vs as-spec'd.
I think that (and the problem of bait-and-switch) still need an answer before it's usable beyond small groups. Maybe small groups is good enough for now. If it goes in any form like software development, we'll at some point split into two camps. One will say YOLO and just implement shared blocks, trusting in good will, and the other group will noodle on foundational principles to get this right. (Group 1 will make the money, group 2 will have the fun. As always ;)
I was describing this as a system for all persons within a given knowledge domain at an organization, not individuals.
That’s what E’s describing as well.
My point is, he’s NOT describing how to communicate between domains, thus, it’s not really the whole organization, it’s the domains (teams) within an organization.
Engelbart was always dead on. But very few listened, and now, unfortunately he’s just dead. And we are stuck with unchangeable web pages laid out for graphic design purposes rather than understanding and information reuse. It will be another several decades before this broadly changes.
This seems to be a preview of an article available as PDF after you sign in. Without the main content the purpose isn't clear to me. Am I missing something?
I am probably overly optimistic that some combination of HTML, linked data, GPT-3 like models for effective summarization of very long texts, better user models (their level of expertise, things they have paid attention to in the past, etc.) will get us part way to Engelbart’s vision.
Our startup's building a browser based AI Tutor that estimates a learner's knowledge level and uses it to explain concepts in context. It's like step-by-step navigation for learning. We're still in the very early stages, so it'd be years before we get to enterprise customers.
Our website isn't active yet, but you can follow our LinkedIn if you're interested in updates. We have an MVP release planned for this month.
https://www.linkedin.com/company/conceptionary/
I think the issue with this is that as individuals we attach our identity to our dialog and put it out into the world to share it, express ourselves, to provide value to the world, and also to collaborate and iterate and learn ourselves… with other individuals. The problem with deconstructing human ideas and information like this is that it ignores temporality. What is true today may not be true tomorrow, what is relevant today may not be relevant tomorrow. For the preservation of scientific or historic information this works great, and even allows copywriting of works. However, the true order of the day is leading and making decisions which ends up taking place in dialogue.
Dialogue is the single-threaded brute forcing and tree traversing activity for making decisions. Within a single thread, multiple subthreads do branch off, but then they become decentralized which is not what we want. We want each subthread to be conscious of it’s uniqueness to all other threads globally and for any duplication to be catalogued or ideally merged and then hyperlinked and indexed within the grand scheme. This is difficult when the most granular we can get in terms of categorization is the factual information or simply the topic itself. Places like HN and Reddit do well in allowing you to provide an entire article or website as the context and allow that to be explored, but how would you categorize this information? The real question is how do you make this visible and accessible from a single standpoint: an individual trying to learn something or make a decision. I think search engines like Google have solved this problem well simply by using the heuristic method of merit (backlinks, page rank, keywords) and so to implement this on a miniature scale for an organization would be useful, but the problem is that it would require users within the organization to forgo going to Google and instead use the provided Intranet which initially would be less useful and again may never be as useful as Google. So the solution might be to use Google as an indexing service which currently is not possible.
I think at the end of the day we are ignoring is the incentives. If people were incentivized to construct the information in this way, get some sort of reward, then it would happen naturally. But I don’t even think it is possible for a human to do that, say, store all of HN dialogues globally for example in their head and consider THAT before writing anything otherwise hyperlinking to what is already there.
The main issue is that knowledge is a graph, not a tree, and dialogue is linear. So in order to consume knowledge through text as quickly and efficiently as possible is more of a graph or traveling salesman problem which is very hard to solve, maybe something that Quantum AI can help with. One thing would be to incentivize hyperlinking. Another might be to store information in images using something like a way more advanced DALL-E that allows the human mind to grasp concepts simply through looking at trippy images or videos. To somehow use spatial and visual understanding to up the number of dimensions of what knowledge we can store and download into our brains.
In reading the pdf linked by Jtsummers, I recall how some of the sales people I supported had 20+ gigabytes of email in their inboxes... which drove me nuts (Exchange Server really didn't cope well with it back then), and now I know why, they were building a hyperlinked database (well, threaded) that they could search through.
Here in HN, we do the same thing over time, building a tree structured database that preserves context and is searchable. The main limitation here is there's only one possible tree, and it's the privileged view.
There's at least a billion dollars of knowledge captured here in HN. Yet, it takes Dang and a strong community to keep that one view rational, due to the corrupting forces of human nature and profit motives from spam and scams.
There's got to be a better way.