Many projects really miss some section on their homepage which briefly compares itself to similar or related projects. I assume there are many related projects. Most big desktop environments (commercial or open source) provide some similar service.
I wonder about e.g. Baloo (Baloo is the file indexing and file search framework for KDE Plasma, https://community.kde.org/Baloo). Is this similar? Or different? How is it different?
Just yesterday my dad complained about a couple of Baloo error messages, Baloo crash reports, disk full, Xsession log full of Baloo messages. And he was not really aware about what Baloo actually is.
So I disabled the content indexing, as it was suggested there. The stack trace of the Baloo crashes also suggested that it was related to indexing the content of some documents.
I think this is a problem for many such file indexing services. Even Spotlight on MacOSX frequently crashes when it tries to index some documents. Does Recoll handle this differently?
If Recoll is in general better than Baloo, maybe it should replace Baloo? Or Baloo can be extended to use Recoll under the hood?
Baloo Author here, though I haven't contributed code for over 4 years now.
Baloo used to use Xapian under the hood, like Recoll, when initially launched, but in 2014 (I think?) I moved away from it to a custom DB built on top of LMDB. It resulted in a massive performance increase, one of the main criteria was for results to always take less than 'x milliseconds' (I can't remember exactly how much) so that KRunner (similar to OSX Spotlight) would always feel snappy. Another criteria was for Baloo to operate well with minimal memory and CPU, which wasn't the case with Recoll (Things might have changed since then)
I do remember ensuring that Baloo always did content indexing in another process, though I can't remember at what point it blacklists a file.
Sorry, this isn't the best answer to your question, but most of the details are now quite fuzzy to me, and I haven't kept up with how the project has changed after I stopped contributing to KDE.
In retrospect though, I definitely should have spent far far more time ensuring that Baloo doesn't end up in situations like the one you described, and putting far more limits on the IO and CPU usage it's allowed to use.
I've been running a setup with Recoll and https://github.com/ArchiveBox/ArchiveBox for a few months now [1]. Each morning archivebox scrapes all new links that I've put into my (text-based) notes and saves them as HTML singlefiles. Then Recoll indexes them.
It's very fast and ~4 lines of code. It's surprising how often I rediscover old blog posts & papers that are much better than what Google yields me.
From my experience Recoll isn't very good at searching for aliases sadly.
Tangential but I really love the look of your site, especially the footnotes displayed adjacent to the text. Did you create the layout yourself? I have been looking for a theme like this for a paper discussion website.
I would like to have a personal search engine for the web. It should only index my content on the web (social media, blog post, comments on hacker news, etc...)
If you have periodic screen caps saved to a folder then it can, however it makes more sense to make your own plugin to Xapian which you can do and share with Recoll.
This uses Xapian. Some of the Linux mailing lists, like linux-netdev, use Xapian.
I use Xapian on the desktop but I only store text documents so I have no need for Recoll. It is refreshing to use search that is not tuned for "popularity" and selling online ad services. If search is too easy, then IMO it is not really search. Search should require some skill. It should not be a game where a commercial entity Hoovering up user data and selling ad services runs the search and tries to guess what the searcher is searching for.
I never tried it, but this looks somewhat similar to Recoll:
As a consultant, I am routinely given hundreds of new files from new clients which are usually a mix of Word documents and PDFs of policies, procedures, and standards. Then, throughout the course of the engagement, I often have to pick out certain phrases or topics from these large datasets in order to reference them in a report.
Windows File Explorer can do this to some degree. I am not sure if File Explorer looks at the actual content of the documents or just titles. Either way, Recoll is magnitudes better and is absolutely worth it for my case.
Recoll is not just a desktop search engine, it can also be used as an intranet search engine either 'stand-alone' [1] or as an 'engine' in Searx [2]. I made the Searx engine a few years ago to extend Searx to intranet search, the QT qui is not used (an, on Debian, not even installed - only the 'recollcmd' package is needed).
Is there any desktop search engine that can do topic modelling and/or semantic clustering to allow you to discover related documents or do other kinds of browsing and discovery of large sets of documents where you may not immediately know the specific search terms to directly go to what you need?
Recoll and the underlying search Xapian are incredibly powerful. I am surprised that almost all server side search use such bloated frameworks. Xapian seemed really easy to theme but their site and docs need some love. I also don't recall, but if they had a way to parse JSON search hints I'm sure that it would be perfect as a lightweight server side search for static sites. Not that JS search isn't cool, but until browsers implement a native site searcb feature, it is nice to have a way to search from non-JS browsers and clients.
Wow, you also solved the JSON issue. I've been wanting to implement Xapian as a server-side search server for Hugo. I believe by just creating the template using Hugo, it should be pretty easy to have it search the JSON. I've got quite a bit of work to do though to figure it out.
For Windows users, a shout out for Copernic Desktop Search (content is indexed so search results are instant and presented via an appropriate embedded view for some types; you can filter but it doesn't support grep), which I've used for many years.
It's not free, but one of the utils I wouldn't be without.
recoll is a killer app. it should somehow integrate deeper with linux desktops and take personal information management to the next level.
it is incredibly useful in itself, has a CLI/scripting interface etc, but its homemade GUI is quirky and using its output for next steps in workflows can be cumbersome (may need coding). better integration with file managers and other desktop apps would help it really takeoff
Yes, it can, I'm using it personally and can confirm that the PDF indexing works without you having to install anything else yourself, it seems like every binary needed for it to run is bundled together with the main app.
Spotlight does indeed do this so Mac users don't need this. Might help Windows and Linux users though. I use voidtools.com's Everything tool to search for filenames quickly on Windows as it uses the NTFS index to do near-instant searches, since the Windows Indexing Service has been poor since XP...
This might help if you are looking for content instead of just filenames.
I wonder about e.g. Baloo (Baloo is the file indexing and file search framework for KDE Plasma, https://community.kde.org/Baloo). Is this similar? Or different? How is it different?
Just yesterday my dad complained about a couple of Baloo error messages, Baloo crash reports, disk full, Xsession log full of Baloo messages. And he was not really aware about what Baloo actually is.
The problem is similar to the one here: https://askubuntu.com/questions/1214572/how-do-i-stop-and-re...
So I disabled the content indexing, as it was suggested there. The stack trace of the Baloo crashes also suggested that it was related to indexing the content of some documents.
I think this is a problem for many such file indexing services. Even Spotlight on MacOSX frequently crashes when it tries to index some documents. Does Recoll handle this differently?
If Recoll is in general better than Baloo, maybe it should replace Baloo? Or Baloo can be extended to use Recoll under the hood?