The combined professional network of founders and initial investors can minimize impact and effects of many factors discussed in other comments here.
"If you need to make a single cold call in the first 2-3 years, consider your network might be insufficient."
Such cold calls would include finding: co-founders, investors, customers, service providers, manufacturers, employees, etc.
Mind you-- I got this wrong in my own startup that didn't survive. Having advised founders and non-tech entrepreneurs in Seattle, SF and Vancouver and of those who heeded the advice, they succeeded where I didn't. E.g., when I had a co-founder flake on me, my network was insufficient to have another person I could trust to step in as subject matter advisor. Not that I would have expected that advisor to write code but perhaps to guide me through the principles of the niche topic such that I could then find the right contractor or employee.
Likewise, for some businesses including certain SaaS: picking your first customers carefully often requires a chain of trust in both directions; i.e., your network.
Have you considered using Common Crawl [1], and if so, what was your assessment when compared to having your own spyders?
Long-term, a combination of theirs and your own could be optimal.
There are strengths and weaknesses with using their dumps: on one hand, benefits include them having crawled and having dealt with being throttled, etc. They offer monthly dumps for general content and daily dumps for news [2].
On the other hand, it's a huge pile of data to wade through, and their index format might not be your preferred method. The archive and index reside officially at AWS, so that may decide where to process it. (Not sure whether other providers maintain a copy as well or not.)
By "huge", specifically:
> October 2020 [...] contains 2.71 billion web pages or 280 TiB of uncompressed content.
From our analysis a few years ago, that was to be the approach for the now-defunct Snagz.net [3] (which never fully launched because co-founders were unable to join due to extenuating circumstances).
There's a list of Common Lisp resources on Xach's blog [1] that includes communities, which remains relevant despite being an old post.
Also, contacting one of the people associated with a blog listed on Planet Lisp [2] would ensure that it appears there and might be seen by a wide yet relevant audience.
There are several topics intertwined with solutions you seek: There's parts of speech (PoS) tagging, reducing to Lemma form, identifying end of sentence, etc.
After having faced a similar learning curve, I put what I know into a lengthy document[0] written in 2018 based upon explorations over 2016-17. That will get you deployed and operational quickly by following just the final section. The first section explains key concepts using conventional ideas as the means of introducing NLP jargon. In between covers theory and practice for getting the most out of any tool you're likely to use in the end.
More general tools are probably available today, such as add-ons for Elasticsearch. I'd start looking there. Interesting items came up when searching ddg for: NLP elasticsearch.
Off-topic with respect to Kotlin but relevant to the Java platform:
From your perspective, what are Clojure's prospects for contemporary and future alignment with the JVM?
After all, Rich Hickey built the core in 2006-07. Looking at his graph[0] of code introduction and retention in his HOPL-IV paper[1] might imply that either he's that good at the core abstractions or possibly the language could suffer a similar set of limitations as commented about Kotlin above.
I think Clojure has a very different design philosophy and very different goals. Specifically, its abstractions do add overhead -- be it due to immutability or lack of typing -- and that's perfectly in line with its goals. As such, it flies higher above the Java platform than the Java language or Kotlin, and is less sensitive to changes in the underlying platform, as it doesn't try to always closely reflect its capabilities.
For others considering Rust + Kotlin for native Android development, there's also the Visly blog post[1] with basic steps. However, since then, Google officially released `androidx`. Unfortunately, Android Studio or the SDK lack `cargo fix` features of modifying your source code and instead just magically compiles with a compatibility layer. There's a PR on Visly's GitHub repo with these code fixes.
Also, their code base is minimal: does just enough with Rust shared library to have an app generate a log message. That's enough if you already know Android development.
For a complete end-to-end example search query screen with results screen (Activity, Fragment) using Rust, Kotlin, androidx, RecyclerView, ViewModel, etc., there's [2], but it was written while learning all things Android.
Edit: Yes, that second one is mine. Had fun learning but hopefully will never have to touch mobile dev again beyond maybe a PWA. (I'm a server-side dev.)
I think the build system side is still a bit rough around the edges. It's not hard to get something working, as your examples show, but I haven't seen a solution that feels really well-integrated yet. I'm hoping that tools like bazel/buck/pants continue to get more useful and can provide that kind of experience.
As someone just getting into Nix and NixOS myself, I found one particular comment there offers an excellent summary after talking with a few others in person as well:
https://news.ycombinator.com/item?id=22891648
Essentially, after an initial investment of time for learning curve, the huge payoff comes when migrating to another machine because of system-wide configuration being reproduced with precision.
In my case, moving back and forth between laptop and desktop/workstation was a pain point with Xubuntu. A delta of less than one hour between `apt-get dist-upgrade` on two devices was enough to translate into divergence of versions in some packages.
We've come a long way since days of tsx-11, but I digress.
I love the idea of Nix, but it is pragmatically unusable in practice. I installed it on my work laptop running OS X, and doing anything with it would often lead to searching through obscure gitlab or forum posts. The core commands are unintuitive. Try to get ‘cargo install’ to work for common Rust tools dragged on for weeks of on and off incidental work on the side. When something pulled in the Network Foundation SDK, I couldn’t figure out how to specify it with nix-shell. After multiple attempts asking in chat, nor, did it seem, anyone else knew either. Eventually I found the package I was trying to compile in nix and just gave up on getting packages with HTTPS dependencies to work with ‘cargo install’.
When it got to the point of reinstalling docker, I just gave up and used brew.
My personal box runs Arch Linux, which uses pacman. It’s fast, and the commands are based on combining a common set of letters.
To me, nix is the epitome of the problem with OSS. A brilliant idea crippled by abysmal UX because enough emphasis isn’t put on what’s necessary for people to use it if it isn’t the center of their life. If I were working on nix daily and focusing on it constantly, I’m sure I’d feel the commands are obvious because I’d incidentally memorize them all. But when using it as just one of many critical tools, it simply demands too much attention for someone to be productive.
That isn’t to say it’s worthless. It’s still an impressive achievement and I’m glad someone did it. What will hopefully happen is that someone will write a new ’nix’ frontend tool to make it simple and intuitive.
There are tools that require a big time investment to learn, and Nix is one of them. It much more "pragmatically usable" than other OSes once you learn all the bits and pieces. It's kind of like Vim. But yes, it's quite frustrating at start that you can't do seemingly simple stuff with ease.
Résumé/CV: Linkedin.com/in/dpezely or Pezely.com/daniel
Email: first name at last name dot com
Currently specializing in Rust:
- Over 2.5 years as primary language, and began with 1.0-stable in mid-2015
- Rust mixed with Kotlin for native Android app, where Kotlin handles just UI via RecyclerView, ViewModel, etc. (otherwise, I'm relatively new to mobile dev; Rust lib originally deployed in web app, and the plan is to use the same Rust library with Swift for iOS)
Generalist from prior specializations in concurrency, exotic data storage, dev-ops, Natural Language Processing
Samples:
- GitHub.com/dpezely/anagram-phrases (using prime number factorization, in Rust)
- GitLab.com/dpezely/chat-server (mio as the only networking library, in Rust)
Daniel Pezely — Authorized to work for any employer in US or Canada
Location: Vancouver
Remote: Negotiable, experienced
Willing to relocate: Negotiable
Technologies: Rust, Python, Cython, Common Lisp, Erlang, C; AWS, GCP; Debian-based Linux, FreeBSD, macOS/MacOSX; old school pure JavaScript in browser to exercise server-side API for delivering end-to-end functionality; learning Kotlin+Rust for Android Native
Résumé/CV: Linkedin.com/in/dpezely or Pezely.com/daniel
Email: first name at last name dot com
Currently specializing in Rust:
- Over 2.5 years as primary language, and began with 1.0-stable in mid-2015
- Rust mixed with Kotlin for going full Android Native, where Kotlin handles just UI (otherwise, I'm new to mobile dev)
Generalist from prior specializations in concurrency, scaling, exotic data storage, dev-ops, and long ago as a Unix sys-admin.
e.g., at Splunk/BugSense: rewrote the BugSense back-end from Erlang/Lisp/C to pure Erlang for ironically 25x performance increase. Server cluster saw several billion inbound requests per day non-stop from all time zones. (See GitHub for slides from Erlang Factory and other presentations.)
Samples:
- GitHub.com/dpezely/anagram-phrases (using prime number factorization, in Rust)
- GitLab.com/dpezely/chat-server (mio as only networking library, in Rust)
"If you need to make a single cold call in the first 2-3 years, consider your network might be insufficient."
Such cold calls would include finding: co-founders, investors, customers, service providers, manufacturers, employees, etc.
Mind you-- I got this wrong in my own startup that didn't survive. Having advised founders and non-tech entrepreneurs in Seattle, SF and Vancouver and of those who heeded the advice, they succeeded where I didn't. E.g., when I had a co-founder flake on me, my network was insufficient to have another person I could trust to step in as subject matter advisor. Not that I would have expected that advisor to write code but perhaps to guide me through the principles of the niche topic such that I could then find the right contractor or employee.
Likewise, for some businesses including certain SaaS: picking your first customers carefully often requires a chain of trust in both directions; i.e., your network.