By the way, Ahmia publishes a blacklist with the hashed addresses for every onion URL that they have discovered to host abusive content. You could use that blacklist to filter out those sites so you don't even crawl them and also to periodically purge any matching URL that may have already made it into your index.
Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.
Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning.
I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses:
> Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning.
What about when the FBI/CIA does it? Genuine question.
But it is totally possible to host your own server with flawed security that you are able to legally hack. It might be rare for that to be the actual use case, but it is totally a possibility. With the illegal material being discussed here, there is no such equivalent no matter how out of the box a justification one is willing to aim for.
Probably okay to have the source code to the engine.
However, if you have used the the system which creates a database of questionable dark web links on your machine, that could be tricky to explain... and easy to implicate
Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example.
I am guessing you have some personal instance you use at least for testing/"education", right?
> On June 20th, board members of the „Zwiebelfreunde“ association in multiple German cities had their homes searched under the dubious pretence that they were „witnesses“ while their computers and storage media were confiscated.
That's something very different though. Exit nodes are providing a service and are, for all intents and purposes, the only visible client on the clearnet (and might not even be involved: there's nothing stopping you from running a private proxy on the same machine you run your exit node on). TOR-developers that do not run exit nodes but contribute to TOR typically don't get searched, at least to my knowledge.
Content that's illegal to possess is a different issue, though I'm sure they'd make for an interesting case because a crawler downloading, saving and parsing an HTML page isn't as clear cut as a human evaluating and deciding what to download and store. "The suspect has the hard- and software necessary to download this content" shouldn't be enough to convince a judge to issue a search warrant, but then again, judges probably have very little technical knowledge.
The Zwiebelfreunde raids were not because of their TOR (hi dewey) activities, but rather they collected donations for the riseup email service.
If the police can convince a judge to raid the board members and their families of registered club just because they, among many other things, collected some donations for an US org, then some overzealous police detective or DA going after some dev who made a webcrawler for the "dark web" and is probably in possession (knowingly or not) or illegal content isn't much of a stretch either.
They can, but I'm not so sure that "may or may not possess illegal content" is enough for a search warrant - it's true for most of the population after all (running an exit node or collecting funds on behalf of a third party on the other hand is true only for a tiny fraction of the population). Granted, the chances are somewhat higher for IT people and higher still for people that write crawlers, but "we think he might, it's not impossible that he doesn't" is a bit thin, and unless they're trying to go after you for unrelated reasons, DAs don't love to have their asses handed to them by judges.
In many jurisdictions around the world it is enough. In Germany you need a "begründeter Anfangsverdacht" (reasonable initial suspicion) and what's reasonable is essentially up to the judge signing the warrant.
Hell, they used to raid people accused by third parties of copyright infringement (for private personal use), about a decade back or one and a half, but thankfully that stopped now. They would come early in the morning, present you with a warrant that said "based on evidence provided by <third party>..." (i.e. somebody somehow collected an IP address you might have used off of some file sharing swarm), take all your shit and scare your neighbors and quite often your parents because they raided a lot of minors too.
I know two people who this happened to personally.
One guy wanted his stuff back, to which the DA replied that if they got to keep his stuff they would drop the case (I kid you not), and the other guy had his stuff returned about 2 years later, except for his HDDs. And his stuff not only included computers, CDs, DVDs, a printer, but they had actually seized books... paper ones... wat. Neither was convicted of any crimes in the end (IIRC they both had the thing dropped because "minor offense" not worth pursuing).
Turns out that a little googling of the German internet around that time turned up a lot of similar cases and some people claiming the police and DAs did that to get new computers "cheap"...
The OP wrote a crawler and used it to crawl Tor. Depending on where they live, accessing the content might be illegal, and storing some of the content in your computer might be illegal as well.
Law enforcement might be monitoring some domains, or have set up some honeypots that the OP might crawl automatically.
You don't want to end up in court having to argue about why your computed accessed some child pornography and downloaded it, and trying to explain to a jury that you did not did those things, but that the crawler that you programmed to do those things did.
Sure, nobody might end up raiding the OPs home, and even if they do, the OP might be able to successfully survive a jury. But just having to go through that might suck.
If the OP only wrote the software and never used it, then they are fine. But from the article, they did use it, so who knows where the crawler landed. Chances are nowhere good.
I didn't talk about the dev getting contacted about third parties abusing the software, but about the dev keeping a DB of indexed content for development/testing/"education" that would most likely include illegal-to-possess content.
And that some eager police people like to "inconvenience" people connected to TOR somehow isn't exactly new, either.
E.g. there have been multiple raids against TOR exit node operators in different countries around the world in the past, even when the police was fully aware it was a TOR exit node that did not store information.
Maybe I'm just too paranoid - then again I used to run a TOR exit node myself and had a bunch of less than pleasant run-ins with the police, tho thankfully no raids.
