For those of us - myself included - who run a hosts file list (either using dnsmasq like Pi-hole or directly), here are the sources that Pi-hole use so you can add to your own solution:
There's a few on there I don't use and will look to implement. There's also a few they seem to have missed (perhaps intentionally?) so below I have included the lists I use in case it's useful for anyone else:
uBlock actually has a very good list of blockfiles, most of which (from memory) correspond to those you list. Last I checked it was on the order of 60k hosts.
I've used that to populate my own hosts + dnsmasq blockfiles. Using just the winhelp2002 list does a passably good job on a DD-WRT imaged router (~13k entries).
It's just a shell script to manage dnsmasq running on my FreeBSD home server. It's not sophicated but equally it's written specifically for my server so no very portable either.
I did think about writing something to share but projects like Pi-hole have done a better job serving the community than i could have. So i just share the sources i use instead incase any like-minded sysadmins find it useful.
Thanks for the recommendation but the shell script i have works good enough and has been for a few years now. Plus the container (FreeBSD jail to be precise) is pretty low footprint so while i don't have an issue with Python itself, it's an additional package I don't really need.
ublock only acts on the browser(s) to which it's been added. PiHole, other DNS blockers, or firewall-type (CIDR-block) blockers defend all applications from either DNS-supported services (most of them), or direct network access (most of the rest). To bypass the second, you'd need apps which had, say, VNC or proxy access to somewhere, and I strongly suspect those proxies would be fairly trivial to add.
I've been running various blocking and reporting systems for coming on 20 years myself, and find that applying my own hygiene controls to network traffic is ... surprisingly effective. Not bulletproof by any means (though reasonably effective against "bulletproof" hosting providers), but it massively reduces attack service.
I've been thinking a lot about networks, size and scale, and the corresponding levels of abuse. To a rough approximation, the Internet grew by an order of magnitude from 1969 to 1970 (1 to 10 nodes), 1977, 1985, 1987, 1989, 1993, 1995, 2000, and 2012 (1 billion nodes).
The "Linux Sucks" guy (multiple presos at Linux conferences on the state of breakage within Linux) did a preso recently on the IoT, noting that we're going to be looking at roughly 80 billion nodes by 2020. I'm wondering what that will do to various forms of abuse.
In the 1970s there was Phreaking and John "Captain Crunch" Draper. In the 1980s, the first boot-sector viruses (Brain: 1986) and the Morris Worm (1988). War dialing was a thing.
Through the 1990s, there was Usenet spam, first jokes like Make Money Fast, then Green Card. With the spread of the WWW, the first banner ads, pop-ups, and click fraud.
2000s: adware and spyware (bug-for-bug compatible reimplimentation on Android now), ID theft, high-profile viruses (Nimda, Code Red, Welchia, Slammer...), DDoS, and Phishing.
In the 2010s, click fraud and various forms of doxxing and pranking lead to wholsesale attacks on liberal democracy itself, with A/B tested bots and informational attacks.
I'm looking for a concept of network scale and problems encountered out of this.
It would be nice for mobile browsers to support a blacklist/reporting of malicious ads. In particular I'm referring to the type that hijack the browser through redirects, preventing you from returning to the source page. I see these alot originating from major news sites. They're usually in the form of "Complete this 3 question Facebook survey to win an iPhone" or "Your device is infected!" The people originating these ought to be arrested for wire fraud.
and a community driven list of good advertisers that are transparent in how they deal with our information and take the quality of the adverts seriously.
I definitely hate ads like anyone, but it should be acknowledged that ads serve a purpose too, of creating an economy for free content and giving people an avenue to get some income in exchange for their efforts, and possibly be able to devote their full attention to a free project and not have to support themselves with other income. Ads give creators some incentive to create stuff - although the income generally isn't great unless your audience is massive.
That said, the ad networks out there seem pretty awful in terms of privacy, slow-running javascript code mess, huge images or videos adding megabytes to the size of what should be a simple text page, etc. So if it's not doing it already perhaps this project can filter and put some pressure on the ad networks to clean up their mess a bit and not harm the user experience so much, and if an ad network is playing nicely, allow it through as a way to support free projects and their creators.
I've been on the Internet since before advertising became common, and I am not convinced that advertising is the only, or even a good, model for funding either content or infrastructure.
The move to advertising as the basis for the Internet has hidden, at least from most users, the cost of running a website and putting content online, which is frankly quite low compared to what it used to be, both in terms of actual money and the time/effort/experience required.
While in the mid/late 90s running a website could be a financially risky proposition (there are all sorts of semi-funny horror stories about people who ran labor of love sites that got Slashdotted and ended up with hundreds of dollars in bandwidth bills), today you can stand up a VPS on a Gigabit connection that's capable of serving orders of magnitude more users than most sites will ever see for a few bucks a month. I don't think that is necessarily a bad entry point for a communications platform that lets you talk to a significant percentage of the global population in one shot.
I'm not convinced that having the dominant model be "labor of love" as opposed to "make $$ off advertising" is necessarily worse or would result in less quality content. The advertising-driven model has given us clickbait and even machine-written garbage articles, link farms, etc., none of which would really exist under a publisher-pays model. There are significant hidden costs to the ad-based model, which are externalized on all players, including hard-to-define costs of privacy in terms of the way major players are incentivized to build detailed dossiers on users in order to improve ad targeting.
So if you are not convinced the advert driven model is not "the only, or even a good, model" what is then an alternative.
I think the ad model needs to improve where the major players will start enforcing strict rules on what types of adverts are allowed for example, they could reject any advert that waste CPU cycles, reject ones using too much memory, ban buggy adverts.
Plus, the number one concern these days, the privacy concern, I am forced to use adblockers everywhere on the Internet except for a few sites because I do not want some company giving me product recommendations just because I had a conversation with a friend about something related.
Subscriptions. My company only provides services via subscription, which means we can provide great support, continuous development, and no ads. Our customers love this.
You could argue that this doesn't solve your problem, because we have to identify you to tell if you have paid or not. But actually paying for a service radically alters the relationship - it's suddenly in our interests to keep your information private, because if we don't you'll stop paying us and move to competitors who can provide better privacy. So we're now on the same side on this issue.
The old adage that if it's free, YOU are the product is now understood by more and more people.
Subscriptions are nice, but not for everything, if the entire Internet was a subscription model, that would suck so much you would have to spend atleast 100usd a month to just keep using websites. Even if we have a model where you can subscribe to an array of sites using one service, we will still have the same problem of a few companies being in charge of controlling the information we have access to. Hell it would make tracking even easier.
Honestly I don't think we should even think about getting rid of adverts, but I definitely think that these adverts need to focus on being non intrusive, they have to start respecting people's privacy. I would willingly support a product which helps keep something I love online. But I would definitely think twice if that said company tries to get through to me by tracking and observing what I do online.
This is great for a lot of things, but I find I suffer from "subscription fatigue" lately where I'm reluctant to sign up for anything new because the amount of subscriptions slowly gets out of control. It might work for netflix and other giants but I'm not sure if it's viable for smaller players.
I think there's a business opportunity for someone to build a subscription management service that acts as an intermediary between users and those smaller players. Make it easy for users to subscribe / unsubscribe, track free trial periods, provide customer service, etc.
I've maintained for a long time that the ad networks themselves are the problem, not advertising as a whole. There is a place for video advertising (around content that is also video), there is a place for audio advertising (on the radio, around podcasts, on music services), and there is even a place for animated, interactive advertisements, on sites that are expected to dance and move anyway, like online games.
The trouble is, most third party networks are so laser focused on getting people to "interact" with their advertising that they've skewed the game in favor of distracting users from the content they arrived to consume. They're so focused on targeted advertising that they regularly invade users privacy, utilize questionable data collection practices, and break down security barriers when browsers try to shut down their violations of user trust. It's no wonder ad blockers are on the rise, because the existing ad networks are all untrustworthy.
There will always be a place for advertising when it's done properly. Watch an NFL broadcast and observe the product placement, the logos and sponsors everywhere. It's organic, integrated into the very fiber of the broadcast in a way that couldn't be blocked with even the most sophisticated blocking software. And it's also usually quite tasteful, there to promote the product, but not distracting enough to detract from the game the viewers came to the channel to watch.
This kind of advertising, organic product placement and sponsorships, where the content creator and the advertiser have a real partnership and coordinate their efforts, this is the kind of advertising I want to see more of. For all the drama they tend to draw, I commend the Gawker sites (Gizmodo is particularly good about this) for their Sponsored posts and their frequent "Deal of the Day" posts, which are first-party advertising that my blocker regularly fails to block. And you know what? Some of the deals are genuinely interesting to me, because Gizmodo clearly knows their audience, and selects advertising partners that make sense on their blogs. More of that please!
If the ad floats over the content. If the ad overlays my screen. If the ad forces me to do something to dismiss it, I don't care what the product is. I'm going to be less inclined to buy.
Put a normal, rectangular ad on the screen that doesn't dynamically resize itself while I'm trying to read the content and don't spread the content across 31 different pages just to increase the number of ads you show me.
> ad networks themselves are the problem, not advertising as a whole
I would perhaps say that it's the way that ad networks interact with content sites that's the problem.
With the current "automated just-in-time auction of the eyeballs of the person-we-identified-using-the-site's-metrics, with the site itself just providing the rectangle of space to slot the result into" model, content providers are essentially entirely beholden to whatever the ad network thinks is the best thing to put on their page.
An ad network could instead be a sort of "marketplace" service that lets content providers browse ads from various sources (or get ads suggested to them, using the same algorithm they'd have used originally to force ads on users), and then approve for display the ones they find tasteful/in line with their brand.
That is, after all, the model for running ads in any other medium: the publisher gets to provide ultimate editorial judgement on whether a given ad belongs in their publication.
(Also, in such a model, the content provider would likely be the one hosting the resulting ads, so we'd be able to avoid the whole ads = tracking beacons problem we face today.)
As a publisher, I've made the decision to sell advertising in monthly blocks like a magazine would as opposed to an eyeball-based auction system. Anecdotally, I think this deters me from trying to maximize how much money can be made off of every eyeball that visits my website.
I also sell and self-host my own advertising which can only be non-animated jpgs/pngs which gives me ultimate judgement. There is a barrier to entry on this style of ad sales, but overall I feel like it has been worthwhile pursuit so far in the 3 or so years I've been doing it.
It can be done relatively simply when you make direct deals with ad agencies or cross-promotion deals with other product companies. The problem is that no ad network wants to be a part of it; and for the little guy, ad networks are the only thing with economies-of-scale large enough to be interested in purchasing your nearly-worthless ad-space.
Thus my view: we need a "catalogue of creative, you pick what you run"-style ad network, so that the little guys have somewhere to turn instead of acquiescing to the existing networks and ending off with their sites showing chumboxes[1].
> Watch an NFL broadcast and observe the product placement, the logos and sponsors everywhere. It's organic, integrated into the very fiber of the broadcast in a way that couldn't be blocked with even the most sophisticated blocking software.
As an aside, I've seen multiple cases of live video-processing to replace or overlay advertisements on sporting events, such as those plastered on the walls of arenas. (Generally done during broadcasts of those events, as part of deals that separate the market for broadcast and in-person advertising at those events.) So, don't underestimate the ability to block advertisements given sufficient motivation.
I wonder how long before someone writes an ad-blocker that can remove or replace those video-processed overlays. I mean, if they can put them there in the first place, it ought to be easy to remove them.
I've thought about that many times when watching old sports videos, especially some motor racing and football which used to have very prominent cigarette advertising - thankfully illegal now here in Australia.
Often the ads are blurred which I actually don't like as it kind of ruins the feel of the show. I'm not a fan of such retroactive censorship, it feels historically dishonest. What is a film if not a recording of the zeitgeist? And within that zeitgeist, cigarette advertising was acceptable. I'm not sure if it's actually required legally as I could have sworn that I've seen some without the blurring.
Customisable TV ads is not an idea I'd be prepared to release onto the world. If there's one thing that humanity doesn't need, it's more advertising.
There is a much bigger barrier to entry when it comes to monetizing via product placement and sponsorships. For smaller entities, developing dedicated advertising partnerships is just not on the table.
So if the the ad networks themselves truly are the problem (and as a result adblocking is ever on the rise), it sounds like there is an opportunity for a better, more "organic" solution accessible to the long tail of smaller sites looking for monetization options. Sounds like a tricky nut to crack, as I'm sure anyone who works in the industry would tell me.
Adblock Plus has an "Acceptable Ads" program that allows non-animated, non-sound ads to remain on the page. People who think supporting the sites they visit is the right thing to do, can enable acceptable ads while blocking annoying ones.
I agree with this. One of the places I don't mind advertising at all is podcasts. Typically it's a short 10-20 seconds, integrated into the content, products relevant to the audience and none of the tracking or security risks that come with traditional online advertising.
Some great examples of this are the TWiT network and the Co-Optional podcast.
I agreed with you right up to the point where they decided it was acceptable to periodically jank the text I'm trying to read out from under my eyes.
I switched right from "well, ads support the internet so" to "kill this s### with whatever fire I can enable my browser with".
Web advertisers, we gave you a chance and the rich benefit of the doubt. You completely and totally earned every bit of this thats coming down on you now.
I agree, ads serve a useful purpose, but their implementation is bad. I'd like a compromise where we have ads, but they're less bad.
The trouble is that incentives are misaligned so we can't get to a middle ground position.
- Content owners and ad networks have little incentive to make ads that are better for users. They just want to get an ad system going that's simple to implement and allows tracking and fraud detection.
- Users who dislike the privacy, security, resource use issues with ads find it easier to just block ads completely. It would be much harder to work out a solution involving proxying, sandboxing, etc.
These conflicting incentives mean ad networks produce something good for them but bad for users and users (who know what to do) will do something that's good for them but bad for ad networks. There's a middle position, but neither party has a reason to attempt it.
Whatever fancies publishers/distributors - as long as end-users are in charge of what their computer's doing and have the final saying.
If end-user can instruct their computer "do this and don't do that" and freely share those scenarios with other end-users - I guess, everything's fine... or it isn't?
(Of course there's a never-ending-until-the-singularity arms race between ever-complicating advertisement delivery code vs ever-improving advertisement blocking rules... but that's just bound to happen, so personally I'm just looking forward to whatever would speed things up to their logical conclusion, whatever it will be.)
Generally almost everything also has positive consequences, so you have to take both the positive and the negative sides into account.
Ads are not the only way to support free content, and in most cases free content is paid for with advertisement (at least not amounts you could see as a payment).
On the other hand ads lead to consumerism, overconsumption, poor spending habits, waste of resources for the adverts and the products they sell, and they reduce our attention to things that actually matter (like the traffic while we're driving cars).
Then they should start to respect the user in every way. For me, it is not about ads being shown (as long as no pink flashy flashy banner popup hell). It is about privacy and snooping around, creating profiles, without me knowing, selling my data behind my back. I want to control, what my device sends and to whom. That is it.
I think, privacy, spam and ads-in-general are related but distinct concepts that don't necessarily involve each other (although more likely to be so than not, when it comes to real-life situations).
Ads are about what's being sent from publisher to consumer. It can be solicited, unsolicited or something in-between. And privacy is about what's being sent in the very opposite direction.
I think a lot of publishers seem uninterested in untargeted advertisements though, so the distinction -- while existing in principal -- is not very relevant in practice.
Certainly the untargeted adds I see (or even those I can tell are by virtue of the place I am, instead of my own customer profile, seem much lower quality.
With all the ad blocking technologies that are coming up, I wonder if Google is devising something to counteract these efforts.
For instance, since browser-based ad blockers work from what I know by blocking known domain names, couldn't Google create random subdomains and serve the code from a different subdomain every day or even every few hours, as well as change the way their JavaScript and HTML looks?
Even something very expensive to run would be justified with all the money that advertising brings in.
If Google can create software that can tell what's in a picture, or if a person in a picture is happy or not, why can't they find a way to fool ad blockers..?
- Google Chrome for mobile doesn't allow add-ons so you can't install ad-blockers. (You can install browser extensions with Firefox for Android.)
- Google Chrome uses a dark pattern where the address bar tends to send users to the Google search results page instead of to their final destination (compare the behavior with Firefox's). That means that even if you have an ad-blocker, many users are likely to click on Google ads on the way to the destination site, even if they are blocked on the destination site.
- Android doesn't provide fine-grained permissions control or root access, so users can't block ads.
- Some of their content is designed to coerce users to buy restricted Android-based content-consumption devices. For example, you can't buy movies on YouTube and watch them HD in Google Chrome (at least on my computer). You have to buy another computer that doesn't have root access (an Android device) in order to consume the videos in HD. Once you're on the restricted device, it's harder to block ads.
- Google introduces projects like AMP that try to convince webmasters to restrict their monetization options and make it easier to appify the WWW. AMP even serves your content from Google's servers. The more control of the content they have from server-to-eyeball, the more options they have for stopping ad-blockers (and the worse it is for open technology).
> - Android doesn't provide fine-grained permissions control or root access, so users can't block ads.
One caveat here - Google devices are probably some of the most allowing of root access and full device ownership - easily unlocked bootloaders basically allow it to be a one button process.
Some manufacturers make you put your device on a shitlist with them before they'll give you a key to unlock the bootloader and root it - others, like Apple, won't allow you to at all.
Once you are rooted, you do have full ability to block everything and get fine-grained permission control via XPrivacy for example. Android devices are actually some of the best here mostly due to strong community support. You can't even get this control if you want it on many mobile devices these days.
And yet Apple introduced a first-class ad blocking mechanism into iOS that even works in 3rd party apps (using SFSafariViewController). There are dozens of ad blocking apps available on the App Store, which are much more accessible to the average user than having to root your device and install apps that have full root privileges.
Yeah, that's definitely a step in the right direction, I certainly wouldn't say that Google does nothing to protect their interests in advertising - especially given their recent actions on the Chrome store with AdNauseum.
However, what I'm trying to get at is that, for example on iOS you still can't block in-app non-Safari ads at all. On Android you can do that if you want to, and a lot more, you can also block specific connections, block device-specific identifiers, APN lists, accelerometers, wake state, etc - it's a better compromise for someone concerned about privacy than other platforms even with this considered right now.
I've never seen a way to root my phones without some risk of turning them into bricks. How would I quickly and safely get root access to a Galaxy Note III without risking the destruction of the device?
Samsung is an example of a manufacturer that doesn't make it easy. Oppo, One Plus and Google (LG) all have very easy to unlock bootloaders that have "official" instructions from the manufacturer (and it's two commands away).
Can you still brick it? Yes. Is it likely if you type the two commands as you're told to on the website? No.
The Note 3 is not a Google-made device, it's made by Samsung. Google's own Nexus and Pixel devices allow it in a single button press, I'm unsure of other manufacturers except OnePlus which does the same as Google. As I said in my post, some devices are a lot better than others when it comes to this.
Also, bricking doesn't just happen, if you understand what you're doing you'll be fine. That "some risk" isn't random, it's in the case that you do something incredibly stupid. Get a custom recovery on there as soon as possible and you can pretty much recover anything.
This is simply not true. I've spent 3 years flashing and messing with my phone in many ways.I've hard bricked it only once. When it was updating my phone froze.
I didn't do anything stupid. It simply froze and messed with all partitions. It's hard to fuck your phone but it's a possibility.
Sorry, but "froze and messed with all partitions" makes it sound more like you fucked up and flashed the wrong rom or something - flashing an update should never mess with all partitions - only /system and /data really.
Use magisk or suhide and they'll work just fine. When you have access to the bootloader, there's nothing they can do to stop you - any attempt will ultimately be futile.
From what I've seen, there are no actual cases of this in practice at least not on Google devices - Google devices don't even have such a fuse. Maybe some Samsung stuff - but that's an easy fix, don't buy from Samsung. Also, it does not fail SafetyNet even on those Samsung devices, just some Samsung-specific stuff.
From what I've seen the only mention of such fuses on Google devices is one that enables secure boot from factory and forces you to run the unlock command as I mentioned to reflash your bootloader - nothing to do with safetynet or anything similar.
Let me know if I'm missing a device which does actually do this. Yes, it's theoretically possible, but requires sufficient crypto hardware and protection to make it significant and it'd be a significant shift in direction for Google to go this route currently in my opinion.
Of course a full safetynet emulation that would spoof the check as a different device would also be an option... but it'd be a pretty big undertaking, but one which I'm sure would happen if this ever became a remotely significant threat.
- Android apps like YouTube and Google News use a built-in browser frame to show navigated-to web pages, rather than opening them in an external browser. The built-in frame uses Chrome tech, without add-ons. They used to allow you to open links by default in an external browser, but not any more.
> The built-in frame uses Chrome tech, without add-ons
To block ads in Chrome Custom Tabs [1][2] you can either use Chromer [3] to change the custom tab provider to Brave [4] or use the article mode in Chromer.
Just a happy user and not affiliated, but Brave was a hallelujah-moment for me. Never got Firefox to be especially fluid, not even close to Chrome. But Brave, IIUC, takes Chromium and adds some adblocking. It's not perfect, and some ads makes its way in, but it removes quite a lot while still being very snappy. To the best of my knowledge and experience, I can really recommend it.
Now I just hope that Brave doesn't turn out to be too evil :S.
Ad-Away (installable via F-Droid if you are rooted) allows you to block ads at the hosts-file level, which will work in Chrome and in other apps that display ads.
Before anyone chews me out, as this effectively renders many free apps equivalent to their premium (ad-free) counterparts, I usually pay for said premium versions to compensate their developers.
The catch here is that it isn't as good - using uBlock in FF for Android for example will properly block all ad frames.
Using adaway or a pi-hole for that matter will not - they will only cause ad loading to fail (in some cases) - which sometimes results in frames showing errors on the page instead of a clean rendering of nothing with ad divs removed.
Personally I wind up using several methods - AdAway to kill most app ads, uBlock for web and Xposed to kill YouTube ads since the hosts-based methods seem to work rather poorly for them as their subdomains change all the time.
DNS-level blocking is, IMO, only a second line of defense. Nice to have for all those devices and applications where you cannot have a proper ad/tracking/malware blocker.
> You have to buy another computer that doesn't have root access.
It's called HDCP, and I believe it's a combination of your GPU, monitor and cable between it supporting it. Netflix and most other online streaming services do this too. It has nothing to do with root access.
Ad blockers do a lot more than just block domains. They look for and remove divs/DOM elements that contain known ad naming schemes and remove them from the page.
And if a site decides to start combating the ad blockers, the adblock list providers will update their rules specifically for the site in question. Adblock users get upset whenever they see ads, and report them pretty quickly.
The business I work for tried blocking adblockers and after a bit of back and forth they trumped us by blocking all AJAX requests on our site. We gave in after that.
To me, that would be a sign that you are winning the battle. If enough websites fight ad blockers, especially big sites, and the people behind those ad blockers make clumsy mistakes, then users will reach a point where they uninstall the blocker.
> people behind those ad blockers make clumsy mistakes
What "mistake?" That the developers couldn't foresee and block the exact elements against ad-blockers before they became real?
Reading the parent comment, it seems users are not opposed to reporting specific instances for the greater good. This is obviously more energy consuming than uninstalling, so where do you get the idea people will all of a sudden stop using an adblock because it failed to get passed one or two sites?
We're already seeing anti-ad-blocker-blockers being developed to remedy this problem. And if they don't come fast enough, users can just turn off their adblock for one page and be done with it.
Direct advertising is a dying legacy tactic. The most successful ads these days are the ones you can't tell are ads. They're also the stronger poison of the two.
It sounds like the people behind the ad blocker broke Ajax for that website while trying to block the ads - that's the mistake I am referring to.
You mention that native ads are gradually replacing direct ads as if it's a good thing. The good guys in the publishing industry go out of their way to prevents ads affecting content, they don't allow their writers/presenters etc to touch advertising and everything is clearly separate.
In the long term, ad blockers will just push out those people who are driven by ethics and you'll be left with the sleazy publications that are driven by PR. This is coming from someone who has dealt with PR agencies and constantly turned down proposals.
> It sounds like the people behind the ad blocker broke Ajax for that website while trying to block the ads - that's the mistake I am referring to.:
> The business I work for tried blocking adblockers and after a bit of back and forth they trumped us by blocking all AJAX requests on our site. We gave in after that.
I took this to mean that adblockers had to evolve and block AJAX on the website to make it accessible. Not, that they were hasty and disabled all functionality.
> You mention that native ads are gradually replacing direct ads as if it's a good thing.
On the contrary: "The most successful ads these days are the ones you can't tell are ads. They're also the stronger poison of the two."
> The good guys in the publishing industry go out of their way to prevents ads affecting content, they don't allow their writers/presenters etc to touch advertising and everything is clearly separate.
I didn't know this. However, I think it's a losing game. Consumers don't seem to care too much about the "good guys," unless a moral campaign is spear-headed (a la Tesla by The Oatmeal), only not seeing ads at all.
It's ironic really. We block ads so we're not influenced by them, but then we lower our guards and become susceptible to the indirect kind.
It's the natural order anyway. There will be those in the coming generations that will be like just like us. Except where we fought against direct ads, they will fight against the indirect. There've always been those unorganized who value critical analysis in all contexts, but their findings and ways never reach the public and make any impact.
Or maybe we've just made ourselves out to be sheep. As long as the coyotes aren't around, out of sight out of mind.
> In the long term, ad blockers will just push out those people who are driven by ethics and you'll be left with the sleazy publications that are driven by PR. This is coming from someone who has dealt with PR agencies and constantly turned down proposals.
As is with all things. You do not survive by being ethical, but by being the most adaptable, and sociopathy happens to be a great adaptation for sales.
I'm more interested in what happens next after the sleaze epoch. Will ads continue to become more and more manipulative then, finally after reaching too far, begin to wither and fade into the anals of history (albeit likely not as known as it should be, because of "out of sight out of mind."). Or will someone finally shake up this industry?
Does it matter if they stop visiting the site? Considering that they are using resources without giving something back in return? I guess it could be argued that they might be promoting the site to other users.
Other than that, they are essentially freeloaders and if a website has too many freeloaders it has to either get rid of them or convert them to something else. Or, I guess, the website could shut down completely.
>Considering that they are using resources without giving something back in return?
Funny. Ads are consuming CPU time and electricity that I pay for in addition to my attention and time, and compromise my decision making. To me, any of those are infinitely more valuable than resources the website expends on serving ads or trying to.
Does it matter if they stop visiting the site? Considering that they are using resources without giving something back in return?
Buzz. I block ads but I also post links to articles on social media that are then followed by people who do not block ads.
I have a couple of areas of focus where I am exceptionally knowledgeable and people who have anything more than a passing interest in those subjects check out links I post.
Or, I guess, the website could shut down completely.
If you're going to pirate a movie, you're probably not buying that movie. This is why on the whole piracy doesn't actually affect bottom line that much. Sure some folks would have rented it from their cable provider for $6 or whatever, but on the whole if you're going through the trouble of finding a good torrent and downloading it, you're not in the market to spend money on it.
Just the same, if you have an adblocker installed it's unlikely you're the type of person who is going to be clicking on ads anyway. And PPC ads are much, much more prevalent than impression-based ads. So if someone is blocking your ads, you're not losing anything. You get paid when someone clicks an ad, and if they're going to the trouble of blocking all ads, they're not clicking anything anyway.
You create a system where they won't or can't visit your site, and your traffic decreases, which will have an effect on your ad rates.
Impression ads are important for certain industries and bigger publications. I will agree that PPC ads are common in other areas and ad blocker users wouldn't click them. For impression advertising the ad blocker will result in lost income. PPC ads are the option of last resort for an established publisher because they are almost like gambling.
You can have all the ads you want, but if people aren't visiting your site - and sharing links to it, talking about what they read there, recommending it to others - they're not going to get you much revenue.
For informational sites, I'll block sites that violate my preferred conditions, and load them up in third-party tools, e.g., Archive.is, on the rare occasion I find an interest in their content.
When over 30% of your userbase is adblocking, it's not worth it. Your users will complain to you, not the adblockers. They don't even understand what's going on. Asking your users to whitelist the site doesn't work, either.
I thought "hey, the Newsletter and Security Notices icons aren't clickable", opened a ticket, and found that the input and submit elements that should be under those icons were being blocked by disconnect.me :)
It seems like a lot of the "Show HN" shares do that kind of thing...from domains that my work blocks. If it sounds really cool I'll load it up on my phone. Otherwise, that's a great way to make me decide "not worth my time" and move on to the comment thread.
Yes, that's why I also mentioned changing the HTML. For instance, change the ID of the div that contains the ad. It could even be random and change every time.
I don't have experience fighting ad blockers, though (I actually enjoy my ad blocker).
If Google did it, I doubt ad blockers would be able to block all ajax request on Google, or everyone would simply uninstall the extension.
I think Google can save ads about as effectively as AOL can save dialup. Probably their plan is to just milk advertising until the last Windows 95 computer dies too.
Unfortunately, ads pay for a lot of web content just like they pay for most TV content.
The 5% (of which pretty much all of HN is a part of) can probably afford to pay for the cost of creating that content. The 95%, consisting of not just hourly wagers but also salaried folks generally can't afford to. The trade off is ad supported content.
Something like 2/3 of millennials are blocking ads which means Google has already encountered just about everyone willing to consume their advertising. Ad-supported websites would be unprecedented if anyone really cared about them not having to find a new way to make money. Nothing truck drivers want is going to make driving trucks a career forever either.
The most promising front against ad blockers is forcing users to access your content with an application you control. That's why more and more websites want you to install their apps (Google search is installed by default on lots of android devices). It's very unfortunate that the incentives of users and content creators / advertisers aren't aligned very well and we will probably end up with most content creators dumbing down their web experiences in favor of better mobile applications.
I think pi-hole will at least partially block such things -- it intercepts at the network level. I guess you could have the app break itself if the ads won't load, but you can (mostly) do that in a browser.
I guess, for network level ad blockers, I'm not seeing obvious advantages to a program written as an app vs a program written to run in a browser.
As others have mentioned, you could just serve ads and content from domains which you wouldn't want to block, like www.google.com. And if an app receives its content encrypted (eg. SSL) you can't do a lot of filtering at the network level apart from domain names.
True. I want to be able to add my own root ssl cert to my phone for this exact reason. Then it will think that my MITM adblock proxy is the real deal. This should work great for web browsers. It does get a bit nastier for apps.
That's an immediate way to lose my eyes. I hate using my phone when I don't have to, and I hate installing site-specific apps even more. I'm already using a computer with a nice screen, mouse, and keyboard. Why would I want to use something with a tiny screen and touch input instead?
That's another good point. The people who are actively blocking ads are probably not the ones going to click them, even if they are not blocked. So not worth the money to invest in unblocking the ads going to people who aren't going to click them.
True. I am not meaning to say I am perfect (or that anyone can be), the hunch is just the expected click through rate of someone such as myself is likely much lower than someone who doesn't even know what an ad blocker is.
I think it's based upon the idea that people with ad blockers see less ads and so are still sensitive to ad impressions. In contrast, advertisements in really crowded situations (most people that walk about in urban areas, for example) require more effort to stand out as advertising saturation point hits.
I mostly combat any ad influence on me by making a point to _not_ buy anything that is advertised. Works out pretty well for helping me keep my sanity and conscience a little.
...? I think you're misreading my comment. I was suggesting that most people may think they are unaffected by ads, but probably are overestimating their immunity.
My second statement was highlighting how ones own opinion of oneself is not objective.
> My second statement was highlighting how ones own opinion of oneself is not objective.
And I retorted that ones own view of oneself can not be extrapolated onto the population at large if it is not objective.
But I should have been clear, when I mean "second statement" I mean so inside my own post. Ex:
1st Statement: "People," in this context, isn't defined either. If however we were to say that by "people" you mean millennials, then you would be wrong."
2nd Statement: "But if you meant "the population at large" you would be correct."
Which was in defense of the tech-literate and the young, but after researching some more my statements were based on old information.
The only people that come to mind who are easily susceptible to ads are the old and tech-illiterate, i.e those who don't have much experience with ads. Though this is just conjecture.
You would just block *.addomain.com at that point. Which is already what is being done on many of the blockers. There will always be domains to block, even if they add new domains, the blockers will probably be able to block them just as fast.
Sure, you could whitelist www.google.com, mail.google.com, etc, but couldn't they keep ahead of you if they were ok with using their main domain? They could even start using www.google.com/ad-id
Currently domain-based filtering is probably too small a proportion of traffic to even care about, but as ease-of-use comes to the masses there may come a point where the pro-ad side will implement this.
As far a I understand domain fronting works by sending a different host in the SNI header than later in the actual http request, thus hiding the actual requested host from an adversary watching the connection from the outside.
This would require cooperation from the sending http stack, in this case the browser. I doubt that this is a viable option in this case.
You really wouldn't want them on google.com/something since restricting cookies to sub-uris is a painful mess. Subdomains is slightly easier, but also easier to block.
There has been a - now defunct - working group which tried to address the issue of setting cookies on sub URIs: https://www.w3.org/TR/csp-cookies/ Particularly interesting is the proposal by someone from Akamai: https://lists.w3.org/Archives/Public/public-webappsec/2013Se... to include a path scope for cookies. I don't know if there are any more recent developments on cookies and their scope policies.
What if the New York Times decided to host all ads themselves? nytimes.com/ad-42.jpg couldn't be distinguished from nytimes.com/todays-front-page-image.jpg
Obviously it's more likely for ad ad-provider like Google to do this, but even then, if there's new content from Google (say a blog post) I expect to be able to see it.
I'm pretty sure that /ad-.jpg or something like that is already a blocking pattern. But you're correct: I'm seeing ads on some smaller sites that host the ad banners on their own servers. And you know what? I'm okay with this. I explicitly use uMatrix instead of uBlock since blocking trackers and malware is more important to me than blocking ads. (Getting rid of the more annoying ads is a bonus, though, and I might change my mind if first-party ads become annoying, too.)
The web advertising firm I work for has random looking domains for that purpose. The current domain generation algorithm seems to be using one to two domains a month. Although I've don't have access to any of the frontend code, I can only assume the DOM elements on the page are also obscured/randomized to an extent.
It is also my understanding that some of our boxes are here solely to proxy requests (either assets or websockets, which are increasingly popular in the field) to 3rd parties, to make sure it bypasses client side countermeasures.
Decreasing ads will hurt sites that aren't providing enough value I guess. Maybe others that could but were fragile. Cynically a natural selection of some sort.
Seems like one could block google's DNS servers -or perhaps even all DNS requests not destined for your pi-hole server in this example- on the router side and the Chromecast will fallback to what DHCP is providing.
I believe Pi Hole have tried to block YouTube Ads but Google randomise the domains they serve those ads from, and also from what I have read, they also serve some fundamental features via those domains too. So if you kill the ads, you kill YouTube.
Could youtube-dl the video then redirect the browser to a locally-served (locally from the pi-hole device, that is) page embedding the video, provided you've got the disk space to cache it and don't mind waiting a bit for the video to download (admittedly a lot of caveats here, but could work for many people). That'd actually be a better experience for most sites youtube-dl supports, I'd think, not just Youtube.
Safari itself is an ad-blocker, with its "reader mode". I could hardly imagine opening a newspaper page without it. Actually, my iPad 2 isn't powerful enough anymore to open a newspaper page, except if I use the reader mode.
Firefox as well. I use Pocket as a perferred reader generally, though its failure to provide a means of navigating directly to sites, or of showing a navbar, is annoying.
(Pocket ... has multiple annoyances. It's better than the alternatives, so far as I've tried, but that is one hellaciously low-set bar.)
The compounds in this medicine are public knowledge, but taking them could be dangerous. For a safer experience, review all medical literature pertaining to these compounds before consuming.
Not really the same. One of the main issues with curl pipes is that the server (or MITM) can detect that the request goes into a pipe.
This allows an attacker to display one (safe) source when you view it in your browser on your workstation, or wget it, and serve a different (nefarious) source when you curl/pipe it.
So, a more complete analogy would be: a bottle that gives you a safe chemical compound when you extract it for analysis, but throws in some VX when you go to administer it.
Summary: Fill your script with an invisible payload that fills any buffers, and put something time consuming (say `sleep 5`) early in your script in order to detect that the script is being executed directly rather than just stored to disk. If the client halts before having read all data, it is likely a `curl | bash` scenario. If it just keeps reading, it's a regular browser just downloading.
That way, it is the same as running cURL without piping the output to bash, so people can easily check the code without worrying if the server is sending them different code when they pipe to bash
If you're going to manually audit the source, you can curl into a file then run bash on it. If you're not going to read the code anyway, there's no harm in curl|bash.
You acknowledge the prioritization of ease-of-use/adoption vs. security so I think we're on the same page.
I doubt step-by-step instructions including a review of the script's content would improve the average user's security, in much the same way that click-through ToS dialogs always garner such much scrutiny.
Edit: another comment quoted the warning on the page, at this point it feels like complaining is tilting at windmills.
One thing is for certain, ad blocking is going to become more and more prevalent and never less.
The end game for ad blocking is to all but eliminate advertising. An ad blocking client could, ultimately, just block any domain that has aggressive anti-ad block features.
With enough users doing this, new sites that are ad free would quickly replace the old ad driven sites. Some of the ad driven sites would modernize.
Ads are a failed path. By eliminating ads we open the door to novel solutions. Only a cynical fool could believe technology isn't up to solving this minor problem. There are already a dozen potential solutions waiting for the incentives to change.
My ideal would not be for all sites to go ad free. It would be for advertising to go back to following the model it does everywhere else: Places that sell space for advertising either sell it directly, or go with a 3rd party to handle the advertising, but still maintain some control over what ads are presented, how and when.
I have never found ads in buses and trains to be particularly obtrusive, and usually don't find them to be tasteless. Advertising in print media is fine, and oftentimes even useful. Advertising on TV and YouTube is generally tolerable.
The only place where ads really become toxic is when they're being served up through ad networks. As far as I can tell, nobody likes them. Users obviously hate them - adblockers are darn near ubiquitous nowadays, and it seems that folks have generally realized that most Web ads have more in common with junk mail and telemarketing than they do with other forms of marketing. Advertisers don't seem to like them much, either, or at least they don't like them enough to be willing to pay anywhere near the price that they'll pay for ads delivered any other way. And content providers have to be aware that they've made a Faustian bargain.
But consider the advertising on a site like knitty.com, which is tasteful and relevant. And it does it using a targeting model that's eminently sane and civilized, namely, placing ads on a site you pick based on knowing that their audience and your target market are one and the same. That's an online advertising model I can get behind.
How do you imagine sites that are ad free would appear? What's their business model? I don't think that's a given, especially since most users won't pay (see the rise of freemium style apps vs paid ones in the app store).
I block ads, yet I financially support content creators by voluntarily subscribing to them via Patreon or tip them money (e.g. Twitch streamers). I also subscribe to publications to support journalism.
I'm also fine with ads in podcasts since the creators will sometimes at least use the product and it's implemented in a way that's largely expected, e.g. (plug in the beginning, plug in the middle, maybe a plug in the end...)
I remember back in 1994. There was actually better quality content back them. Think of half the sites being Ian's Shoelace Site. People that are just really into something and make a site about it. Static HTML sites that were fairly cheap to run. I had a site about all the special tricks in Tony Hawks Pro Skater. It was simple HTML and I got a ton of hits (this was later in 1999). No ads, I just loved the game.
Default settings use remote, shared DNS caches run by an advertising company.
Regardless, this is a step in the right direction. DNS is highly effective for this filtering out advertising.
Personally I just run my own authoritative nameserver(s) with all the IP addresses I need. No recursive cache.
When I browse to websites where I have never been and may not return, I am never using graphical browser that loads "resources" automatically from any random domain.
I am using a browser I compiled myself. I am only reading text.
Binary resources, e.g., video, can be downloaded non-interactively with an ftp/http client.
If it is an important website that I use repeatedly, then I have all the IP addresses for the resources the website's pages will need stored in a zone files. Then it is "safe" to use a browser written by an organization company that makes money from ads. All DNS requests are answered by my server(s).
I can retrieve (refresh) the IP addresses for my zone files very quickly with custom software I wrote to do this. My lookups are faster than a cold recursive cache and send out fewer requests.
IMO, the way to think about "ad-blocking" is not to try to imagine how to block every possible ad server. Instead, just focus on what web content you want and figure out what addresses you need to get it.
At one point a certain browser written by an advertising company had its own DNS resolver. Imagine your /etc/resolv.conf being completely ignored. Food for thought.
Do you find a setup like this takes more time to gather the information you're looking for? What is your primary reason for doing it this way? Privacy? Avoiding malware?
I really like the approach. This is where the net is headed. Graphical browsing is unnecessary in a way. Now, the only issue with that is the way the web is currently heading. A site hosted at aws or cloudfront is hard to whitelist...
A domain like *.amazonaws.com has nothing to offer that I have ever seen, pure rubbish. Blocking it will stop ads in mobile apps.
Cumulatively I think AWS is adding a hefty amount of latency.
The way their DNS is configured is often convoluted, requiring excessive layers of gratuitous lookups. It sometimes borders on absurd, much worse than I have ever seen with CDNs.
Fastly is doing a much better job with minimizing DNS queries. Certainly better than Akakami ever did.
For graphical browsers uMatrix is a good way to do this sort of thing. For every website you visit, you can tell it what resources from which domains to load and it'll block everything else.
This is just a DNS filter. Why is it a big deal? DNS filters sort of work, but they're not new or magic. They trigger some ad-blocking detectors when the ads don't load.
Easy to run at home with a web-based GUI for whitelisting, blacklisting and such. I like this tool a lot, personally, and it helps me feel better about keeping the family safe(r) online at home.
Agreed, it's been really nice to run at home. Took about an hour to set up from a diskless Pi I had sitting around, to a fully running system. Maintenance has been zero so far, and haven't had any downtime or anything.
The best part was when I first fired up the web interface and saw that it had already blocked 14 requests after hardly being up for more than 5 minutes. Nobody was home at the time, so it was kind of a wake up call to see idle devices reaching out to potential ad servers.
I was amazed at how much it was blocking on my Roku device hooked up to my TV. I ended up having to whitelist a few things in order for it to work, but it's blocking ads there too.
If you need something more powerful you can have a similar setup with Privoxy. I have a script[1] that regularly downloads the adblock lists and converts them into filter/action files for Privoxy plus some custom rules.
It's faster than a browser plugin and every device on the LAN can use it. It has a somewhat antiquated web UI and writing the rules it's not particularly user-friendly but it's working really well for me.
I used Privoxy since the days it was called Internet Junkbuster (90's), but got rid off it half a year ago, or so. It just does not filter enough anymore. Most ads came through. uBlock Origin (Firefox) gives much much better results. Privoxy is anti-innovative, as hell. Where are the regular updates to the filter-lists, like with all the other tools? Or sharing them in a cloud? Still no javascript injection/inline-execution in order to satisfy the complex Javascript ads? Still all being based on regular expressions? Before HTML5 we had ten years of XHTML, that could have been xpath'ed, xslt'ed, used for filtering. Now we have nodeJS. None of these modern technologies get used. It's still C and regex).
Nobody is making a "big deal" about it[0]. What they did was wrap something well known up into a relatively easy to install "package". They're not claiming this is new or magic, just simple.
Of particular added value there was mention of Android apps that can be setup to self-host an ad-blocking VPN / hosts filtering without rooting: https://news.ycombinator.com/item?id=13853408
Alas the plugin that does host-based blocking is not available in the play store version, it's/was a paid feature for the version on GH only. (edit: Don't know the current state though. remember from some time ago when I last checked it).
I went so far as to set up Dnscrypt with a pi-hole setup recently and it was almost as painless as advertised. And it finally gave me something productive to do with my RPi3! https://github.com/pi-hole/pi-hole/wiki/DNSCrypt could use a little wordsmithing, but it wasn't too bad.
Using this at home. Really interesting to see the blocked domains on the dashboard. Realised my two Samsung Smart TV's were constantly calling home for example. You'll eventually whitelist some things that break like Spotify/Sonos IIRC.
I may switch to an Odroid C2 if I go with a permanent VPN connection as the throughput of the RPi3 network port is not the best.
I love how they recommend curl-bash-piping, but then put a disclaimer beneath it, even with a detailed post about it. As if people would not just copy paste it anyway. I think they were just trying to dodge the usual curl-pipe-bash-is-evil comment thread – unsuccessfully, since I started it anyway.
I've been running a DNS based ad blocking for ages, but I realized that recently, youtube has been serving ads from the same domain as regular videos. I wonder if anybody has seen this.
When I tried pi-hole I often noted some urls were added 'automagically' to the whitelist, they will show up a few days after I removed them. All of them were weird domains.
Pi-hole Dev here. The only domains added to the whitelist are the domains on which the source lists are themselves hosted. It's probably complete over-kill, but the reasoning behind it is just in case one list tried to blacklist another.
Looks like your comments were getting killed with [dead] automatically. Maybe because of a new HN account combined with linking to the same site a couple times cause it to flag some spam detector? I vouched for your comments so they appear. Thanks for the project!
This could be in response to any number of comments on this page:
I use uBO and a few other blockers. I almost never see an ad.
A few days ago I saw an ad, and I was surprised. It was for Cadillac cars. I hovered over the ad, and it seemed to go directly to cadillac.com. And I was sort of OK with that.
The page, and the ad, seemed to be designed like any other legitimate link to another page or site. I don't know how the image made its way on to the page and in to my browser, but it appeared much less intrusive than a totally ad network-served ad.
Certainly the 1st party site could collect data about my visit and send it somewhere, but at least they appear to be more in the loop than just opening their site to all comers.
And if I clicked through to cadillac.com, they could do the same.
Anyway, that's more along the lines of what I've been wishing for as a consumer in web ads.
If all ads were a picture with a link served as part of the page, I wouldn't use an adblocker. I read newspapers full of ads and don't really mind -- the ads make the local weekly free!
But it wasn't me who started an arms race decades ago with pop-over/under chains and escalated with tracking scripts, auto-playing videos, and bidding platforms serving malware.
Are advertisers really surprised people opt out of such toxic behavior?
Would have been interesting to know who the domain was homed to. Might have just been an ad network with a CDN and cadillac.com CNAME'd to their servers. Looks like an origin served ad, but still flows through the tracker.
I add this to my modem/wifi ap. and then just let every device use it to resolve. if the device allows to set a hostfile, I also add a local copy for when iam not in my network.
* GitHub - StevenBlack/hosts: Extending and consolidating hosts files from a variety of sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, yoyo.org, and potentially others. You can optionally invoke extensions to block additional sites by category. || https://github.com/StevenBlack/hosts
I take the hosts file approach to blocking on my VPN/DNS/Proxy servers.
I also add most Google/Facebook/etc domains to cut down on tracking and bandwidth from remote resources. Somehow the tentacles of Google (Analytics/Fonts/etc) and Facebook extend to most of the web. Their embedded javascript is everywhere. Blocking at the hosts file level seems to be the easiest and most convenient approach.
I have to admit to doing as little as possible from web browsers on phones, but on the desktop I rely extensively on uMatrix + NoScript (don't know if adding PrivacyBadger on top would buy me anything). However, NoScript for Android seems to be moribund and I don't think there is a uMatrix for Android either. DNS-based ad-blocking seems very 90s (i.e., designed for an era that's less invasive than today), and there's a ton of javascript content that really needs to be filtered as well if you want to counter all the ads + tracking. Is there any equivalent to NoScript + uMatrix on Android?
I only have uBlock Origin on Firefox for Android, as opposed to uMatrix on desktop Firefox. It's better than nothing wrt tracking blocking, and it kills most of the ads.
I haven't had time to look at the code yet. I have some questions though.
As an experiment a while back I wrote a simple dns server that blocked ad-related domains. https://github.com/geuis/lead-dns. While it technically worked, it made using the web almost non functional. Nearly every site was broken in some way. So blocking purely by domain isn't going to work. I wonder how pi-hole is dealing with it.
We have a web interface with a whole host of tools to easily identify and whitelist the domains that may or may not be causing issues with sites you browse.
Everyone's mileage varies, but I have only had to whitelist 5 or 6 sites using the default blocklists.
PiHole is pretty cool, very 'plug n play' which I like. A sufficiently advanced average user can set it up without too much trouble just following on a guide, even a relatively tech savvy 'lay' person can do this.
If you like a more technical solution I prefer something like running a Unbound + NSD server
Here's some great tutorials on that:
(Kudos to the people who write Calomel, i really liked these tutorials, it was a great way for me to get started and look into these services deeper once understanding what was going on here)
I'm really big into having ones own DNS server on the network instead of completely using outside solutions. There is little overhead with a sufficiently modern implementation.
Also, these solutions run on FreeBSD/OpenBSD for those who prefer.
As a complete aside. Aren't most routers, esp. business class routers, running modified Unix/Linux anyway? Why on earth hasn't a reputable company made a guns ready router that lets you have access to the Linux/Unix underpinnings without flashing (albeit awesome) Open Source alternatives? I would think in the 'business/enterprise' class hardware side this would be more prevalent.
Maybe I just don't know of any solutions like that available stateside. I found one in Europe:
I'm actually wondering if they simply didn't repackage this hardware with a better then average design for a case, frankly. The specs are very similar. I think even though these are AMD embedded processors they're ARM, not sure though, it didn't say (or i missed it).
I'm running HomeAssistant from an Odroid C2 --its working perfectly.
If you haven't checked out DietPi yet, you should. It includes optimized installations for a few hundred things (including PiHole), from Mumble servers, to MAME emulators.
How are people dealing with whitelisting when a website becomes broken because of an over zealous block. I find myself turning off ublock for some websites at least once a day, not because there are ads on the page but because they're issuing a dependency somewhere that has been blacklisted.
Yeah, first think I did after setting up pi-hole was go and visit all my primary sites, and see what was being blocked. Then white listed things I was OK with, or things that were breaking sites.
Also, not sure how I feel about having this device as my primary DNS server for my entire internal network. What if the project gets compromised and injects a number of malicious DNS entries, now my entire network is toast?
You can always review the code for yourself over on our github repo (https://github.com/pi-hole/pi-hole)! As well as the devs, there are so many pairs of eyes on the code that if anything fishy were to happen, everyone would know about it (not to mention it would completely undermine the hard work we've put into it over the past couple of years!)
Updates are manual, too, so unless you intervene, there is no reason that a working system would suddenly become compromised, except if somebody had access to your network. But then you have a different issue...
To reliably run this DNS server I would have to have another host on the network that uses a separate DNS route that checks every entry on pi-hole against the secondary to confirm something fishy is not going on.
I've been doing this at home for probably close to 10 years or so, set up manually using dnsmasq, and periodically fetching new blacklists. Nice to see it wrapped in a tidy package--great work.
That's true, but instead of empty ad being server (as in case HTTP), HTTPS request timeouts and increases loading time for many pages, because the page waits until all JS loads.
Another problem is that some browsers will retry failed request several times (Chromium) effectively prolonging the time before final error is confirmed.
It's not officially part of the Pi Hole project, but I've been using https://hub.docker.com/r/diginc/pi-hole/ for about six months on a Raspberry Pi (though they support other platforms) and had no issues. They also keep up to date with new releases within a day or two usually.
The name has just kind of stuck, even though you can in fact run it on most linux distros (We officially support Ubuntu/Debian based distros, but there is limited support for Centos, and even forks for Arch, and Docker!), on a whole host of different hardware.
Stupid question: how does it work when the majority of traffic is through SSL, making a request to an ad and a request to actual content indistinguishable?
I don't think all websites serve ads from a different host. Do they?
Of course it can! If it has a domain name, it can be blocked. What you may see is the browser grumbling about it.
Essentially all that is happening is a client asks for a domain (e.g ssl.google.com) and if it is on the blacklist, then it returns it's own local IP address for that domain instead of the real one. Which is where you will find the browser complaining. All we return is a blank page/blocking information page, but not via https.
Of course, we could probably get around that by generating self-signed certificates on install, but the user would then need to install that certificate on all of their client machines (I think, I'm not massively proficient in this area!) which makes it an extra, and not vital, step that may scare off the more novice users.
I wonder if there's any way to integrate this with OPNSense firewall. I already have that set up as my LAN's dns server so I can name my internal computers.
That's the direction I was thinking of going. My main concern is that I spent quite a while figuring out the fastest dns servers to use, and I have them set in OPNSense. I wonder how adding pi-hole to the mix will effect speed... (As in, finding the dns servers that select the best servers for my streaming video apps...) Guess the best thing to do is to try it. I would hope pi-hole would let me select it's upstream servers as well. :)
Network-level means no easy opt-out on a per-site basis. You cannot choose to support certain sites or view occasional sites that, understandably detect ad-blocking.
There is a disable button on the web interface[1] which allows you to either disable it permanently, or for a specific amount of time. Of course, client devices need to clear their DNS cache, too, in order for this to work properly, but at the moment there is no way of automating that.
There are also other tools to help with blacklisted domains that cause issues/site breakages, such as a query log to identify them, and the ability to whitelist with ease!
Is it? I'm sitting behind a similar DNS-level adblocker and the only thing that I've noticed broken from time to time is analytics.twitter.com (some external links in the Twitter app route through there, but not enough to annoy me). Or maybe I'm just avoiding all those blocker-blocking sites effectively.
When you say unbearably slow, do you mean the actual DNS resolution, or the Admin interface?
An Rpi3, even a Rpi B or zero is plenty good enough for DNS queries! The Admin interface has been a bit of a bugbear for a while, but we are working on some massive improvements for the 3.0 release (coming soon™)
Most definitely, although it of course depends on what else you're running on that VM. I have a home server running, among other things, a very similar setup [1] based on dnsmasq and a simple shell script cronjob, and the dnsmasq weighs in at ~12 MB memory usage (RSS).
I used pi-hole until I got pfSense set up. Then I migrated to pfBlockerNG, which has a lot of the same DNS blocking capabilities built in. You may want to check it out.
https://github.com/pi-hole/pi-hole/blob/master/adlists.defau...
There's a few on there I don't use and will look to implement. There's also a few they seem to have missed (perhaps intentionally?) so below I have included the lists I use in case it's useful for anyone else: