In the settings, http://duckduckgo.com/settings.html, you can turn on POST requests as well as disable favicons and 0-click. Switching to POST alone should fix this issue for you. In particular the Referer heading then becomes:
Or make it automatic or at least default for https? If you're using https, the only thing you have to hide is your search term, right? And that's what's being exposed to Amazon, or whoever else is seeing your URL along the way.
If that separate domain can be used in the Chrome URL bar, count me in. Thought it would be even better if it could be done using the main URL in the Chrome URL bar somehow...
Some people, perhaps most, don't like POST because it is annoying to copy URLs and use the back button. I'm not opposed to making it the default for https, but I don't want to make a default that most people don't want either.
That's a good point. Maye you could put a small warning on the page if they disable 'POST' that there might be leakage of their search terms to the sites they visit?
"Hello dear user, you probably know exactly what you're doing, but on the off-chance that you don't, please realize that disabling the POST option for https connections may leak your search terms to the visiting site".
The problem is not that the referer leaks to the sites you click through to. The referer leaks to sites as soon as the results page is displayed because there are externally hosted images embedded in the results page.
But what's the point of an https search if it's not really secure? The only thing the user is trying to hide is the query, and it's not being entirely hidden.
For users who are annoyed, you could explain to them somewhere on the site that you don't put it in the URL because it exposes their query. If they really want a secure search, I imagine they'll understand the tradeoff.
Where 34g7h3giuh3g would be the ciphertext generated by encrypting "hacker news". That page knows what the search term was because it will have decrypted the parameters on the server side, but any referers would just contain "garbage", and it would also mean people can copy/paste the address bar about.
all the other person has to do then is perform a search to see the secret terms. granted, it's better than it showing up in plain text in the referrer, but you could easily write a script to scrape the actual terms...
What if you use the IP address of the user as a seed for the encryption? Then if someone else used the same key from a different IP they'd get different search terms?
Furthermore, the setting cookies are simple transparent cookies (f=-1 to turn off favicons), they are not tracking cookies. They are only used when you set the preference and then only contain your preference itself.
Good catch, really. Trivial to fix fortunately. Not that I care too much about super secure secret searches, but if they're advertised as such they should be. And even then, you probably should assume they're not. (secret).
> Not that I care too much about super secure secret searches, but if they're advertised as such they should be.
Exactly. Sure you could use Tor and disable everything and log on a random wifi from a stolen computer that will then be smashed to bits and melted, but that's beside the point; if DDG advertises privacy, it should do everything it can to deliver on that promise.
I can't figure out what the trivial fix is. Would proxying all S3 requests through his server fix the issue, or do the headers get passed through to Amazon anyway?
One fix is to not directly send the users to the result page. Instead link to a redirect script on the ddg servers i.e, duckduckgo.com/goto.php?link=http://search-result.com/ and then have goto.php remove the REFERER from the request headers.
Not sure if that will work in all browsers, iirc a 301 or a 302 can still pass those headers on. The only trick I know of that will not do that is by using a 'meta refresh' with a time set to '0', but that has bad implications for the working of the 'back' button.
Referer is a header the browser sends, Location is a header the server sends. Also, the Location header needs either a relative url on the local machine or a fully qualified one. In this case it would have needed a fully qualified one.
I wouldn't say there is a trivial fix. One option would be to encrypt the search term client-side with blowfish (or some other 2-way hash scheme) and use that as the url string.
I think you might be confusing S3 with EC2? I think "offsite images" is an apt description for stuff served from Amazons web frontend to S3... DDG don't run it, they just upload the content to it.
If super secret squirrel searches are your need, you may wish to look at things like, disabling Referrer, Tor, Disabling Flash completely (as it represents it's own version of a cookie), disabling cookies, regularly DBANing your system, and even then realizing that you are still screwed according to EFF's research in their Panopticlick project (https://panopticlick.eff.org/)
True. torbutton does a lot of that, however, so if you enable it and completely disable javascript you can get your browser uniqueness down to about 1 in 1500
Sure, Amazon S3 could be spying on your searches just like any web host (especially a cloud host) could spy on connections to its customers' sites. I don't think this is very likely, however.
Nothing shown in this blog post indicates that DDG's S3 account is logging IPs or that there's anything at all wrong with the privacy policy.
You misunderstand. The images are being served from Amazons web frontend to S3. They probably log the requests, not for "spying" reasons, but simply because that's what people tend to do when serving websites; log the website requests that are made against it. Because they do that, a government could contact Amazon and ask them for the data.
Knowing that, DDG may as well log the IP+Search themselves, as it makes no difference. The data is already logged and retrievable by contacting Amazon, therefore what is the point in DDG not logging it anyway?
Isn't this sent by the user's browser, and not by DDG? It's a client-side configuration issue -- turn off referrers in your browser. Your browser would send the same header if you clicked through a DDG search result.
It is a duckduckgo issues because they should not be loading third party graphics. The referrer will be on in almost all cases. People just look at the 'lock' item and will assume they're safe.
Whatever happened to that 'mixed content' security warning, I thought that was pretty effective against stuff like this?
Because complaining about "mixed content" was completely pointless in the general case -- you could mix content across multiple https sites, but the certificates were never correlated, and the second site would still get all the headers just the same.
It's extraordinarily user-hostile, and would just add to the pile of pointless wankery that keeps people from using https (see also: shitting all over self-signed certs when in reality the CAs don't do shit for their rent and identity is useless anyway).
The actual solution is to never send Referer headers for cross-site requests from an HTTPS page.
> The actual solution is to never send Referer headers for cross-site requests from an HTTPS page.
That should be on someone's todo list at the major browser vendors. You're right, there really is no point in sending that header along, and sending it can cause all kinds of trouble.
I think the combination of a POST instead of a GET and the images via https should be pretty much bullet proof. If someone is stupid enough to re-enable GET requests for their https connections they have only themselves to blame if there is any leakage to the target sites.
"First they ignore you, then laugh at you. Then they fight you. Then you win." (Gandhi via Robbie Williams)
Rightly identified and well addressed by epi0Bauqu. And also worth noting that this sort of constructive criticism is a great sign of positive market traction. I'm bumping into DDG more often on the web, which is excellent.
POSTs and HTTPs are good, but here's a few other ideas:
1) Embed the images in iframes, which are then embedded in the page. The iframes will swallow the referrer, so provided they are hosted somewhere where logs are discarded then it should be fine (I'd want to cross-browser test this before relying on it though).
2) If the browser supports data: URIs, then embed the image in the page. Obviously this might have some costs, but you could do it for HTTPS only perhaps?
3) Request the images in Base64 encoded form (or binary strings) via XMLHttpRequest after the page has loaded. You can overwrite the Referrer header in XMLHttpRequest
4) Preload the images BEFORE the search is done (ie, on the search page). With appropriate headers Amazon won't see a request (this won't work from the browser bar, though. I could imagine some ways around that, but I'm not sure they are worth it)
> 1) Embed the images in iframes, which are then embedded in the page. The iframes will swallow the referrer, so provided they are hosted somewhere where logs are discarded then it should be fine (I'd want to cross-browser test this before relying on it though).
Almost all web servers keep a log of HTTP requests if only to weed out troublesome pests.
The worry is not that the information is sent. The headers, IP address, and search query is always sent to any search engine regardless of their privacy policy or whether it's sent as POST or GET, so the worry is not that information is being sent, but rather that the query string in GET requests is most likely kept in a log file at least for a few days.
If Duck Duck Go was upfront about how long this HTTP request log was kept, would that make the default search with GET requests acceptable? I think having search queries sent by default as POST would be irksome for a default setting.
>As it stands, they should append this to the end:
>"but the architecture of our site lets Amazon log it."
This is the best suggestion yet. Of course there are numerous hacks possible to make it more secure, but each one at the cost of user experience. DDG has so many things going for it, I don't think for the majority of users security is at the top of their list.
Do a http redirect to a url which has an encrypted version of the search term in the url parameters before displaying the search results. That fixes the problem without hitting the user experience.
However, at the end of the day, everything can be logged anyways. If someone is really concerned about this, they shouldn't be using the internet.
Edit: I want to be clear. If someone is concerned about the fact that things can be logged, then they are better off not using the internet, as pretty much everything has the potential to be logged. This is, after all, what the complaint is: the potential of being logged.
Well doesn't duckduckgo use other search engine's results and combines them with its own? Or does that not involve sending the search terms to the other search engines?
Yes, but then it is duckduckgo.com doing the query. Here they're passing on the query in the referrer string and the users original browser makes the non-secure request to amazon.
That sort of gives it all away.
So if duckduckgo would just send back the answers to the query and not use a page that requests resources from third parties they'd be fine.
It's a fine story, and it is fixable. And it probably should be fixed, if only because fixing it is less work than talking about it. And due to a terrible oversight the header is actually mis-spelled and it came out as 'Referer:'.
Fixing this will make duckduckgo.com even better, which is awesome.
No, definitely not, I just wanted to point that out.
I knew about this for years, having stared at a bug for a whole day before I realized it was the header that was misspelled. It never even crossed my mind until I was away from the keyboard for a few hours.
Except if the same user clicked through to a result, the server of the result page would receive the same referrer header w/ the DuckDuckGo query in it.
If you don't store the search term in the URL, then it doesn't get passed along in the referer. Store an encrypted version of the search term in the URL instead.
Referer: https://duckduckgo.com/