Dominic White - Privacy

Tracking the Trackers (my mods to the Collusion AddOn)

nospam@example.com (Dominic White) — Thu, 01 Mar 2012 02:12:34 +0200

In July last year, Toolness, released a cool Firefox add-on, named Collusion, that draws a pretty visualisation of who's tracking you as you visit different sites. It gained some popularity after Gary Kovacs, Mozilla CEO, showed it off in his TED talk yesterday.

It's a great little add-on for making something quite hard to explain to people quite visible. However, I didn't like the fact that it only showed trackers that set a cookie. For example, the requests to Facebook to fetch Like button JS, or calls to Google Analytics were being missed. There are lots of ways to track people other than cookies. So I edited the add-on to include third-parties to whom a request was made, but where a cookie wasn't set.

This ends up providing much more data, and the graph gets busy quick. So I re-enabled the dynamic radius function that grows nodes with many incoming links. This gives you a quick visual way to see which trackers you're hitting the most. I also updated the trackers list, as it was over 6months old. I'll switch to the API once/if privacychoice makes it available.

Here's an example of what a request to memeburn.com looks like with all tracker included, followed by the same session after hitting privacychoice.net's top10 most tracker-heavy pages.

I've sent a pull request to the maintainers, so hopefully it'll get merged. In the meantime, my edited add-on can be downloaded here:

collusion-singe.xpi SHA1: 6a08ee743e22da29d97fff5b0e525264666fbcfb

I'd love any feedback you may have.

Caveat: This will show domains that obviously aren't trackers, such as CDNs, but that's minimal, and easy to spot with the new radius stuff.

A Response to Seth Godin's "The Illusion of Privacy"

nospam@example.com (Dominic White) — Fri, 17 Feb 2012 22:48:26 +0200

Seth Godin is a smart guy, and people listen to what he says, but he's recently ventured into an area he knows little about, privacy, and made some mistakes I feel should be corrected.

Seth lays out two claims, the first is that we have no privacy, and the second is that consumer "privacy scares" are actually just because consumers don't like surprises. The first is the most important. It's something you hear all the time, and it damages the potential work privacy advocates and developers can achieve.

You have no privacy

The first mistake Seth has made, is to assume that he knows anything about privacy. As Iain Currie points out in his excellent paper "Some implications of a dignity-based conception of privacy":

[Much] writing about privacy tends to be ‘intuitionist’. This is a form of moral argumentation that relies on people’s innate intuitions of right and wrong. [] The difficulty with [intuitionism] is the unreliability of its results. What some people experience as shameful violations of privacy, others do not. A more general problem with intuition as a basis for ethical decision-making is that some extrinsic quality-control measure always seems to be required to test the rightness of one’s intuitions. The fact that a lot of people feel strong moral revulsion at, say, the idea of interracial or homosexual sex is generally not thought to be a good reason for judging those practices as immoral.

It turns out that privacy is actually quite a tricky concept, and both the fields of moral philosophy and law have spent a considerable amount of nailing it down. I tweeted that a good place to start any readings on privacy, is the Stanford Encyclopaedia of Philosophy's entry on the matter. Just wading in to the field and declaring that credit card, phone and web logs are the sum total of your privacy is the first mistake. The second mistake in his final sentence, is assuming that this is a fait accompli. Just because it's happening, doesn't mean we shouldn't be fighting it. Companies make lots of money by collecting information and selling/monetising it. User's don't really understand or directly/immediately/accountably experience the violations, so all we have to keep corporate greed in check are the privacy advocates, who get privacy, and are working to out the abuses in a way the average user can grok. Letting Seth get away with propagating this stuff hurts us all.

We don't like surprises

In the second part of his entry, Seth attempts to boil privacy reactions to not be about privacy, but rather the fact that we don't like surprises. The obvious rebuttal to this is that lots of people do like surprises. Personally, I hate it when I know what my wife is getting me for my birthday. There's nothing intrinsic to a surprise that should make it a negative, or use it as a design guideline for developers. So if we are to be chartiable to Seth's argument, possibly he meant, we don't like bad surprises. This makes sense, nobody likes getting mugged for example. But, in the end, the "surprise" part appears to have nothing to do with it, and the "bad" part has everything to do with it. It's the loss/trauma of the mugging that is bad, not that it was surprising. What this means, is that the charitable interpretation of Seth's point is: "Consumers don't like it when you do bad stuff with their data."

I wish Seth had analysed his argument, and realised that's what he was actually saying. Because, the next logical step is to realise that his advice to developers should be, to stop doing bad stuff with users data. Not, that it's too late to worry about privacy.

Since Seth didn't, I will, here's my advice to developers:

Don't use data in a manner that does not benefit the user.
If you must, gather actual consent, and only use the data in the consented to manner
Allow the user to opt-out, and still retain some service either up-front or at a later stage

Conclusion

In the end, Seth has propagated a lie that many before him have told. He's just a big public figure. Privacy is hard, you can't knee jerk it. Online/electronic privacy is an active field of research, and improvements should be supported not put down with tired throw-away lines. What's more, technical ways of doing this are available and should be investigated, no matter their surprise value.

P.S. If you're interested in this, you may also enjoy my rebuttal of Paul Rubin.

Mobile Privacy-Enhancing Proxies

nospam@example.com (Dominic White) — Fri, 11 Nov 2011 16:06:42 +0200

Modern web-browsers support all sorts of add-ons and plugins. From a privacy perspective, this means you can block adverts and trackers, use tools like GoogleSharing and other request re-directors. However, mobile devices typically don't have the same extensibility. While searching for a way to implement this, I came up with using proxy.pac as a way to do some more advanced network jiggery pokery, without requiring platform specifics (i.e. should work on iOS, Android or even Firefox & Chrome), or the need to jailbreak.

Unfortunately, 1984.za.net is down, and since then I've done a bit more work on this. I presented this briefly in my ITWeb presentation last year (slides 27-30), and figure it was about time to make this properly public. I've put it up on my github at mobile-proxy (have I mentioned I love github).

This is still pretty rough, but it proves the methodology and can be extended.

Two interesting things to come out of it are:

iOS Mobile Proxy Configuration

You can edit the proxy used when your phone is on a mobile network (i.e. not wifi) by editing the file (once jailbroken): /Library/Preferences/SystemConfiguration/preferences.plist and adding the ProxyAutoConfigURLString key as below:

<dict>
	<key>HTTPEnable</key>
		<integer>0</integer>
	<key>HTTPProxyType</key>
		<integer>2</integer>
	<key>HTTPSEnable</key>
		<integer>0</integer>
	<key>ProxyAutoConfigEnable</key>
		<integer>1</integer>
	<key>ProxyAutoConfigURLString</key>
		<string>https://<host>/proxy.php</string>
</dict>

It was pointed out on twitter that the iPhone Configuration Utility should allow this to be done without the need to jailbreak. I'll test it and update things if it works.

BlackHole Proxy

The second interesting thing, is that to block access to a website just redirecting to a non-existent server won't work as WebKit based browsers in particular will try again without using the proxy. Thus, a blackhole proxy was needed. Gert at Sensepost wrote a quick 'n fast twisted server for those purposes, and I extended it to drop privileges to reduce attack surface. It's included on github.

Apple's PR on Location Data

nospam@example.com (Dominic White) — Thu, 28 Apr 2011 06:37:02 +0200

Apple responded to the location logging stuff with a Q&A aimed at dispelling some of they myths all the hype has created. The only problem is, they try to dispel some of the facts too.

1. Why is Apple tracking the location of my iPhone?
Apple is not tracking the location of your iPhone. Apple has never done so and has no plans to ever do so.

3. Why is my iPhone logging my location?
The iPhone is not logging your location. Rather, it’s maintaining a database of Wi-Fi hotspots and cell towers around your current location, some of which may be located more than one hundred miles away from your iPhone, to help your iPhone rapidly and accurately calculate its location when requested. Calculating a phone’s location using just GPS satellite data can take up to several minutes. iPhone can reduce this time to just a few seconds by using Wi-Fi hotspot and cell tower data to quickly find GPS satellites, and even triangulate its location using just Wi-Fi hotspot and cell tower data when GPS is not available (such as indoors or in basements). These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple.

4. Is this crowd-sourced database stored on the iPhone?
The entire crowd-sourced database is too big to store on an iPhone, so we download an appropriate subset (cache) onto each iPhone. This cache is protected but not encrypted, and is backed up in iTunes whenever you back up your iPhone. The backup is encrypted or not, depending on the user settings in iTunes. The location data that researchers are seeing on the iPhone is not the past or present location of the iPhone, but rather the locations of Wi-Fi hotspots and cell towers surrounding the iPhone’s location, which can be more than one hundred miles away from the iPhone. We plan to cease backing up this cache in a software update coming soon (see Software Update section below).

Their claim pretty explicitly states, that they aren't storing location data based on your actual position. The facts would appear to indicate otherwise (these are based on the copy of consolidated.db that was on my phone:

The tables "CellLocationHarvest" & "CellLocationLocal" store both "Speed" and "Course" entry (several others have these fields, but did not have any or valid data in them). Unless cell towers have a habit of moving about, this would appear to be logging *your speed & direction* and not just "tower data". Granted, the "CellLocation" table containing the most significant amount of data, did not have valid data in the speed fields.
The table names imply different uses for e.g. we'd expect CdmaCellLocation, CellLocation & WifiLocation tables to store the info they speak about above. But the "LocationHarvest" table not only stores valid speed & course fields, it also assigns a unique "Trip ID" e.g D47CA532-84C9-40CD-8BE6-B3895837DA3C. This looks like a unique identifier based on *your* movements, not those of the cell towers.
Even if this was downloading offline caches of cell towers & APs for assisted GPS, given this includes details as granular as my neighbours Wifi AP, this is still more than enough to track your actual location. We've seen large data sets with "unique anonymous" identifiers deanonymised many times.
The data is good enough for forensic investigators to use, here's a screenshot from a book on iOS forensics: "consolidated.db [snip] is potentially one of the most forensically rich files an analyst can use." It strikes me that if it's good enough to use in the courts, then the implications may be a bit wider than Apple claims.
And finally, further down the QA, Apple contradicts their statement of "The iPhone is not logging your location" by explaining that it is, and this will be used for traffic information. This explains the "LocationHarvest" table mentioned above.

8. What other location data is Apple collecting from the iPhone besides crowd-sourced Wi-Fi hotspot and cell tower data?
Apple is now collecting anonymous traffic data to build a crowd-sourced traffic database with the goal of providing iPhone users an improved traffic service in the next couple of years.

On the up side, they acknowledge at least one bug:

7. When I turn off Location Services, why does my iPhone sometimes continue updating its Wi-Fi and cell tower data from Apple’s crowd-sourced database?

It shouldn’t. This is a bug, which we plan to fix shortly (see Software Update section below).

I haven't seen what is actually transmitted to Apple, so can't comment on how much is uploaded or downloaded. However, I can attest to have seen the iPhone populate the file with tower & AP information when first populating it with data (123 cell towers, and 401 wifi APs). So that part is at least true.

In conclusion, I certainly don't think this is a serious threat, but this file does store rich location data that can be used by anyone with access to it to disclose a significant history of your movements. Apple has attempted to play that down, but for people to who the privacy of that data may be of critical importance (think protesters in Lybia or Egypt), they should take steps to protect themselves. Finally, it is also my belief, that based on the data in the file, if Apple has access to the same data, then there is enough information for them to uniquely identify both you, and your location history. They claim they aren't, but it just takes one breach for all of this data to end up somewhere we need to make different assumptions about, and I'd prefer that the location data Apple (and others, like my mobile service provider) collected without my consent, be deleted.

Blocking iPhone Tracking (consolidated.db) Solved

nospam@example.com (Dominic White) — Wed, 27 Apr 2011 00:43:32 +0200

After several days of trying all the different solutions proposed as the story has emerged, I think I've finally got a solution that is both usable (i.e. doesn't break anything) and permanent (i.e. apply once and let dry).

My original suggestion of rubbish values + read-only didn't work, untrackerd takes up valuable memory & battery and misses nearly all the worrying data & the SQL triggers file from Tehtri also missed some data and breaks some functionality (most notably the compass).

However, Tehtri's idea was the best. They proposed a set of SQL triggers that would reset the consolidated.db to a clean state and prevent it filling up with your location data. All this without requiring a persistent daemon or the need to re-apply the fix. I've edited their SQL (you can see the changes here, this is merely for those interested, don't run it) to reset consolidated.db to how it looks when locationd creates a blank new one, then modified the triggers to do the same (rather than just blank all the tables). I've also extended it to include some tables they had missed, and not delete some data it shouldn't (e.g. blanking TableVersions makes locationd unhappy, and it has no location data in it anyway) . Finally, I leave the last entry of the compass calibration (in the trigger too) so you don't have to constantly recalibrate your compass (every minute or so it was). I haven't found it break anything yet (even location via nearby wifi BSSID works without storing the values). Grab the final, clean version from here, and apply with the sqlite command:

sqlite3 consolidated.db '.read singe-iphone-privacy.sql'

There are three ways to do this:

On a jailbroken phone with sqlite3 installed, you can scp or wget the file to the device and do it there & then.
On a jailbroken phone, you can copy consolidated.db off, apply the patch, then copy it back.
On an unjailbroken (aka normal) phone, you can use the backup & restore method

If you're jailbroken, you can figure it out.

Update: The below instructions no longer work after iTune 9.2 implemented a new proprietary backup format. I'm hoping the documentation here will allow a quick update of the file hash & size to let the restore work, but until I or someone else has time. You'll need to be jailbroken to protect yourself.

For normal people, follow these instructions:

Plug in your iPhone and let iTunes make a backup. Make sure the backup isn't encrypted, we'll do that later.
Go to your backups directory. On OSX it will be in /Users/<username>/Library/Application Support/MobileSync/Backup/ In Win7 it will be in \Users\<username>\AppData\Roaming\Apple Computer\MobileSync\Backup\ other windows locations are listed here. It will contain several randomly named directories, change into the one with the latest timestamp (sort by last-modified date) to work on your last backup.
Get hold of the iphonels.py file. Either by copy pasting from the original here, or just downloading this one.
Look for the randomly named file that maps to consolidated.db by running the iphone-ls.py and grepping for "consolidated" e.g.: ./iphone-ls.py | grep consolidated. It will look something like '3086b93ce76d2847dc283405811e284a7c815839'. If you're on Windows, you'll need to install python.
The value in brackets is the name of the file as it is stored in the backup folder. This name will be consistent across all your backups.
Apply the SQLite modifications from here to the file, either use the sqlite3 command line utility e.g. sqlite3 3086b93ce76d2847dc283405811e284a7c815839 '.read singe-iphone-privacy.sql', or use your favourite GUI.
Overwrite all copies of consolidated.db in each backup directory with the new version. This is easy to do as the random file name is consistent across backups, so just copy the new file into each backup directory.
Next, plug in your phone, and restore your backup. Remember to re-encrypt your backups.

Update 1: Restoring to a non-jailbroken phone doesn't work. Updated the .sql with the 'vacuum' command to flush out old data (thanks Istvan).

Quick note on the iPhone Location Tracking Disclosure

nospam@example.com (Dominic White) — Thu, 21 Apr 2011 07:45:27 +0200

Update 3: I've modded Tehtri's approach and it appears to be working nicely, read this post.

Update 2: untrackerd seems to clear out two tables only, and not the most worrying tables either (at least in my file). After 2 days of use, it didn't change a single entry in my consolidated.db (I was using v0.2). So I've ditched it. However, the guys from Tehtri Security, posted a leet idea to Full Disclosure of using triggers (I had no idea SQLite3 could do triggers). The triggers ensure that the relevant tables get auto-truncated when written to. You can download this SQL file, and apply it to consolidated.db with the command (assuming it's in the same directory):

sqlite3 consolidated.db '.read tehtris-iphone-privacy.sql'

I've checked and applied the triggers, and they seem to be functioning (I watched the file shrink as loc data was written), and location services are working. So far so good. You can either use the backup & restore method discussed below, or if jailbroken, you can scp the file off the device, apply the change and scp back, or install sqlite3 via Cydia and do it on the device.

Update 1 - Warning: This breaks location services. I didn't notice because I spoof my location to a bunch of apps, whoops. The specific aspect that breaks location services appears to be the use of the stub consolidated.db file. The read-only permission flags get ignored on an otherwise "correct" file. You can delete the file regularly and it won't cause any problems however. There is a jailbroken application, untrackerd, which will run a daemon to do it for you. When I get a chance, I'd like to extend the SBSettings GPS switch to delete the file too (i.e. delete consolidated.db on GPS switch on).

Yesterday, Pete Warden and Alasdair Allen released some research and a tool that showed that Apple has been collecting detailed location data since v4 of iOS in a file called consolidated.db. Apart from the worry of wtf Apple is collecting such detailed information, this file is available in the clear in all your iTunes backups, meaning any application on your computer can access it if you haven't encrypted your backups. To demonstrate that, Pete and Alasdair released a demo app that gives a scary amount of detail about your movements.

The only advice given by the researchers was to encrypt your backups. This will prevent other apps from reading the file out of them, but it won't stop the file from existing at the source. I did some quick poking and found a better solution. You can edit your consolidated.db to contain junk data, and replace it in your backups, and restore your phone. If you've got a jailbroken phone, you can also remove write permissions to the file, and it won't get updated (based on the limited testing I performed).

Here's the step by step guide:

Plug in your iPhone and let iTunes make a backup. Make sure the backup isn't encrypted, we'll do that later.
Go to your backups directory. On OSX it will be in /Users/<username>/Library/Application Support/MobileSync/Backup/ (note, there's no "s" on the end of Backup like the iPhoneTracker FAQ suggests). In Win7 it will be in \Users\<username>\AppData\Roaming\Apple Computer\MobileSync\Backup\ other windows locations are listed here. It will contain several randomly named directories, change into the one with the latest timestamp to work on your last backup.
Get hold of the iphonels.py file. Either by copy pasting from the original here, or just downloading this one.
Look for the randomly named file that maps to consolidated.ls by running the iphone-ls.py and grepping for "consolidated" e.g.: ./iphone-ls.py | grep consolidated
The value in brackets is the name of the file as it is stored in the backup folder. This name will be consistent across all your backups.
If you like, open the file up in your favourite SQLite editor and mess up the tracking values. Or to save time, you can use this one. In my file, I truncated as many tables as made sense (e.g. I didn't truncate the "versions" table), and for those which couldn't be truncated, overwrite the private data with 1's.
I then overwrite all copies of consolidated.db with the new neutered version. This is easy to do as the random file name is consistent across backups.
Next, plug in your phone, and restore your backup. Remember to re-encrypt your backups.

The problem with this approach, is that it will need to be done regularly to keep clearing out the new location data that gets written. If you have jailbroken your phone, you can take the additional step of overwriting the file on the device (it's in ~/Library/Caches/locationd/consolidated.db) then chmod'ing it to 440 to make it read-only (this doesn't work, the perms are ignored, you'd need to SetFile). I did this then tried several things such as switching the GPS on and off, reconnecting to the cell network, turning wifi on/off, turning on my GPS app, airplane mode on/off etc. and nothing updated the file (because I was spoofing my location, whoops). Although, a -journal file does get created for brief short periods, that quickly disappears (too fast for me to grab a sample, and too inconsistently for me to repeatedly force it's generation).

Todo (or if you feel like contributing): Modify the iphone-ls.py file to allow changing values, most notably the permissions (2 byte integer) to allow the backup to mark the file as read-only.

Do Not Track & AP News Registry

nospam@example.com (Dominic White) — Wed, 06 Apr 2011 00:11:27 +0200

Firefox 4 implemented the Do Not Track header. This is an option, sent via an HTTP header, to specify to a webserver that the user would like to opt-out of advertising/behavioural tracking. The news came in soon after that the AP News Registry service had implemented support for DNT. So I decided to have a quick look at what this meant. It ended up highlighting why I think DNT will never be a solution by itself, and why it's intended use may even be tenuous.

First off, I needed to know what domain and from what sites the AP cookies are set. It turns out that the service relies on the hNews microformat, and a quick google brought me to The Aspen Times. If you have a look at the source for Aspen News, you'll see some content loaded from analytics.apnewsregistry.com and apnewsregistry.com. Since "analytics" seemed to be the most likely tracking source, I made two simple HTTP requests to the URL referenced in a news story, one with the DNT header, and one without.

$ nc analytics.apnewsregistry.com 80
GET http://analytics.apnewsregistry.com/[snip] HTTP/1.1
Host: analytics.apnewsregistry.com

HTTP/1.0 303 See Other
Set-Cookie: ASP.NET_SessionId=lwymgdbt4u5go3e55al1uxwc; path=/; HttpOnly
[snip]

With DNT:

$ nc analytics.apnewsregistry.com 80
GET http://analytics.apnewsregistry.com/[snip] HTTP/1.1
Host: analytics.apnewsregistry.com
DNT: 1

HTTP/1.0 303 See Other
Set-Cookie: ASP.NET_SessionId=vy1r33ambeja03fev4e1ognw; path=/; HttpOnly
[snip]

In both cases, you can see a unique session cookie is being set. That didn't seem right. So I set up a brand new Firefox 4 profile, hit aspen news, then checked the cookies. Without DNT I saw the following cookies related to apnewsregistry:

I then set the DNT option under Preferences -> Advanced -> General -> "Tell websites I do not want to be tracked", and saw the following cookies get set:

This tells me that it's not the analytics site, but the other which is affected by the DNT header.

Does this mean the AP News Registry conforms to the intention of Do Not Track? The answer is that we have no idea. They're still dropping a unique identifier, even with DNT set. Even if they weren't dropping any, the combination of other browser attributes could prove unique enough. In the end, someone would need to perform a code review of their server-side code to make sure the unique identifiers aren't being used for tracking.

This is important, because that's the primary intention of DNT. To quote Harlan Yu when asked about this issue:

Of course, Do Not Track needs a regulatory framework with effective enforcement mechanisms. This is the ongoing policy debate in Washington, whether Congress should give the FTC authority to define and enforce DNT regulations and what these regulation look like.

But enforcement is going to be very hard if situations like the above are allowed to persist. Do Not Track needs to result in no cookies or other unique identifiers being set on the client side and an independent audit of the tracker's server side code for it to be a meaningful label that can be meaningfully "breached".

In short, I'm not saying DNT is useless, just that implemented as AP News has done it, is equivalent to an unverifiable promise. In the end, it is my belief that we need to rely on technical means *first* for provable privacy, and let ideas like DNT provide a *secondary* legislative mechanism.

In the meantime, DomCorp will be offering free "DNT audits", just send me all your codez and passwords :)

Stub Cookies

nospam@example.com (Dominic White) — Thu, 03 Mar 2011 22:40:31 +0200

This is a quick note, partially for my own purposes of memory, of an idea. I tried to hit a GoToMeeting page earlier today. I didn't need to log on, just needed some basic information. The problem was it has one of those irritating cookie detector pages. Essentially, even though it doesn't need to set a cookie, it tries to, and if it can't, redirects you to "Sorry, you don't have cookies enabled."

In those situations, you need to allow the site to set a cookie, and then remove the cookie afterwards. Add-ons like CookieSafe let you use "Temporary Permissions" but those are set for much longer than a single page request. So you end up with an unnecessary cookie, potentially used for tracking that you don't need.

The cookies it sets are:

Set-Cookie: g2mVisitor=FirstVisit%3D1299181701998%26LastVisit%3D1299185151317%26RSN%3DDEFAULT; g2mSession=SessionInfo%3D200000000028062301%253A41EA01704E81824; JSESSIONID=abcldXoZn-6ZjaEQ4q95s

What I tried, was to send a fake Cookie: header, with all three of the cookie names it was looking for, but with blank values for each. It worked perfectly. They looked like:

Cookie: g2mVisitor=; g2mSession=; JSESSIONID=

My suggestion then is that CookieManagers provide a "Stub Cookie" option, where a site that wants cookies, but doesn't need them, can think it has set the cookies, but in truth just be getting blank values. It's a quick change that should have minimal impact. I had a quick look at CookieSafe's code (I can't seem to find any contact details for the author), and I'm hoping it's as easy to implement as it looks.

Time, time, time...

GoogleSharing For Other Browsers

nospam@example.com (Dominic White) — Mon, 20 Dec 2010 00:32:37 +0200

GoogleSharing is something I've written about before, and strongly believe in. It's a way of proxying connections to unauthenticated Google services in such a way that:

Google can't work out who you are (random session cookies are used)
Google can't work out that you're using a proxy
The proxy can't see your searches (if using SSL)

However, right now it only runs in Firefox. While there are some people looking to port it to other browsers, there are some options available in the meantime, especially for mobile browsers.

The first, and most portable is to use the front-end I've previously blogged about. It's currently sitting at http://1984.za.net/. Unfortunately, this will not be encrypted, and the webserver will be able to see your searches, however, the other benefits remain, and you can use it when you're not at your computer.

The second, and which allows you to continue to use Google as normal, and works for more than just search, is a dynamic proxy.pac file hosted here. By default it gives you a working proxy.pac that will proxy *all* Google services (even authenticated ones, to be fixed) via GoogleSharing. The options are:

proxy.php - will load the default .pac which will specify a DIRECT connection, without proxy for non-Google services
proxy.php?proxy=<proxy>&port=<port> - will allow a specific proxy & port to be specified if you are being one e.g proxy.php?proxy=192.168.1.1&port=3128
proxy.php?proxy=<proxy>&port=<port>&socks - will do the same as the previous, except specify the default proxy as a SOCKS proxy

Additionally, it will blackhole Google Ads and the Facebook like button on non-webkit browsers (on webkit browsers it will ignore the blackhole, to be fixed).

This is in alpha right now, but I've been using it for a week on my iPhone (more on how to change 3G proxy settings on the iPhone later) with no major problems. Feel free to make a copy of the output and create your own proxy.pac. Any feedback would be appreciated.

Todo:

Only proxy unauthenticated Google services
Implement a blackhole that Webkit respects
Provide a full ad-blocker blacklist from EasyList as an optional extra

Killing the Evercookie - Part2 MobileSafari

nospam@example.com (Dominic White) — Mon, 18 Oct 2010 11:54:33 +0200

UPDATE: An iPhone developer has turned this into an awesome little SBSetting addon. You'll still need a jailbroken phone but can install it via Cydia.

My previous experiments in killing the Evercookie in Safari sparked similar posts describing how to do the same for Chrome and Firefox. However, my second most frequent browsing platform is my iPhone, and I thought I would investigate how Apple IOS, MobileSafari & embedded WebKit fares. It does much worse. There are two problems; the first is, any app which embeds MobileWebKit has it's own stores for normal cookies, browser cache and HTML5 storage. Even if you go to your Safari settings (Settings -> Safari -> Clear {Cookies|Cache} & Settings -> Safari -> Databases -> Edit -> (delete all present) ) and delete everything, you haven't cleared the cookies, caches & stores in the other apps (e.g. even a simple cookie set for singe.za.net in Twitter.app's embedded browser, will still exist). The second problem is that, in MobileSafari, even if you do clear your MobileSafari store, the HTML5 localStorage mechanism isn't properly cleared and the evercookie reloads itself.

To hard clear all the WebKit datastores, including normal cookies, I put the following quick script together (you'll need a JailBroken iPhone). It will iterate through all WebKit databases, including MobileSafari's and clear out the evercookie. You'll need to close (not suspend) all apps running WebKit for this to be effective (the evercookie reloads itself in seconds if they're open). Note, it produces ugly output, and prompts before you delete files, but I wanted some visibility into who is storing what where. The first run deleted over 30 cookies in various places.

#!/bin/bash
echo "Deleting evercookie locations Safari missed (see samy.pl/evercookie)"

for DIRNAME in $(find /var/mobile/Applications -maxdepth 3 -type d -print|grep WebKit); do
        #Delete HTML5 SQLite DB
        ls "$DIRNAME"/Databases/*
        rm -ri "$DIRNAME"/Databases/*
        rm -ri /var/mobile/Library/WebKit/Databases/*
        #Delete HTML5 local storage
        ls "$DIRNAME"/LocalStorage/*
        rm -ri "$DIRNAME"/LocalStorage/*
        rm -ri /var/mobile/Library/WebKit/LocalStorage/*
        #Delete normal cookies
        ls "$DIRNAME"/Cookies/*
        rm -ri "$DIRNAME"/Cookies/*
        rm -ri /var/mobile/Library/WebKit/Cookies/*
done

I know this and my previous entry are scorched earth tactics. I'm okay with that for initial work and for browsers I don't use as my primary, due to limited privacy controls. Eventually these controls will need to be built into browsers (control to prevent, visibility into what is set when allowed, and an ability to delete). Something I can see all browsers (possibly except Chrome, because Google wouldn't be able to make money monetising your personal details then) doing.

In short, what does Apple need to do to fix this? They first need to update the MobileSafari preferences to properly clear HTML5 local storage. Currently, there is no way to do this without jailbreaking. Second, they need to add the ability to clear the history/cache/cookies/HTML5 storage for all apps with an embedded WebKit browser. How they do it is up to them, but a central option to clear all would be a good start.

Update: Clarified what the two separate problems are, and added a section on what Apple should do to fix. Also, hello to all the Slashdot and ThreatPost readers :)

Killing the Evercookie

nospam@example.com (Dominic White) — Wed, 13 Oct 2010 06:56:00 +0200

(Hi Slashdot & The Register readers. Make sure to check the 2nd part on killing iPhone Evercookie's too)

Samy Kamar recently released his tool, evercookie. This uses multiple persistent data stores to set unique identifiers that can be used to identify your browser to a website. While my default Firefox browsing setup is safe against it, I noticed that the "disposable" Safari instance I used was not. I sometimes use a clean Safari instance to test or access things the tinfoil on my Firefox does not let me. After each use I reset everything in it. However, I noticed that evercookie would persist. Here's how to delete it and others using the same mechanisms for Safari on OSX 10.6 (working out the same for other browsers/OS' isn't too difficult):

When the evercookie is created, is shows as existing in the following locations (note: just visiting the site sets up some of the evercookie containers):

userData mechanism: undefined
cookieData mechanism: 362
localData mechanism: 362
globalData mechanism: undefined
sessionData mechanism: 362
historyData mechanism: undefined
pngData mechanism: 362
etagData mechanism: 362
dbData mechanism: 362
lsoData mechanism: 362

If I reset Safari, but don't restart it, the cookie persists in these four locations. The force-cached PNG uses an RGB value as the identifier and is only cleared after a reset and restart:

pngData mechanism: 362
etagData mechanism:
userData mechanism: undefined
cookieData mechanism: undefined
localData mechanism: 362
globalData mechanism: undefined
sessionData mechanism: null
historyData mechanism: undefined
dbData mechanism: 362
lsoData mechanism: 362

However, even a reset and restart leaves us with the two HTML5 localData and SQLite locations, and a flash cookie:

pngData mechanism: undefined
etagData mechanism:
userData mechanism: undefined
cookieData mechanism: undefined
localData mechanism: 362
globalData mechanism: undefined
sessionData mechanism: null
historyData mechanism: undefined
dbData mechanism: 362
lsoData mechanism: 362

To this end, I wrote a small script (which Bernd turned into a GUI app for OSX) which will remove these and other cookies:

cat evercookie-kill.sh
#!/bin/bash
echo "Deleting evercookie locations Safari missed (see samy.pl/evercookie)"
rm -r ~/Library/Safari/Databases/*
rm -r ~/Library/Safari/LocalStorage/*
rm -r ~/Library/Preferences/Macromedia/Flash\ Player/\#SharedObjects/*

Running the script while Safari is running will have no effect. For it to work fully, you will need to reset Safari, exit, then run the script. This will clear out all the locations currently implemented in evercookie. While checking these locations, I was surprised to find data from all sorts of other sites, hence the removal of "*", but you can replace it with "samy.pl" if you want to target Samy's evercookie specifically (note, that's not the same as someone else's site implementing the evercookie). While the flash cookies had a large number of sites, there were a couple (cnn, foxnews, twitter and a few others I can't remember) using the HTML5 locations.

Online Privacy, a teaser

nospam@example.com (Dominic White) — Sun, 10 Oct 2010 22:58:02 +0200

I'll be speaking at IS' Internetix 2010 conference and this was originally posted there. I was asked to put a blog post together as a teaser for my talk.

Privacy is dead, or so the common wisdom says. But that can't be true. Centuries of philosophy tell us that it's vital for our development and existence as human beings. As a trite example, try imagine having a truly intimate conversation with your partner while knowing someone else was listening. But that's not what I want to talk about here. If you want to have that conversation, start with this paper.

What I do want to talk about is how much privacy invasion we allow in our daily online activities. But first let's talk about Google. Google is a hugely successful corporation. What's more, people *think* it is a hugely successful corporation, and so attempt to copy their methods and business models. A quick look on Amazon for business books about Google shows 1660 books, while a search for the same on Yahoo shows 635. If that's not enough for you, then try and imagine another way of monetising online content other than through advertising (unless you're Rupert Murdoch). Google is so exemplary of the online business model, that the next best example, Facebook, provides little meaningful differentiation when it comes to privacy invasion. So what is this miraculous, often copied, business model; wholesale personal data collection, correlation & aggregation used to better target ads.

You don't have to have thought very hard to have realised by now that Google's services aren't free. Sure, they don't cost you money, but Google needs to make money. They do that by collecting data about you, and using it to better target advertising at you. This doesn't worry most people, as long as that data isn't handed over to creepy government agencies or personal stalkers or allowed to be individually perused by Google employees. While all of those things are possible, and warrant enough worry in themselves, the truth is you don't really know what data is being collected, where it's being exposed, in what form and to who. Let's take Axciom, a company who's, until recently, sole purpose was to buy data about people and sell it back to marketers. How much do they know about you, who are they selling it to and with what controls?

So how does the average website leverage this world of advertising-based monetary rewards? They just include a few pieces of code into their website. This code can do all sort of things, from tracking you around the web to build a behavioural profile, interrogating your browser and computer for information, or just keeping a record of who and where you are. The problem is that sliding in these third party web-sources is easy to do, and there are many rewards to be had, both monetary and functional. The former is the primary driver, the filthy lucre of ad-click monetisation, while the latter gives you all sorts of ways to increase the loot (think analytics). Let's take an example site: memeburn.com. I've chosen this at random, not to single them out, because everyone is doing it. To view the kif content at memeburn, your browser only needs to communicate to the http://memburn.com/ webserver. However, when we hit the front page, before loading anything fancy like JavaScript, content is pulled from two other domains: afrigator.com (from the unsubtly named /track/ directory) and myscoop.co.za. After loading JavaScript, content is pulled from 34 domains in total (6 appear to belong to memeburn, 8 belong to Google, 6 to Facebook and 6 to Twitter with 10 others distributed among others). By way of comparison, a load of techcrunch.com hits 39 domains, this certainly isn't something memeburn only is engaging in. By just visiting the site, before we've even moved the mouse or read an article your browser has contacted, been poked, prodded and queried by dozens of services, none of which actually present you with the content you're there for, and with whom, for the most part, neither you nor the site have any contractual relationship with. Sure, they're privacy policies will state that they only give your information to business partners, aka anyone who will give them money for it. As we move up the stack and start using the web applications, the number of services and amount of information collected only increases. Come to the talk to see how something as simple as your search data speaks volumes about you. Now multiply that by every page you visit, every day you use the internet, over a lifetime; that's a lot of data. If you don't think it says anything about you, come to the talk to have your opinion changed.

The big problem is with finding solutions. For you to individually protect yourself against the multiple methods of data collection is currently a huge burden. If you ever want to see just how big, come and check out my browser setup. The balance needs to be tipped, with companies bearing more of the costs of privacy, instead of it all resting on the consumer. In the meantime, if you're a web developer, start thinking about whether you really need to hand so much of your users' data over to third parties. At the very least, it will result in faster page loads. In the meantime, while us consumers wait for privacy legislation to catch up, there is some help in the form of browser add-ons. For example, AdBlock (Chrome, Firefox) will cut out a lot of the third parties, and not impact your ability to see the content (i.e. no cost to you), in fact things look cleaner and load faster. This is the only way we can vote with our money and attempt to force a change in just how much privacy invasion needs to occur for something as uninteresting to the worlds problems as targeting advertising.

A Response to Paul Rubin's "Ten Fallacies About Web Privacy"

nospam@example.com (Dominic White) — Tue, 31 Aug 2010 20:40:00 +0200

Paul Rubin had a piece in the Wall Street Journal describing 10 fallacies of Web Privacy. This is my response, and the start of my blogs official "privacy" category.

1) Privacy is free. Many privacy advocates believe it is a free lunch‚ - that is, consumers can obtain more privacy without giving up anything. Not so. There is a strong trade-off between privacy and information: The more privacy consumers have, the less information is available for use in the economy. Since information helps markets work better, the cost of privacy is less efficient markets.

There are two problems with this statement. The first counter-fallacy is the idea that more information, any information, makes markets work better; that just isn't true. Take a simplistic example of someone who signs up for a golf magazine and is then spammed by so many adverts for golfing gear that they train their spam filter to block it. The company got some information, used it inappropriately, leading to the client making fewer purchases for no better reason than too much advertising. What's needed is a mechanism for the right (i.e. necessary to enable consented activities in the consumers interest) information to get to the right companies (i.e. not spammy affiliates or surveillance groups). This is exactly what privacy advocates are working for currently; what controls can enforce this rather than the overly permissive current state.

The second problem is that the cost goes both ways. Right now a consumer has to spend the effort in enforcing their privacy. The current technical complexities of, for example, ensuring cookies for services you use, are not used to correlate your identity across affiliate sites, is high and only performed by the few who understand the implications and care enough to do something about it. Thus, the cost (understanding, technical ability, actual work required) is too high for many consumers to reasonably enforce their own privacy. This cost needs to shift to companies in order to achieve a more reasonable middle ground.

2) If there are costs of privacy, they are borne by companies. Many who do admit that privacy regulations restricting the use of information about consumers have costs believe they are born entirely by firms. Yet consumers get tremendous benefits from the use of information.

Think of all the free stuff on the Web: newspapers, search engines, stock prices, sports scores, maps and much more. Google alone lists more than 50 free services‚ - all ultimately funded by targeted advertising based on the use of information. If revenues from advertising are reduced or if costs increase, then fewer such services will be provided.

I don't see fewer services, in return for more control of what information is collected and how it is used, as a poor trade off i.e. it's a cost most consumers would be willing to bear. If anything, efficiencies may be generated in the market with weaker services that exist purely as third party data collection points (e.g. spammers, personal data warehouses (e.g. Axciom) and other organisations that end up with data from our primary service providers that we would prefer didn't) being weeded out. It would be hard to argue that more privacy would result in all information supported services disappearing.

3) If consumers have less control over information, then firms must gain and consumers must lose. When firms have better information, they can target advertising better to consumers‚ - who thereby get better and more useful information more quickly. Likewise, when information is used for other purposes‚ - for example, in credit rating‚ - then the cost of credit for all consumers will decrease.

Giving consumers more control of their information does not lead to firms having worse information. If anything the firms are likely to have access to higher quality information and avoid many of the poor inferences current data sets lead to (e.g. googling for "bomb making" means you're a terrorist). The key quality differentiator is that a consumer can target the intended use with the right information, due to the disclosure of intended use by the firm when gathering consent. The current situation is more akin to my bank knowing my shoe size, just because they can, and sharing that with affiliates; rather than the bank collecting credit rating specific data for their own calculations.

4) Information use is "all or nothing." Many say that firms such as Google will continue to provide services even if their use of information is curtailed. This is sometimes true, but the services will be lower-quality and less valuable to consumers as information use is more restricted.

For example, search engines can better target searches if they know what searchers are looking for. (Google's "Did you mean . . ." to correct typos is a familiar example.) Keeping a past history of searches provides exactly this information. Shorter retained search histories mean less effective targeting.

Once again, we have the counter fallacy: "more information == higher quality service" coupled with a misunderstanding of what sort of control privacy advocates are looking for.

First, a large amount of information currently collected is not collected for direct use with that service; while Google search does collect your search term, it also correlates that use with other services. If Google were to say "we collect exactly this information for this specific purpose, if you don't like it leave" that would be a huge improvement over the current vague statement of "we collect some information, we share some of it, if you don't like it leave, but we'll still try to track you around the web."

Second, privacy advocates, for the most part, have no problem with Google collecting search terms and using that data for the typo correction example above. The problem is strongly associating those terms with an identity and then barely anonymising them. It would be quite possible for Google to collect the search terms and provide typo correction without knowing UserX searched for that term.

5) If consumers have less privacy, then someone will know things about them that they may want to keep secret. Most information is used anonymously. To the extent that things are "known" about consumers, they are known by computers. This notion is counterintuitive; we are not used to the concept that something can be known and at the same time no person knows it. But this is true of much online information.

This "fallacy" is phrased incorrectly. It should be "If consumers have less privacy, then someone *could* know things about them they may want to keep secret." This is not a fallacy. Sure, for the most part there isn't a sweaty sysadmin reading each of my Yahoo mails (although research by others suggests there may be), but if a sysadmin/private investigator/government organisation wanted to they could. If the information is stored and identified then at some point someone will want to consume it. My experience in information security tells me that you can't provide perfect protection, and as the Saudi/RIM lawful intercept saga indicates, gov pressure to be able to violate your privacy/secrecy/confidentiality wins. As the Google/China hack indicates, lawful intercept gets used by the bad guys too.

What's more, the advanced data analytics performed by the likes of Facebook and Google allow additional secret information, that you may not have intentionally disclosed about you, to be discerned. In short, if the information isn't stored, it can't be compromised.

6) Information can be used for price discrimination (differential pricing), which will harm consumers. For example, it might be possible to use a history of past purchases to tell which consumers might place a higher value on a particular good. The welfare implications of discriminatory pricing in general are ambiguous. But if price discrimination makes it possible for firms to provide goods and services that would otherwise not be available (which is common for virtual goods and services such as software, including cell phone apps) then consumers unambiguously benefit.

It may be because I'm not an economist but it sounds like Rubin makes a weak point (coupled with my observation in parenthesis) here: "Differential pricing is bad (mostly to the poor), but some good could come from it (mostly to the rich), so it's okay." The way I see it, if one side has perfect information about the other, but not vice versa, then the negotiation is flawed and will not work to mutal benefit. Even if you could argue that this is not true, people who take steps to prevent their information from being collected and tagged with their identity would be in a stronger bargaining position and would benefit more than the consumers who didn't.

7) If consumers knew how information about them was being used, they would be irate. When something (such as tainted food) actually harms consumers, they learn about the sources of the harm. But in spite of warnings by privacy advocates, consumers don't bother to learn about information use on the Web precisely because there is no harm from the way it is used.

It's true, harm from privacy violations is difficult to asses. If only someone wrote a book about it providing some sort of comprehensive taxonomy of privacy harms. In short, it is very short sighted of Rubin to claim that violations of online privacy cannot lead to harm.

8) Increasing privacy leads to greater safety and less risk. The opposite is true. Firms can use information to verify identity and reduce Internet crime and identity theft. Think of being called by a credit-card provider and asked a series of questions when using your card in an unfamiliar location, such as on a vacation. If this information is not available, then less verification can occur and risk may actually increase.

The panopticon is a well understood and flawed model. Giving firms and governments all the information reduces consumer liberty and gives firms/governments all the power. There needs to be a balance; banks can't have "anonymous" banking with them, and governments can't allow "anonymous" through their borders. However, governments shouldn't be able to ask banks about all their customers because they feel like create some sort of creepy total awareness office. If anything allowing consumers more control over their information and firms/governments less control makes it easier for consumers to keep those firms/governments honest leading to a more efficient market.

9) Restricting the use of information (such as by mandating consumer "opt-in") will benefit consumers. In fact, since the use of information is generally benign and valuable, policies that lead to less information being used are generally harmful.

I'm calling wild assertion on this one. While the mass of information gathered is likely used for benign purposes, the exceptions which cause harm and the potential for this harm to occur if no controls are in place, is enough to justify their existence. That's why even though the majority of the populace don't commit crimes, we still have police for the few who do.

10) Targeted advertising leads people to buy stuff they don't want or need. This belief is inconsistent with the basis of a market economy. A market economy exists because buyers and sellers both benefit from voluntary transactions. If this were not true, then a planned economy would be more efficient‚ - and we have all seen how that works.

If Communism is to economists as Nazism is to moralists, then I'm calling Godwins Law (I know, I lose). That being said, I'm not going to defend this point, as it's a dumb one. Targeted advertising is much better than untargeted advertising. Guess what's better for the consumer? NO ADVERTISING coupled with easy ways of finding out information on products they actually want to purchase. The only reason I allow advertising (and sometimes click the ads) is for sites I want to support who use ad-revenue, for the rest, there's ad block. But I try not to let any of them profile me to offer targeted ads, yet somehow I am still fully empowered to both find products I want, research them in detail and purchase them from companies selling them.

This brings us to the end. In short, I disagree with everything Rubin says. He misunderstands that privacy advocates are looking for a balance of controls, not extremes, and makes unvalidated assertions about how information inherently leads to all sorts of good economic things. He also fails to consider abuses of information, which are the specific cases privacy advocates are trying to protect against.

Information Security South Africa (ISSA) 2010

nospam@example.com (Dominic White) — Tue, 10 Aug 2010 22:06:00 +0200

This is a cross-post from my other blogging home at SensePost.

Last week we presented an invited talk at the ISSA conference on the topic of online privacy (embedded below, click through to SlideShare for the original PDF.)

The talk is an introductory overview of Privacy from a Security perspective and was prompted by discussions between security & privacy people along the line of "Isn't Privacy just directed Security? Privacy is to private info what PCI is to card info?" It was further prompted by discussion with Joe the Plumber along the lines of "Privacy is dead!"

The talk, is unfortunately best delivered as a talk, and not as standalone slides, so here's some commentary:

We start off the problem statement describing why privacy has grown in importance. The initial reactions were based on new technology allowing new types of information to be captured and disseminated. While the example given is from the 1980s, the reaction is a recurring one, as we've seen with each release of new tech (some examples: Cameras, Newspapers, Credit Cards, The Internet, Facebook). Reactions are worsened by the existence of actors with the funding & gall to collect and collate much information to further potentially disagreeable goals (usually Governments). However, the new threat is that there has been a fundamental shift in the way in which we live our lives, where information about us is no longer merely *recorded* online, but rather, our lives are *lived* on line. It is quite possible that for an average day, from waking up to going to sleep, a significant number of the actions you perform will not only be conducted (in part) online, but that it is possible for them to be conducted using the services of one service provider. My intention is not to beat up on Google, but rather use them as an example. They are a pertinent example, as every business book seems to use them as one. The, arguably, most successful corporation of our current age's primary business model is the collection & monetisation of private data. Thus, while Google is the example, there are and will be many followers.

The next section moves into providing a definition of privacy, and attempts to fly through some fairly dry aspects of philosophy, law & psychology. We've done some entry-level work on collating the conception of privacy across history and these fields, however, brighter minds, such as Daniel Solove and Kamil Reddy have done better jobs of this. In particular, Solove's paper "I've got nothing to hide", and other misconception of privacy is a good introductory read. The key derived point however, is that private data is data with an implied access control & authorised use. Which of the implied access controls & authorised uses are reasonable to enforce or can be legally enforced is a developing field.

As the talk is about "Online Privacy" the talk moves into a description of the various levels at which private data is collected, what mechanisms are used to attempt to collect that data, and what sort of data can be gleaned. It was an academic conference, so I threw in the word "taxonomy." Soon, it will be more frequently quoted than Maslow's Hierarchy, any day now.

At each level, a brief demonstration of non-obvious leaks and their implications was demonstrated. From simple techniques such as cross-site tracking using tracking pixels or cookies, to exploit of rich browser environments such as the simple CSS history hack, to less structured and less obvious leaks such as search data (as demonstrated by the AOL leak), moving to deanonymisation of an individual by correlating public data sets (using the awesome Maltego) and finally to unintended leaks provided by meta-data (through analysis of twitter & facebook friends groups).

Finally, a mere two slides are used to explain some of the implications and defenses. These are incomplete and are the current area of research I'm engaged in.

Online Privacy, the next Battleground

Scroogle is Dead, Long Live GoogleSharing

nospam@example.com (Dominic White) — Mon, 05 Jul 2010 08:39:55 +0200

Scroogle is no longer working for the second time this year (I archived the announcement at the end of this entry). The author claims Google deliberately killed the simple interface they were using. I've e-mailed to point out that Google Custom search works fine, but relying on Scroogle isn't going to cut it anymore. The obvious solution is to use GoogleSharing. However, not all devices support it due to the requirement of a Firefox plugin; my phone for example. After meeting Moxie I discussed the idea of including a search interface with the GoogleSharing server. The idea would be that <googlesharing server>:<port>/search would provide a plain HTTP interface to search through the server.

As a precursor to this, I did some playing and realised (later than most it seems) that the GoogleSharing proxy implements a straight HTTP 1.1 proxy. A few quick lines of code, thanks to some help from Andrew Mohawk due to some gzip'ed return data trouble, and you have a very simple PHP interface to GoogleSharing:

<?php
ini_set("user_agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://www.google.com/custom?q=" . urlencode($_REQUEST['q']));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, "http://proxy.googlesharing.net");
curl_setopt($ch, CURLOPT_PROXYPORT, 80);
curl_setopt($ch, CURLOPT_ENCODING , "gzip");
$x = curl_exec($ch);

print $x;

curl_close($ch);
die();
?>

My only worry is that I've been down this road before, 5 years ago, and I want things to happen a little differently this time. What happened then is thousands of porn sites hosting malware decided that privacy enhanced search was just what their customers needed. This resulted in Google seeing several hundred malware infested links linking back to this site. The net result was that I dropped out of Google completely (with no warning or explanation of course). So my intention is not that you use my search interface. That's stupid anyway as you have no reason to trust that I'm not mining your search data. So here is a tarball that can be used to set up your own PHP front-end. You'll need a PHP-enabled webserver with curl. The readme has more.

Archived Scroogle Announcement

July 1, 2010: Here we go again...

We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we've been scraping since Scroogle started five years ago. Actually, we've been using that interface for scraping since Google-Watch.org began in 2002.

This interface (here's a sample from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn't show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google's basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched "Universal Search," which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.

It is not possible to continue Scroogle unless we have a simple interface that is stable. Google's main consumer-oriented interface that they want everyone to use is too complex, too bloated, and changes too frequently, to make our scraping operation possible.

After a lot of suggestions from Scroogle users, and a fair amount of publicity, we found a fix and Scroogle was back in 24 hours. This fix was to insert an extra parameter, &output=ie, into the search terms that were relayed to Google. The extra parameter recovered the same interface that we thought was gone forever.

Now it seems like it actually might be gone forever. Late on June 30, 2010, the results produced while using this parameter began to shift to the usual busy Google interface with ads and a left-margin sidebar. Scroogle users saw a Scroogle page that said, "Google returned no results for this search," when in fact Google returned results but our scraper was unable to deal with them. Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.

Thank you for your support during these past five years. Check back in a week or so; if we don't hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers.

— Daniel Brandt, Public Information Research, scroogle AT lavabit.com