U.S. Spies plan massive data sweep of Internet

There's this current story about massive snooping into telephone conversations by the NSA. The NSA and CIA and other spy agencies are supposed to turn their efforts on targets outside the U.S. but under Bush Administration edict they've been working inside the U.S., in opposition to U.S. law. Yes, the President has been breaking the law.

The conduct of the U.S. Administration is that this is not a war on Terror, but instead a war on Personal Freedom.

Consider: US plans massive data sweep Little-known data-collection system could troll news, blogs, even e-mails. Will it go too far? (Christian Science Monitor, February 09, 2006)

The article describes a little-known system called Analysis, Dissemination, Visualization, Insight, and Semantic Enhancement (ADVISE), a research and development program within the Department of Homeland Security (DHS), part of its three-year-old "Threat and Vulnerability, Testing and Assessment" portfolio.

The project it describes is very similar to the process of search engine companies like Google, Yahoo, Technorati, etc. It's to scoop up a vast amount of data from the Internet and to draw out extra information from it. The technical phrase is "data mining" which is a practice of taking one data source, and putting it to a different use. Data mining is widely performed in businesses.

For example credit card companies perform data mining to detect fraudulent use of credit cards. e.g. they might look for your card being used to make an abnormally large purchase, or a purchase made far from your normal area of activity. And if they see it, they could give you a phone call saying "we noticed suspicious activity, did you make purchase X on date Y".

So long as the ADVISE system is collecting publicly available data, is there a problem?

The difference here between the government activity and what, e.g. Google, would do with it is: The government is looking for "terrorists", and the government has people with guns who are known to use those guns to kill people.

The problem with the government's hunt for terrorists, is they've got a rather loose definition and they make mistakes. For example in the Extraordinary Rendition stories, one was a German tourist who the U.S. agents identified as Al Qaeda linked, they kidnapped him, flew him to Afghanistan, tortured him for months, eventually realized they mistakenly identified him, and dropped him off penniless in Kosovo. And for loose definitions of terrorism, we can think of the people arrested for "ecoterrorism" where they are taking their protests of e.g. logging activities to doing property damage and whatnot. Sure, commiting property damage is illegal and they should be punished, but labeling them as terrorists is going too far.

In other words, I think it's legal to collect data that's publicly available (e.g. published on a web site) and to make secondary uses of it. If it's good enough for Google or Technorati, then it's good enough for the U.S. Government. But there needs to be oversight and measurement to ensure they don't overstep themselves.

For example, what if they made a deal with Google or other search engines to capture some of the private search query data which the search engines have on hand. That is, each time you make a search engine query, the company running your preferred search engine receives your IP address, your web browser, your operating system, etc, along with the query terms. If you've registered with the search engine (e.g. signed into your "mail" account) then the search engine can know exactly who you are.

Now, that's private data which the search engine collects. One way they use it is to further tailor your search results based on past queries you've made. But what if they began handing that data over to the government, which the government would than incorporate into this ADVISE system?

Would you get a knock on the door just because you had a hankering to learn about terrorists and did a lot of google searches about activities done by terrorists? Or you wanted to see for yourself just how easy (or not) it is to get information on making nuclear bombs?

I want to close by reminding the reader of the Total Information Awareness system. TIA is/was a Department of Defense project to that acted as an umbrella over several inter-related projects, some of which would use data mining techniques of the kind described in the CS Monitor article. While a couple minor TIA projects were shut down, it's clear the bulk of them went forward, and that the intent of the Government for several years has been to create a technologically advanced system that can effectively track every action and look for "dangerous" patterns.

The TIA existed before the September 11, 2001 events which "changed everything". The TIA existed before the Bush administration. This is just an ongoing desire by government agencies to vastly step up their capabilities to spy on everyone.

Search engines cooperating with the police

Microsoft Confirms Turning Over Search Data to Feds: This covers not just Microsoft but the other search engines as well. The story is there's an obscure court proceeding going on, under which the Department of Justice (DoJ) has subpoenad records from the search engines as to search terms. MSN, Yahoo and AOL have apparently cooperated with the request, while Google has refused to cooperate.

Let me bring into the conversation this: Re: Do Internet companies need to be regulated to ensure they respect free expression ? In that case some organization is organizing a move to require "Internet Companies" to impede cooperation with repressive regimes who want to surveil what their citizens are doing. But with this request from the DoJ, I wonder just how you want to define "repressive regime"?

And then we have Microsoft's side: Privacy and MSN Search posted by the MSN Web Search general manager.

As the eWeek article suggests, this is a very worrisome privacy issue. Suppose the government surveillance system were to know about every search query you were to make? It's easy to think you're just searching around the Internet, and the search results are very ephemeral and disappear very quickly. But when you're Googling or MSN'ing or Yahoo'ing, the search engine is able to make a record of your searches. The search engine might even remember searches you did before, the items you clicked on before, and helpfully rank the results based on what you searched for previously. This is information the search engines already collect.

It's important to note that the search engines are unable to identify YOU. Instead they know specific IP addresses, and are able to track which IP address did what. Or, in some cases the search engine installs a long-lived cookie, and with data in the cookie is able to correlate activity you do from day to day even if your ISP gives you a different IP address each day.

The MSN general manager says all they turned over was IP addresses and search queries. No personal identifying data.

It would be up to the government surveillance system to correlate IP addresses with people. This isn't easy, but is doable. For example my ISP assigns me a fixed IP address, and therefore the governmnet surveillance system might have an arrangement to query my ISP to retrieve the identity of people to which specific IP addresses are assigned.