Say that I wasn't happy with the salary that Google is paying me. It's a good salary, but I want more. After all, my labor has already generated more revenue for the company than I am likely to make in my lifetime. I, of course, don't have to work with Google - I can quit at any time. But in reality I can't, since my competition is also willing to work for the same wages and I have to eat. So I have to play by Google's rules, even if I believe they don't pay me enough for the value I am making.

Do you have much sympathy for me?

Why, then, do you have much sympathy for Rupert Murdoch and his billion-dollar corporation?

It's an analogy, and like all analogies, it's imperfect. But it's perhaps more apt than you'd expect. You might argue that I could easily quit and get a job at, say, Yelp or Facebook. But News Corp could easily block Google and form a partnership with Yahoo or Microsoft - heck, there's rumors that they are doing just that. Or you could say that I should quit and form my own business - well, News Corp could easily go hire a crack team of 100 programmers to build their own indexing engine. Except they probably won't find the good ones, since everything indicates that Google is a better place for a computer scientist to work than News Corp.

You might also argue that I'm already paid for my labor. Well, News Corp is already paid for the their headlines and their brief teasers: they're paid when people visit their page and advertisers put up money for that privilege. They just want to be paid more. It's not all that different from me wanting a higher salary because I generate far more value to the company than I cost.

One difference is that you could work for thousands of software companies. In search there's Google and somewhere down the road Bing. Everyone else faces a near impossible road uphill. What if the only places you could work is Google or Microsoft? Do you think you would still get a fair treatment from either?

In many other industries, there really are only 2-3 major employers you could work for. My sister's choices (petroleum geologist) are basically limited to ExxonMobil, ChevronTexaco, ConocoPhillips, BP, and to a lesser extent Valero. If you're an auto assembly worker in Detroit, your choices are basically Ford, GM, and Chrysler.

My point isn't that you'll get fair treatment (you won't), it's that life's not fair. It sucks to be an employee, it sucks to be a small business owner, and it particularly sucks to be one of the above when all your counterparties are giant monopolies. But as a society, our response to most people in these positions is "Suck it up and deal - you knew what you were getting into when you took the job." News Corp has been on the winning side of the "We're the only game in town" bargain a lot - they just finally met someone who plays the game better than they do. I really dislike the idea of changing the rules of the game because one billionaire is suddenly losing, when we didn't change the game when millions of middle-class families were losing.

If you use robots.txt and shut out Google, your news source will fade into obscurity. This is obvious. The conclusion, however, is that by indexing their news sites Google is providing the news sites a service, so if anyone should be paying it's Murdoch.

'What exactly did Google steal? Very exactly, Google steal "Headlines and very brief teasers". From the perspective of copyright law, yes, it can be classified as quoting, so they can't sue google, but those "headlines and teasers" do carry news value and in some cases took large amount of resources to produce. Now, Google took it and provided link as a payoff. Is that payoff adequate? Murdoch and many news producers don't think so.'

This seems very much the same as walking past a newsstand, glancing at the headlines of assorted newspapers and magazines, perhaps flipping through one or two, to see what's happening. The stand owner is "giving away" the headlines, but how else are people going to know about the publications?

By similar logic, a newsstand owner should have to pay just to place a newspaper on display. Where has Murdoch been all those years while this robbery was occurring?

Your analogy is logical, but there's still a difference. Newsstand and newspapers agreed beforehand the terms of their cooperation, and while they don't pay just for displaying the paper, they do pay certain amount of money for each sold piece, as is was agreed. But with Google, nothing was agreed beforehand, Google just decided that it will display the part of the content and will provide link as a payoff.

But now I don't want to criticize Google, nor do I want to be Murdoch's advocate. I just notice that the newspaper critics don't look at the issue from publisher's perpective, thus making their articles a bit one-sided.

Google is indexing the full text of the stories, and using that text to help figure out when to show the headline and teaser. They might just be showing you a little bit of the story, but they are making use of the full-text of it in a big way. I think that's worth noting here.

What I don't get is why the newspapers don't shut Google down in their robots.txt and then offer up a "firehose"/API a la Twitter that anyone can get access to for a fee. Yelp, Delicious, Twitter, Flickr, these services all have APIs that can be used to incorporate their content into other services, on their terms. I know the guardian and the new york times are working on APIs, but they should offer up a full-text firehose version that a company like Google could pay a fair price for.

> What I don't get is why the newspapers don't shut Google down in their robots.txt...

That's because their real motives have nothing to do with what they are saying.

You can't assume that they want what they say they want because, of course, they are completely capable of making it happens -- as you say, with robots.txt. It's trivial and I can guarantee you that they're aware of that fact.

What they are probably angling for is some kind of legal settlement, either a lump sum or royalties for when their search results are shown, or Google uses their content to compute rank, or some kind of special treatment, a special relationship with Google.

To really remove themselves from Google would be shooting themselves in the foot, and doubtless they know it.

Or, like you said, they want Google to pay for a firehose. But why would Google pay for a firehose when the content is right there on their sites, which Google indexes anyway?

People, you've got to be more multi-layered in your thinking. I know nerds are known to be more literal than most, but you can't go through the world thinking that other people are like you. You have to compare people's actions, with what they say, and figure out what the gestalt means. And you have to question a person's motives for saying a thing.

Speaking, writing and publicizing are not just low-bandwidth means of transferring pure information.

No human communication ever is.

Newspapers should stop them from indexing and offer up this API instead, at a price.

Look, a web site is a web site.

Facebook doesn't let Google in to index all of it's data, this created an opportunity to eventually sell access to index it.

Newspapers could do the same, stop them from indexing their sites and sell them access to this firehose api, just like _every_ other data-driven startup.

No one flinches when Yelp offers and API with specific usage conditions. Newspapers should do the exact same thing.

I think that you can do this under "fair use", but "fair use" doesn't exist in all countries in the world.

And google has to obey to local laws and local copyrights and not just to US law.

The google book case is just another thing that's going to backfire at google. The ignorant attitude of google making foreign books (books written by authors where it is clearly illegal by the law do scan in the books and make those freely searchable on the net) available to all US citizens, even though the authors don't want their books to be available trough google, will only create more hate against google.

I believe that Google does obey local laws. I've certainly had to jump through hoops because of inane restrictions that the Chinese government puts on web content. A bunch of the stuff I've worked on simply isn't available in other countries because of various laws and restrictions that have to be cleared before we can launch.

Anyone who wants to can opt out.

I personally know an author who is simply delighted that google is making his (out of print) book available. I think he will even get payment if someone wants the whole book.

It should be opt in, not opt out!

You can't also steal something in a store, just because the store didn't tell you not to steal something!

Of course they can, but they don't want to. It's a basic distinction, really.