I know Cuil is the poster-boy of how to suck at launch but it really is a pity they didn't succeed. With Bing powering Yahoo there are really only 4 players in the English language index space, Google, Bing, Gigablast and Blekko. Add in the non English ones Yandex and Baidu and there really is a lack of companies to compete with.
Its because of this I want DuckDuckGo or another company to eventually do its own deep indexing to avoid the very singular view of the web we are presented with.
That's only the first portion of the problem, and in my opinion not the most difficult. Anyone can write a web crawler since its essentially,
while(links) {
get(link)
}
With a little xargs or parallel magic + wget you can suck down billions of pages with little effort.
There is more to it then that to keep things fresh but its not that hard to just kick it off now and then. The thing that's difficult is taking that data and turning it into an index able to span multiple machines.
If it was that easy DuckDuckGo or someone else would have taken that data and done something with it by now.
Had a play then and the results are quite good. Using a sample search for viagra on DDG, Google, Bing, Yandex, Gigablast and Blekko it seems to have less spam in comparison to all but Blekko.
That is an interesting query to test for spam. Now that I think about it though, I could see how that could be a start for a pretty excellent benchmark for filtering, given the ratio of viagra-to-not-viagra spam in my spam box.
Its always been my goto for working out if a search engine has any credibility or not. You do get odd looks when you have 5+ windows open all searching for viagra in the office though!
Me either, if you're not familiar with it this part is worth reading:
One Cuil = One level of abstraction away from the reality of a situation.
Example: You ask me for a Hamburger.
1 Cuil: if you asked me for a hamburger, and I gave you a raccoon.
2 Cuils: If you asked me for a hamburger, but it turns out I don't really exist. Where I was originally standing, a picture of a hamburger rests on the ground.
3 Cuils: You awake as a hamburger. You start screaming only to have special sauce fly from your lips. The world is in sepia.
4 Cuils: Why are we speaking German? A mime cries softly as he cradles a young cow. Your grandfather stares at you as the cow falls apart into patties. You look down only to see me with pickles for eyes, I am singing the song that gives birth to the universe.
5 Cuils: You ask for a hamburger, I give you a hamburger. You raise it to your lips and take a bite. Your eye twitches involuntarily. Across the street a father of three falls down the stairs. You swallow and look down at the hamburger in your hands. I give you a hamburger. You swallow and look down at the hamburger in your hands. You cannot swallow. There are children at the top of the stairs. A pickle shifts uneasily under the bun. I give you a hamburger. You look at my face, and I am pleading with you. The children are crying now. You raise the hamburger to your lips, tears stream down your face as you take a bite. I give you a hamburger. You are on your knees. You plead with me to go across the street. I hear only children's laughter. I give you a hamburger. You are screaming as you fall down the stairs. I am your child. You cannot see anything. You take a bite of the hamburger. The concrete rushes up to meet you. You awake with a start in your own bed. Your eye twitches involuntarily. I give you a hamburger. As you kill me, I do not make a sound. I give you a hamburger.
6 Cuils: You ask me for a hamburger. My attempt to reciprocate is cut brutally short as my body experiences a sudden lack of electrons. Across a variety of hidden dimensions you are dismayed. John Lennon hands me an apple, but it slips through my fingers. I am reborn as an ocelot. You disapprove. A crack echoes through the universe in defiance of conventional physics as cosmological background noise shifts from randomness to a perfect A Flat. Children everywhere stop what they are doing and hum along in perfect pitch with the background radiation. Birds fall from the sky as the sun engulfs the earth. You hesitate momentarily before allowing yourself to assume the locus of all knowledge. Entropy crumbles as you peruse the information contained within the universe. A small library in Phoenix ceases to exist. You stumble under the weight of everythingness, Your mouth opens up to cry out, and collapses around your body before blinking you out of the spatial plane. You exist only within the fourth dimension. The fountainhead of all knowledge rolls along the ground and collides with a small dog. My head tastes sideways as spacetime is reestablished, you blink back into the corporeal world disoriented, only for me to hand you a hamburger as my body collapses under the strain of reconstitution. The universe has reasserted itself. A particular small dog is fed steak for the rest of its natural life. You die in a freak accident moments later, and you soul works at the returns desk for the Phoenix library. You disapprove. Your disapproval sends ripples through the inter-dimensional void between life and death. A small child begins to cry as he walks toward the stairway where his father stands.
Cuil seems like a case study in launching too soon. There was nothing wrong with their offering, as such, but they launched before it was ready and paid the price. Shame.
"Offering" was probably the wrong word. What I meant was that there was nothing wrong with what they were trying to do, but that they launched too soon, and not working is part of that rather than an intrinsic flaw in their mission.
"... there was nothing wrong with what they were trying to do ..."
you're right, as an earlier post suggests it was half-baked & released much too early. How much of the web surface has google indexed compared to what is there?
From what I remember of the relevancy they had multiple machines each serving a different set of results.
The the machines with the best results for your keyword went down the front end just displayed what it had - which often was unrelated stuff from all the other machines that did reply.
Or in other words the lack of relevancy was due to the overload, not in principle due to their search algorithms.
> Cuil seems like a case study in launching too soon.
Nowhere was this more apparent than with "Cpedia" - a project they made available to the public at some point after the main launch. What it was supposed to be was an online encyclopedia (like Wikipedia) based on data automatically collected from their web crawler. What it ended up being was a gigantic pile of Markov spew; ridiculous, rambling, often incorrect, and useful to no one.
So Google have bought up a bunch of patents for search UI. Presumably Google won't use any themselves -- they're just storing them up as legal weapons.
Its because of this I want DuckDuckGo or another company to eventually do its own deep indexing to avoid the very singular view of the web we are presented with.