We Need Viable Search Engine Competition, Now | PEEBS.ORG

It’s become clear to me that we desperately need a viable competitor (or two) in the search engine space. A somewhat related thought I’ve been having is the (probably inaccurate) sensation that bringing out a viable competitor to Google may not be nearly as hard as it has appeared for the last decade.

We need competitors now. Most websites see more than 80% of their search engine traffic arriving from just Google, and this is not a good long term recipe for a vibrant internet.

Inherent Conflict of Interest
Google’s revenue model of placing paid ads next to organic search results operates under the (publicly accepted) belief that there’s a secure “Chinese wall” between the paid and organic functions. It was even more secure, some argued, because ultimately the short-term conflict between receiving revenue for rankings (paid) vs. displaying the best rankings (organic) was not a long-term conflict. Better organic results were always in Google’s interest, because these competitive results maintained their dominance and user’s trust. And so we believed. To be fair, I feel that Google does a somewhat decent job in this area, but I continue to feel that the user experience of Adwords exhibits various dark patterns (more about this here) and Google’s corporate inertia seems to be focused on a walled garden approach with G+ and Android. Lets just say that I’m no longer going to blindly trust Google in the face of a worrying conflict of interest that’s central to their most valuable product. Declining empires under siege are the ones you have to be careful of, after all.

Vulnerabile to Manipulation
Is there anything worse than “SEO”? The very idea of this industry, filled with people whose sole job is to attempt to manipulate Google is bad enough, but the fact that “black hat” SEO can produce material gains is genuinely worrying. Having had to clean up a mess created by a black hat (who insisted he wasn’t) and now in the middle of another mess of toxic back links that may or may not be generated by a competitor, the whole thing is just annoying, wasteful, and embarrassing for Google. I get that they’re trying to clean this up with Penguin and Panda and the various versions therein.

Arbitrary and Corrupt
When RapGenius violated Google’s SEO guidelines, they were only caught due to a public revelation on Hacker News, then immediately penalised by a human (to compensate for where their algorithm failed), then they were permitted to communicate directly with google to discuss ways out of this mess. Not it appears they’ve been fast-tracked back into the listings, albeit at somewhat of a disadvantage.

All aspects of this rub me the wrong way -

Google is making arbitrary rules on how sites should behave, because they have a monopoly. If they didn’t have a monopoly, they might not be able to make these arbitrary rules, and others might not follow them. Google needs these rules, because Google’s rankings are apparently trivial to game. Build a ton of links and make sure you don’t over-optimise your link text. That’ll do it for most key phrases, apparently, as long as you’re not completely obvious. There’s a clear incentive for “Bad Guys” to win using“Bad Ways”, that penalises good sites just trying to get on with business. Does anyone actually believe that the ridiculously obvious, poorly written link farms that Google catches periodically are the only examples out there? Smarter people doing a better job are gaming google all the time, and it appears to be getting worse. Google feels the right to at any time, and with zero due process, transparency, or appeal, to manually penalise sites who successfully ignore their rules yet exhibit a high ranking. This is not transparent, fair, or reliable. It is scary for legitimate businesses, and this kind of instability should not be the norm, but it is. The only organisations or individuals who can actually engage with Google over a penalisation or problem in any meaningful way are Silicon Valley favourites or companies backed by influential VCs, or [insert some other not-avaible-to-the-public recourse here]. This is the definition of corruption.

We Need A Competitive Alternative
Competition could provide a healthy response to many of these items. I don’t think regulation is the answer, but it may become one if these trends continue and intensify. A different revenue model could remove the conflict of interest, a better or different algorithm could be less prone to manipulation, and a search engine that prided itself on a transparent and efficient arbitration process for disputes with regards to rankings could win users trust. Of course, Google could also work on these problems themselves, but it seems like they’re more or less happy with the current state of affairs.

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype, and have missed the enormous opportunity that exists right in front of our noses. Does the next search engine have to be as big, involved in as many things, employ as many people, and fight on the same footing to be accomplish the goal of providing a counterpoint to Google?

Time will tell.