Today morning I came office before time (It happens twice a year or less). This just because I had my morning coffee extra strong which didn’t allowed to me sleep again in such awesome winter. Hope all bosses can understand this 😦
Any How.. Greetings to all my Folks and SEO freaks.. I am thankful to Mike and team for such awesome research on search engine bots. If you like to study the follow this link.
I am shocked to know that Bots are not a lynx based crawling application but they are headless browsers. Now the question is what is a headless browser? So the headless browsers are surfing application, which works, on headless system. I have no clue that how to explain it right hence I copied from Wikipedia. A headless system is a computer system or device that has been configured to operate without a monitor (the missing “head”), keyboard and mouse. A headless system is typically controlled via a network connection, although some headless system devices require a connection to be made via RS232 for administration of the device.
So as we can consider old concept of bot as Mr. bean, then the new will be James bond. By saying James bond I literally mean “girls” and “gadgets”. So lets start with girl (You folk, stop mouth watering, I am not going to show some naughty clips). Mr. bean always act silly and behind full suited girls. Nothing to show. No extra intelligence and even less than normal “common sense”. We use to think that spiders’ aka search bots are Mr. Bean but this is no more a truth.
Now take a snip of Mr. James Bond aka 007, always smart, a lot of gadgets and sexy girls in night suite and bikinis.
Point I was discussing a girl. There is always a tendency to admire and inspect more of G-factor. Hence considering a website as a girl Mr. Bean always failed where James is pure playboy. Matt Cuts, the director always let us feel that bots are silly, they can only look after ugly angle of website (text mode) and scan only text matters. But obviously not true.
Try looking for following patents:
- A patent filed in 2004 entitled “Document Segmentation Based on Visual Gaps” discusses methods Google uses to render pages visually and traversing the Document Object Model (DOM) to better understand the content and structure of a page.
- “Ranking Documents Based on User Behavior and/or Feature Data” patent from June 17, 2004
- Google also owns a considerable amount of IBM patents as of June and August of 2011 and with that comes a lot of their awesome research into remote systems, parallel computing and headless machines for example the “Simultaneous network configuration of multiple headless machines” patent.
- Not to be left out there’s a Microsoft patent entitled “High Performance Script Behavior Detection Through Browser Shimming” where there is not much room for interpretation; in so many words it says Bingbot is a browser.
- Furthermore, Yahoo filed a patent on Feb 22, 2005 entitled “Techniques for crawling dynamic web content” which says, “The software system architecture in which embodiments of the invention are implemented may vary.
So what are these? It means from last seven years search engine bots have been able to crawl everything including Ajax and Flash. Its good for the health of web and web search. But question clicks that why search engines always hide these facts?
Actually they have said but in different terms. Smart people are there who proved me dumb. I am sad L, but the key point here is to understand what is changing. The user behavior has changed a lot on past 2-3 years and there are many robust changes with smartphones and tablet penetration. Now the SEO concept has to be revised. All black hat marketers, I am so sorry for all these and for this article please forgive me and break Mike’s headfirst.
The change will be in website architecture and usability level. Every Webmaster must know ABC of usability testing. If you want to know more then wait till me next blog. I think I wrote pretty enough to understand. Next I’ll cater in my upcoming blog… till then enjoy the winters and Christmas offers.