Google and other search engines have over 30 trillion pages indexed and searchable, going up by around 10-30x every 5 years. Only around one ten-thousandth of pages are accessed by Google every month, with most clicked links only in the first 3 pages. Around 92% of links clicked from searches are within the first 2 pages, and less than one percent of searches are used past page 4. The Google index is huge, and if someone printed off every page which Google has as searchable, the stack would reach from London to the middle of Australia.
All search engines have the same problems however, as many websites have thousands of pages, but they are not coded in a Google-friendly way. Most sites on the World Wide Web have some form of password accessed pages, which are not searchable through Google. Universities, businesses, banks, and private boards are all unsearchable for the content they upload through portals. Even basic things like browser based games such as Kingdom of Loathing are unsearchable; tens of thousands of pages, but only one shows up in Google because the site is coded in PHP.
It's important to note that search engines like Bing and Google only search the 'top layer' of the internet, or the World Wide Web. Nobody knows exactly how many pages there are in the deep web or dark web, but people estimates that Google indexes between 0.04% and 12% of the total number of pages. This is a truly miniscule amount compared to the entire internet, but the searchable results are usually those relevant to consumers.
The main bulk of the internet as we know it is behind password protected barriers, and the dark web is almost equal in size. The dark web requires a special untraceable browser to access, such as Tor, and as such cannot be searched conventionally. Alot of the stuff on the dark web is either illegal, or people simply wish to remain private, so it makes sense that there is no record of it. Things like buying drugs, inciting hate, hiring people for illegal activites such as murder, and many other illegal things are banned from the internet, so the dark web is where people go to resource these things.
There are certain sites on the World Wide Web which Google intentionally does not search however. Websites can be 'blacklisted' from Google for violating their TOS, and mostly happens when people are attempting SEO or similar activities to subvert Googles intentions. Obviously, search engines do not like this, as it goes against their mantra of relevant results or advertisements, so they are put on a list to not-index.
Search engines will always give you the most relevant results for your search query, and can be customised using boolean commands (Google) or basic excepts of script (Bing).
The author does not allow comments to this entry