# Search Engine Market Share Worldwide - September 2020



## getopt (Oct 8, 2020)

Search Engine Market Share Worldwide - September 2020

Google     92.26%
bing     2.83%
Yahoo!     1.59%
Baidu     1.14%
DuckDuckGo     0.5%
YANDEX     0.5%









						Search Engine Market Share Worldwide | Statcounter Global Stats
					

This graph shows the market share of search engines worldwide based on over 5 billion monthly page views.




					gs.statcounter.com
				




Google states that Google Search “operates in a highly competitive environment,” _facing a “vast array of competitors” in general online search_, including Bing, DuckDuckGo, and Yahoo!.

That is what Google told the House Committee on the Judiciary, Nov. 22, 2019.

The question is, how can you trust the Google search engine if they do not get their own numbers right. Are they biased by their own Artificial Intelligence or do they just lie to avoid antitrust regulations?



			https://judiciary.house.gov/uploadedfiles/competition_in_digital_markets.pdf


----------



## a6h (Oct 8, 2020)

getopt said:


> bing 2.83%


I bet most of that stats (2.83), come from Windows 10 Start Menu search suggestions, and also search in Office 365.


----------



## Mjölnir (Oct 8, 2020)

DuckDuckGo recently started an advertising campaign here, at least in Berlin.  Slogan: _Concerned about online privacy? We've got something on that._ (my tanslation from german).


----------



## kpedersen (Oct 8, 2020)

vigole said:


> I bet most of that stats (2.83), come from Windows 10 Start Menu search suggestions, and also search in Office 365.



True. However I also imagine just typing this post on a Chrome web browser would send about 20 requests to Google. So many of these stats probably can't be trusted.

At least with DuckDuckGo almost every one was a legitimate request. Things like Google are even making it hard for them by removing them from the "known" selection on Android.



mjollnir said:


> DuckDuckGo recently started an advertising campaign here, at least in Berlin.


That is actually great to hear. These guys seem less crooked than the rest.


----------



## scottro (Oct 8, 2020)

I use startpage, which isn't included in browser default search engines.
But seriously, at least in the US, they want to break up a monopoly, do something about the ISPs and their collusion. Here in NYC, we have spectrum, which got permission to buy Time Warner, saying  we won't do bandwidth caps at least till whenever, now they're saying, well, we want permission. The only other choice is Verizon. In most parts of the country there are only one or two possible providers. Facebook, Google, and Amazon give you the choice to not use them. Not the same with the ISPs. Pity about the lobbyists--for those not in the US it's basically legalized bribery where they give money to lawmakers and call it something different than a bribe.  But seriously--THAT's the one that affects people here in the US. (Not so much in other countries, I remember in Japan when my wife needed a portable hotspot and my in laws were looking for providers, and you had a choice.)

This is old but still true, Honest ISP Ad.  (Has obscene language)

URL: https://youtube.com/0ilMx7k7mso

Again, I don't *have* to use Google, or any of the others. The blatently unfair monopolies are the ISPs which frequently prevent things such as a city providing Internet service.

A PS, as we were talking about this sort of thing in another thread. This forum seems to embed the video if I just paste a link. Ideally, I think it should just have the link and people can click it or not.  (My original post embedded it. vigole's post below showed me how to fix that.)


----------



## bjs (Oct 8, 2020)

scottro said:


> This forum seems to embed the video if I just paste a link. Ideally, I think it should just have the link and people can click it or not.



I believe there is a checkbox at the bottom of the forum editor that allows you to embed the video or just show the link...


----------



## ekvz (Oct 8, 2020)

scottro said:


> I use startpage



+1



bjs said:


> I believe there is a checkbox at the bottom of the forum editor that allows you to embed the video or just show the link...



I think he was rather getting at embedding being the default so every page of a thread where someone has posted a video by default sends data to google (if i really cared THAT much i'd obviously just block youtube.com but then less experienced user might realize this or know how to).


----------



## a6h (Oct 8, 2020)

bjs said:


> I believe there is a checkbox at the bottom of the forum editor that allows you to embed the video or just show the link...


I can't find that!


scottro said:


> I think it should just have the link and people can click it or not.



My solution:

Method 0:
`[NOPARSE]https://domain/file[NOPARSE]`

Method 1:

Insert Link (Ctrl+K)
URL:  `https://domain/file/`
Text: `https://domain/file`

Method 2:

Insert Link (Ctrl+K)
URL:  `https://domain/file`
Text: `https://domain/file/`


----------



## scottro (Oct 8, 2020)

Using vigole's method of inserting a link then typing URL: and the link seems to work. Thanks.


----------



## Mjölnir (Oct 8, 2020)

Loosely related to the genuine topic: The Atlas of the Digital World (german).  Long story short: the 7 biggest players in the digital world have more than 50% of internet usage on their sites.  The biggest four AAAF (Alphabet, Amazon, Apple, Facebook) are very efficient in keeping users inside their corporate bubble, i.e. to guide them to services owned by the same company, thus disadvantaging competitors.


----------



## fernandel (Oct 9, 2020)

Mine are:
 MetaGear, 
 Mojeek and Qwant


----------



## roper (Oct 9, 2020)

I've been running a yacy node. What the results lack in polish they make up for in spunk. I like it.


----------



## unitrunker (Oct 9, 2020)

The trouble with using non Google search engine on chrome is google still sees what you type - thanks to autocomplete.


----------



## getopt (Oct 9, 2020)

In public statements, Google claimed that “competition is just a click away.”
Is it?


----------



## a6h (Oct 9, 2020)

abc


getopt said:


> In public statements, Google claimed that “competition is just a click away.”
> Is it?


It depends on general consensus! Some people think Google is going to conquer and save the universe. They didn't heard about honourable EIC. They're busy watching Netflix.


----------



## getopt (Oct 9, 2020)

In 2018, Findx—a privacy-oriented search engine that had attempted to build its own index—shut down its crawler, citing the impossibility of building a comprehensive search index when many large websites only permit crawlers from Google and Bing.

Many large websites like LinkedIn, Yelp, Quora, Github, Facebook and others only allow certain specific crawlers like Google and Bing to include their webpages in a search engine index.  That meant that the Findx search index was incomplete and was not able to return results that were likely both relevant and good quality.

When you compare any independent search engine’s results to Google for example, they have no chance to be as relevant or complete because _many large websites refuse to allow any other search engine to include their pages_.

Why do webmasters _deny privacy-oriented search engines_ access to their sites?

Are they told to do so? Do agreements exist forcing such decisions?

And if you look at the robots.txt file from top web-sites why is the monopolist Google often favored in those ACLs over Google's competitors?


----------



## Mjölnir (Oct 9, 2020)

vigole said:


> It depends on general consensus! Some people think Google is going to conquer and save the universe. They didn't heard about honourable EIC. They're busy watching Netflix.


Well, that's not bad per se: the docudrama _The Social Dilemma_ is in the top-10 on Netflix.
_„Home-keeping youth have ever homely wits.“_ - William Shakespeare
_„Wer stets zu Haus bleibt, hat nur Verstand fürs Haus.“_ - William Shakespeare (german translation)


----------



## ekvz (Oct 9, 2020)

getopt said:


> In 2018, Findx—a privacy-oriented search engine that had attempted to build its own index—shut down its crawler, citing the impossibility of building a comprehensive search index when many large websites only permit crawlers from Google and Bing.
> 
> Many large websites like LinkedIn, Yelp, Quora, Github, Facebook and others only allow certain specific crawlers like Google and Bing to include their webpages in a search engine index.  That meant that the Findx search index was incomplete and was not able to return results that were likely both relevant and good quality.



While i understand the problem with this simply throwing in the towel is a bit weak in my opinion. As far as i know robots.txt is not even an official standard let alone support for it being legally required. Sure if you disrespect it there would likely be IP bans to follow but it's not like this can't be worked around. It's obviously not a nice thing to do or something i would really want to spend time on but if my only other option was to close up shop there wouldn't be much of a question about how to proceed.


----------



## getopt (Oct 9, 2020)

vigole said:


> Some people think Google is going to conquer and save the universe. They didn't heard about honourable EIC.


Thank you for comparing Google with the East Indian Company (EIC) aka Honourable East India Company (HEIC) in a more military context.
The British Crown needed to call them "honorable" in these times, as otherwise one might get the idea that serving in a regiment for protecting the EIC's commercial interests were more likely the repression of an exploiting occupier.

One may take comfort in the fact that even the EIC once stopped to exist at a certain point. It took the Indians to a rebellion and parts of the company were nationalized in the aftermath.

I leave it to the readers here to use crosslinked thinking in what direction global monopolits are heading in a digitalized (post)modern world.


----------



## Sevendogsbsd (Oct 9, 2020)

I try to use DDG but what I find is the results are not as comprehensive as Google. I understand Google is the "veeger" of the universe (Star Trek movie reference) but they have better search results, at least in my opinion. Probably because of market share I am guessing, plus unlimited budget...

I use Google exclusively for searches at work, on chrome, but on any browser using Google on any platform, I never sign into Google services. I know you can still be tracked even if you don't sign in.


----------



## ekvz (Oct 9, 2020)

Sevendogsbsd said:


> I try to use DDG but what I find is the results are not as comprehensive as Google.



That's sadly also my impression. Even while i an used to startpage (which isn't exactly google either) DDG still feels a bit lacking.


----------



## ralphbsz (Oct 9, 2020)

getopt said:


> Many large websites ... only allow certain specific crawlers like Google and Bing to include their webpages ...
> 
> Why do webmasters _deny privacy-oriented search engines_ access to their sites?


And I've been doing the same thing, for the last ~15 years, in effect. But as I'll explain below, you have the reasons for the denial all wrong. I look at my web server logs, and I find that a large fraction of traffic comes from crawlers. Then I look at each crawler, and make decisions. If it is a reputable search engine, I allow it access (in my cases, not to everything, I don't allow them to load pictures). If it is clearly a hack attack (looking for scripts etc.), I completely deny them. I also deny all crawling from Russia and China, because my web page is intended for family and friends, and I have no family and friends in those countries, and (for lack of language skills) I can't validate whether crawlers from those countries are reputable. If a crawler ignores robots.txt, it gets immediately blocked (by IP address, and I don't bother being surgically accurate, I typically deny the whole IP range used by the crawler's company or hosting provider).

When I say "reputable", I mean: the crawler honors robots.txt, it doesn't crawl excessively, it doesn't probe for vulnerabilities, their web site has clear instructions for webmasters about how to control crawling. In particular, I try to not block crawling by academic researchers, unless they get out of hand.

The real problem is that the crawler space is dominated by attackers, and completely incompetent want-to-be search engines. Real search engines (Google, Microsoft, Yahoo when that was still a thing) are really good at crawling, very efficient, and have minimal impact. They honor robots.txt, and typically give you feedback on what they see. The hacking attackers are obviously no good, and I block them quickly. I sometimes wonder whether some of the crawl entries where attempts at low-level DoS attacks, they were so impossible to explain otherwise. I finally gave up, and configured my robots.txt to only allow Google and Microsoft crawlers, since trying to stomp out all the evil/stupid ones was too much work. But here are two anecdotes:

Friends of mine are the founders of the (failed) search engine CUIL, which was founded by a former search engineer/manager from Google, and her husband, who was a computer scientist from IBM who specialized in semantic web. And I had to block CUIL, because their crawler was a god-awful mess: it ignored robots.txt and walked into directories it shouldn't have, it crawled a few files every minute or two even though they hadn't changed, and so on. While they were not evil, they were incompetent. Second anecdote: I used to work at IBM, and our research lab had a very high speed connection to the world-wide internet. One day I saw an enormous number of crawls of my personal web site coming from the public IP address of the lab where I worked. And clearly it wasn't a web browser, it was a crawler gone insane. First step was to block it at the IP level. A little investigation showed that it was a young researcher, trying to crawl the web looking for some form of content, except that they forgot to rate-limit the spider, and forgot to not crawl the same resource multiple times (which is admittedly hard if you are using a large distributed system), and were just generally a big mess. Fortunately, colleagues got them to stop before they got in big trouble.

So the answer to your question is: You are jumping to conclusions. I don't know webmasters who deny _privacy-oriented_ search engines because of being anti privacy. I know webmasters (including myself) who deny crawling that is damaging and pointless.


----------



## getopt (Oct 9, 2020)

ralphbsz said:


> But as I'll explain below, you have the reasons for the denial all wrong.


Isn't that just a little too ego-centric? You compare your privat homepage with sites like Facebook and Linkedin?


----------



## drhowarddrfine (Oct 9, 2020)

Sevendogsbsd said:


> I try to use DDG but what I find is the results are not as comprehensive as Google.


Sometimes I only get advertising results for some searches. That's when I switch to ddg where I'll often get what I'm searching for. 


Sevendogsbsd said:


> I understand Google is the "veeger" of the universe (Star Trek movie reference)


I get it! One of my favorites!


----------



## phalange (Oct 9, 2020)

Sevendogsbsd said:


> I try to use DDG



Me too. I set DDG to my default in FF because for most (probably all) _casual_ searches -- meaning (1) I don't type in a full url, so I'm basically using the search engine to give me the link I want or (2) I'm just pulling up fairly common stuff -- DDG is a-ok. In fact MUCH  better since it has much less garbage advertising and other distraction.

But for more esoteric questions, I sometimes have to jump to google. It seems to do better sifting through things like stackexchange for example. DDG just can't do needle in haystack searches quite as well.


----------



## ekvz (Oct 9, 2020)

drhowarddrfine said:


> Sometimes I only get advertising results for some searches. That's when I switch to ddg where I'll often get what I'm searching for.




I think this is kind of a trend that might be advantageous to alternative search engines. I remember that at some point google would give very accurate results as long as appropriate search terms/options were used. This seems to be no longer the case or at least the results are getting more and more inaccurate and polluted with random garbage that at least on the surface has little to do with what is being actually searched for.


----------



## ralphbsz (Oct 10, 2020)

getopt said:


> Isn't that just a little too ego-centric? You compare your privat homepage with sites like Facebook and Linkedin?


No, and I can't speak for why Facebook or LinkedIn deny certain crawlers. But I find it possible (even likely) that their experiences are similar to mine, just at a much larger scale. However, there is one significant difference: If one of the big content providers on the web (like Facebook or LinkedIn) denies crawling by a some indexing or search company, we can assume that contacts are made, and the decision whether to crawl or deny is made consciously.


----------



## kpedersen (Oct 10, 2020)

One reason why I try to not get too religious with search engines is because once they reach critical mass, they then start to mutate into something considerably less ethical. There is nothing stopping DDG going the same way once it gets popular. Check out this old Google ad.


----------



## a6h (Oct 10, 2020)

kpedersen said:


> Check out this old Google ad.


No weather, no news feed, ... they're trashing yahoo![!]


----------



## getopt (Oct 13, 2020)

phalange said:


> But for more esoteric questions, I sometimes have to jump to google.


Some questions to Google may just be too esoteric.








						Google is giving data to police based on search keywords, court docs show
					

Court records in an arson case show that Google gave away data on people who searched for a specific address.




					www.cnet.com
				




An application for search warrant looks like this:


			https://assets.documentcloud.org/documents/3519211/Edina-Police-Google-Search-Warrant-Redacted.pdf


----------



## phalange (Oct 14, 2020)

getopt said:


> Some questions to Google may just be too esoteric.
> 
> 
> 
> ...



Yeah that's distressing. Hard to read without feeling disgusted by it.


----------



## drhowarddrfine (Oct 14, 2020)

If it helps catch bad guys I'm all for it. Sign me up if they need help. I'm tired of people wanting to protect criminals.


----------



## kpedersen (Oct 14, 2020)

Yep, I don't really mind Google giving useful information to the police. I actually find this considerably less creepy than giving my data to random companies just to make profit (which is a lot more common).

I wonder how much money Google is making the police pay for the data


----------



## T-Daemon (Oct 14, 2020)

ekvz said:


> Even while i an used to startpage (which isn't exactly google either)


According to https://startpage.com 's statement it's exactly google:


----------



## getopt (Oct 14, 2020)

T-Daemon said:


> According to https://startpage.com 's statement it's *exactly* google:


The *difference* is this:








						Our Privacy Policy
					

We created Startpage to help you search and browse the internet privately. Not because you have something to hide, but because you have a lot to protect!




					www.startpage.com
				



Startpage can be seen as an GDPR compliant proxy to the Google search engine.

While all this looks fine on the first look, some questions remain. While some narrative is offered








						About Us
					

We want you to be able to dance like nobody’s watching!




					www.startpage.com
				




where Startpage B.V. claims to be profitable, Privacy One Group, owned by adtech company System1, acquired a majority stake in Startpage in October 2019. How does that fit in?

Startpage is paying to Google adding to the revenues of the global monopolist. But from where do Startpage's revenues come from?


----------



## a6h (Oct 14, 2020)

Here's the deal (I'm doing my joe!): They provide free service. I want their free service. They want my data. We have to price ours (data) and theirs (service). Then let the free market decide. I have no problem to share my data with companies like Microsoft (not google), to use their free services. But we need to know how much our data is worth. It is fair. Isn't it?


----------



## T-Daemon (Oct 14, 2020)

getopt said:


> The *difference* is this:
> 
> 
> 
> ...


*ekvz* was responding to *Sevendogsbsd*'s comment regarding DDG results are not as comprehensive as Google. I also was referring to the search results.



getopt said:


> While all this looks fine on the first look, some questions remain. While some narrative is offered
> 
> 
> 
> ...








						Could you provide information about your company's profits and income? - Startpage.com Support
					






					support.startpage.com
				




Info about Startpage and Privacy One/System1:





						What is Startpage's relationship with Privacy One/System1 and what does this mean for my privacy protections? - Startpage.com Support
					






					support.startpage.com
				








						Startpage CEO Robert Beens discusses the investment from Privacy One / System1 - Startpage.com Support
					






					support.startpage.com


----------



## getopt (Oct 14, 2020)

vigole said:


> But we need to know how much our data is worth. It is fair. Isn't it?


How to measure fairness? When is a price fair, agreed on in a free market?
Does marketpower add to fairness? Can monopolists be considered fair trading partners? How to monetize the risk, that your personal data are given to government agencies?


----------



## ekvz (Oct 14, 2020)

T-Daemon said:


> According to https://startpage.com 's statement it's exactly google:
> 
> View attachment 8597



I know. It's not really true though and to be honest i am not very happy with their claim. I'll give them the benefit of doubt but from my experience what they state in this paragraph is misleading at best. While their search results are in fact quite similar to Google they are not identical and Google itself will often times have results that simply don't exist in Startpage's data. I don't have an example right now but there are actually search terms that will yield zero results on Startpage while Google still manages to return a couple dozen and even if of those results are regularly not exactly high quality it's also not that rare for them to have the single exact hit you were looking for. I guess that paying Google for access to their search API simply doesn't mean that you get to clone it even if Startpage's description seems to suggest that.


----------



## scottro (Oct 14, 2020)

I haven't really  investigated, just started <heh> using it on a friend's recommendation. I did find at the time that a google link had about 8 lines of what I assume was tracking info, whereas the startpage link was just to the page or image itself.


----------



## T-Daemon (Oct 15, 2020)

ekvz said:


> I know. It's not really true though and to be honest i am not very happy with their claim. I'll give them the benefit of doubt but from my experience what they state in this paragraph is misleading at best. While their search results are in fact quite similar to Google they are not identical and Google itself will often times have results that simply don't exist in Startpage's data. I don't have an example right now but there are actually search terms that will yield zero results on Startpage while Google still manages to return a couple dozen and even if of those results are regularly not exactly high quality it's also not that rare for them to have the single exact hit you were looking for. I guess that paying Google for access to their search API simply doesn't mean that you get to clone it even if Startpage's description seems to suggest that.


Default settings on both search engines could have an effect on the discrepancy of the search results:

Google:


			Search Settings
		






						Startpage - Private Search Engine. No Tracking. No Search History.
					






					startpage.com


----------



## ekvz (Oct 15, 2020)

T-Daemon said:


> Default settings on both search engines could have an effect on the discrepancy of the search results:
> 
> Google:
> 
> ...



That's interesting. While i've never touched any settings on Google i actually was using an URL that should have included not filtering searches on startpage. Looking at the settings it seems like this is no longer respected. While i'd think what i experienced was also happening while "don't filter" was still applied by default i can't say this for sure as i don't know when exactly my custom URL stopped working. I've just generated a new URL and i'll keep an eye on this.


----------



## getopt (Oct 20, 2020)

In a lawsuit, filed in a federal court in Washington, D.C. today, the Justice Department accused Google of using several exclusive business contracts and agreements to lock out competition.

Such contracts include Google’s payment of billions of dollars to Apple to place the Google search engine as the default for iPhones. By using contracts to maintain its monopoly, the suit says, competition and innovation has suffered.









						What Is Happening With the Antitrust Suit Against Google? (Published 2020)
					

The suit is the first antitrust action against the company to result from investigations by American regulators.




					www.nytimes.com
				




It has long been known that Google relies on search traffic from Apple’s       popular line of phones. Google’s flagship search engine is the preset default on Apple’s Safari phone browser, meaning that when consumers enter a term on their phone, they are automatically fed Google search results _and related advertising_.

What’s new is just how central it is to both companies, and to the antitrust case.


----------

