# Something More Revelant then BSDSTATS.org



## vermaden (Jun 30, 2010)

... recalling the recent thread about BSD usage statistics:
http://forums.freebsd.org/showthread.php?t=14522 

Generally *http://bsdstats.org* is only known by BSD users (and definitely not by all of them), so even having BIG stats out there is more or less pointless.

But there is other way to 'impress' other people with BSD stats ... *http://distrowatch.com* portal. It mainly focuses on Linux distributions but it also gathers stats for BSDs and OpenSolaris/Solaris 'distributions'.

First we would need to add *www/lynx* package (it does not need x11 dependency as *www/links*):
`# pkg_add -r lynx`

Having this simple line in all machines attached to network for 24/7 will add actual BSD users/machines count to *DistroWatch* portal stats:
[CMD=""]0 4 * * * /usr/local/bin/lynx -dump -useragent="Lynx $( uname -spr )" http://distrowatch.com > /dev/null[/CMD]

Laptop users will propably have their system up and running in different times of the day/night, so it would be better to put such line into the /etc/rc.local file, so their system will 'send' stats at boot:
[CMD=""]/usr/local/bin/lynx -dump -useragent="Lynx $( uname -spr )" http://distrowatch.com > /dev/null &[/CMD]

Such _'lynx dump'_ will create that line in the logs:
[cmd=""]8.8.8.8 - - [30/Jun/2010:14:15:24 +0200] "GET / HTTP/1.0" 200 75 "-" "Lynx FreeBSD 8.0-RELEASE-p3 amd64"[/CMD]

... so it definitely works.

We can even put our favorite browser to increase its 'stats':
`% lynx -dump -useragent="Opera/9.80 (X11; $( uname -spr ); U; en) Presto/2.2.15 Version/10.00" [url]http://distrowatch.com[/url]`

... and here is the 'opera' result:
[CMD=""]8.8.8.8 - - [30/Jun/2010:14:19:21 +0200] "GET / HTTP/1.0" 200 75 "-" "Opera/9.80 (X11; FreeBSD 8.0-RELEASE-p3 amd64; U; en) Presto/2.2.15 Version/10.00"[/CMD]

Some may say _'isnt that cheating?'_, definitely not, we only appear one enter per one machine a day, and this *DistroWatch* stats are about that, _'hits per day'_ count about various operating systems.


----------



## graudeejs (Jun 30, 2010)

Frankly.... great idea....


----------



## DutchDaemon (Jun 30, 2010)

Good one, Vermaden. Those wanting to run 'links' on servers: don't use the package (which has X11 'on'), use the port: remove the X11 options (which are on by default).


----------



## vermaden (Jun 30, 2010)

DutchDaemon said:
			
		

> Good one, Vermaden. Those wanting to run 'links' on servers: don't use the package (which has X11 'on'), use the port: remove the X11 options (which are on by default).



Other sollution may be just adding the *www/links* package without dependencies:
[cmd='']# pkg_add -i ftp://ftp.freebsd.org/pub/FreeBSD/ports/$( uname -m )/packages-8.0-release/Latest/links.tbz[/CMD]

The best sollution seems to be *www/lynx* with that arguments:
`% lynx -dump -useragent="Lynx $( uname -spr )" [url]http://distrowatch.com[/url]`

... which will left this in logs:
[CMD=""]8.8.8.8 - - [30/Jun/2010:14:15:24 +0200] "GET / HTTP/1.0" 200 75 "-" "Lynx FreeBSD 8.0-RELEASE-p2 amd64"[/CMD]

Going to edit the 1st post to make it 'lynx by default'.

... we can even put our favorite browser to increase its 'stats':
`% lynx -dump -useragent="Opera/9.80 (X11; $( uname -spr ); U; en) Presto/2.2.15 Version/10.00" [url]http://distrowatch.com[/url]`


----------



## DutchDaemon (Jun 30, 2010)

Yeah, tested fetch too, but that was equally useless. Not sure what wget reports, don't use it.


----------



## mix_room (Jun 30, 2010)

ftp/wget does a wonderful job of the same task

[cmd=""]wget -O /dev/null http://server/file.html[/cmd]

Output from httpd-access.log

```
MYIP - - [30/Jun/2010:14:41:11 +0200] "GET /file.html HTTP/1.0" 301 239 "-" "Wget/1.12 (freebsd8.0)"
MYIP - - [30/Jun/2010:14:41:12 +0200] "GET /file.html HTTP/1.0" 200 1012858
```

[cmd=""]wget -O /dev/null --user-agent="`uname -spr`" http://server/file.html[/cmd]

Output from httpd-access.log

```
MYIP - - [30/Jun/2010:14:44:27 +0200] "GET /file.html HTTP/1.0" 301 239 "-" "FreeBSD 8.1-PRERELEASE amd64"
MYIP - - [30/Jun/2010:14:44:27 +0200] "GET /file.html HTTP/1.0" 200 1012858
```


----------



## vermaden (Jun 30, 2010)

mix_room said:
			
		

> ```
> MYIP - - [30/Jun/2010:14:41:11 +0200] "GET /file.html HTTP/1.0" 301 239 "-" "Wget/1.12 (freebsd8.0)"
> MYIP - - [30/Jun/2010:14:41:12 +0200] "GET /file.html HTTP/1.0" 200 1012858
> ```



But there is a question if 301 RETURN values are evaluated into stats, cause 200 for sure are, but there is no 'os data' by the 200 value with *wget*, *wget* can also be 'blocked for stats' since its not a browser.


----------



## crsd (Jun 30, 2010)

You'll need to set HTTP_USER_AGENT for /usr/bin/fetch to report something useful. From fetch(3):
	
	



```
HTTP_USER_AGENT     Specifies the User-Agent string to use for HTTP
                         requests.  This can be useful when working with HTTP
                         origin or proxy servers that differentiate between
                         user agents.
```


----------



## graudeejs (Jun 30, 2010)

crsd said:
			
		

> You'll need to set HTTP_USER_AGENT for /usr/bin/fetch to report something useful. From fetch(3):
> 
> 
> 
> ...



tested

```
$ export HTTP_USER_AGENT="Test Agent"; fetch http://www.bsdroot.lv
```

logs show

```
83.241.11.135 - - [30/Jun/2010:16:52:51 +0300] "GET / HTTP/1.1" 200 1342 "-" "Test Agent"
```

FOR *crontab -e*:

```
0 20 * * * /bin/sh -c "export HTTP_USER_AGENT=\"Opera/9.80 (X11; $( uname -spr ); U; en) Presto/2.2.15 Version/10.10\"; /usr/bin/fetch -o /dev/null http://distrowatch.com > /dev/null 2> /dev/null"
```
or /etc/crontab

```
0 20 * * * nobody /bin/sh -c "export HTTP_USER_AGENT=\"Opera/9.80 (X11; $( uname -spr ); U; en) Presto/2.2.15 Version/10.10\"; /usr/bin/fetch -o /dev/null http://distrowatch.com > /dev/null 2> /dev/null"
```
something like this


----------



## vermaden (Jun 30, 2010)

@crsd

It seems that best sollution is even in the FreeBSD's base system, thanks for great suggestion.


----------



## Carpetsmoker (Jun 30, 2010)

The distrowatch site _expects_ to be monitoring the web browser their visitors use. Not a bunch of _$X_ fanboys artificially pushing their OS/browser to the highest possible position.
IMHO this is stats poisoning ...


----------



## mk (Jun 30, 2010)

let's make botnet and change every browser to report that is run on fbsd machine.
why you want those 'hits per day' ?


----------



## itsbrad212 (Jul 1, 2010)

Out of the 8 *BSDs on bsdstats, 99.06% of them are FreeBSD or FreeBSD derivatives! :O


----------



## kdemidofff (Jul 3, 2010)

maybe integrate this in bsdstats port??


----------



## soupbowl (Jul 6, 2010)

Carpetsmoker said:
			
		

> The distrowatch site _expects_ to be monitoring the web browser their visitors use. Not a bunch of _$X_ fanboys artificially pushing their OS/browser to the highest possible position.
> IMHO this is stats poisoning ...



 I agree, but still enjoy this.



> maybe integrate this in bsdstats port??



 I agree with this and we should let the maintainers of bsdstats know about it.


----------



## fairy (Jul 6, 2010)

*vermaden*, to figure one's own User-Agent you don't have use www server
`$ true | nc -lk 3333 &`
and visit http://0:3333. It would display smth like this

```
$ fetch http://0:3333
GET / HTTP/1.1
Host: 0:3333
User-Agent: fetch libfetch/2.0
Connection: close
```
or from a browser
	
	



```
GET / HTTP/1.1
Host: 0:3333
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; en-US; rv:2.0b2pre) Gecko/20100704 Conkeror/0.9.2 Firefox/4.0b2pre
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
```


----------

