# New hardware database for BSD-systems



## aponomarenko (May 30, 2020)

A new database of supported hardware for BSD systems from the creators of the Linux-Hardware.org has been opened. Among the most popular features of the database are the search for drivers for devices, operability tests, anonymization of collected system logs, and statistical reports. The options for using the database are diverse - you can simply list all the devices on board, you can send logs to the developers to help fix a bug, you can save a “snapshot” of the current state of the computer for the future, so that you can compare with it in case of system failure, etc.

Search for drivers is performed with the help of the lists of supported devices extracted from all FreeBSD, OpenBSD and NetBSD kernel versions.

As for Linux systems, the database is replenished using the hw-probe program (version 1.6-BETA was released specifically for BSD). This program allows you to abstract from the differences between BSD-systems and display a list of devices in a single format. Recall that, unlike Linux, in BSD systems there is no single way to list PCI / USB and other devices. In FreBSD, pciconf / usbconfig is used for this, in OpenBSD, pcidump / usbdevs, and in NetBSD, pcictl / usbctl.

Among the tested supported systems: FreeBSD, OpenBSD, NetBSD, MidnightBSD, DragonFly, GhostBSD, NomadBSD, FuryBSD, TrueOS, PC-BSD, FreeNAS, pfSense, HardenedBSD, FuguIta, OS108 (if your system is not listed, then please let us know).

Please participate in BETA testing and replenishment of the database. This will greatly help the project at an early stage. Instructions for installation of the database client program and creating your hardware probe: https://github.com/linuxhw/hw-probe/blob/master/INSTALL.BSD.md

*UPDATE 1:* the telemetry package can now be installed by `pkg install hw-probe` and executed by `hw-probe -all -upload`.
*UPDATE 2: *the dump of the database is now available in the GitHub repository.


----------



## unitrunker (May 30, 2020)

The github page advises this:

`curl -s https://raw.githubusercontent.com/linuxhw/hw-probe/master/hw-probe.pl | perl`

This is very bad form.


----------



## ralphbsz (May 30, 2020)

There is a giant fundamental problem: you ask people to take a large piece of code (~18K lines of pretty dense perl), and run it as root, knowing full well that the results will be uploaded. That requires a level of trust that is unrealistic. Matter-of-fact, if I did this on any computer owned by any of my employers (current or past), I would get fired - on the spot. Matter-of-fact, I strongly suspect that running this script at work would cause my phone to ring within a few minutes, because my employers typically have reasonable privacy mechanisms. So your results will be biased towards (a) people who don't care about security, and (b) computers that are either privately owned, or installed in places where they don't care.

On the other hand, I understand why you are doing this, and it makes perfect sense. What you need to do is to get trust, and demonstrate to the user a chain of trust. Let's begin with: Associate yourself with an organization that has a sterling reputation ... which github isn't (anything can be stored on github). On the web site and in the script, identify very clearly who you are (names, locations), and make an argument that you are trustworthy. As far as I can see, the web site has no identifying information whatsoever. By using whois, all I can see that it is registered to an anonymous organization in Russa, not a country that has a sterling reputation in computer security circles. Looking at the author of the script, I see that they are employed by an organization that's incorporated in Switzerland, headquartered in Singapore, and staffed with Russians, again something that does not create a warm and fuzzy feeling. Now I happen to know what your employer A... is (and I don't have any reason to believe that they are evil), and I think you are trustworthy, and there is in reality nothing wrong with Switzerland, Singapore, or Russia, but: Guilt by association. You need to do a much better job demonstrating that you are of good will to people, since otherwise, all the alarm bells will go off.

Having said that ... let's test the script (I know how to sandbox a machine):

```
> mkdir foo
> ./hw-probe.pl -save foo
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LC_ALL = (unset),
    LC_COLLATE = "C",
    LANG = "en_US.utf-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
```
Wrong, my locale is perfectly supported and installed. Yes, I know, that's not your code, that's perl doing it. But you are responsible for the cosmetics of your product.

So try again as root:

```
# ./hw-probe.pl -save foo
ERROR: can't access '/root/HW_PROBE/LATEST/hw.info', please make probe first
```
I was expecting that the program would just run when run with minimal arguments ... instead of asking me to make a probe first, it could just do it. Also, the error message could be clearer, perhaps: "Before saving the results, you have to make a probe. The probe is usually stored in /root/..., and I can't find it. Please make a probe first, with the command ...".

So try it with -probe:

```
./hw-probe.pl -probe
ERROR: 'dmidecode' package is not installed
ERROR: 'hwstat' package is not installed
ERROR: 'lscpu' package is not installed
TIP: install missed packages by command (execute it by adding `-install-deps` option):

     pkg install dmidecode hwstat lscpu
```
No, those packages were never installed before. I have never needed them. Asking me to install packages is a pretty heavyweight step ... what if they break something (yes, I've had smartctl crash my computer before!). And offering to install packages for me, while a good idea from one viewpoint, is also quite troubling.

And this is where I stopped. Honestly, I don't think this will be going anywhere. Which is sad, it would be nice to have a good database of hardware configurations that people have had good experiences with.


----------



## Jose (May 31, 2020)

I will not run this on any of my systems, for the reasons Ralphbsz explained so well.

Break it up into two steps: One runs the probe and generates a plain text report. Another uploads the report to your servers. Better still, allow me to upload the report using a standard tool like curl(1).

This would give me the chance to review and edit the report to make sure no details I don't want leaked are uploaded. Right now I'm expected to trust that you will do this correctly.


----------



## aponomarenko (May 31, 2020)

The program is now in ports: https://www.freshports.org/sysutils/hw-probe/

Please install by:


```
cd /usr/ports/sysutils/hw-probe
make install
```




unitrunker said:


> The github page advises this:
> 
> `curl -s https://raw.githubusercontent.com/linuxhw/hw-probe/master/hw-probe.pl | perl`
> 
> This is very bad form.


----------



## aponomarenko (May 31, 2020)

You can create the probe in three steps:

1. Collect data:

`hw-probe -all`

2. Verify & edit:

`ls /root/HW_PROBE/LATEST/`

3. Add to database:

`hw-probe -upload`



Jose said:


> Break it up into two steps: One runs the probe and generates a plain text report. Another uploads the report to your servers. Better still, allow me to upload the report using a standard tool like curl(1).
> 
> This would give me the chance to review and edit the report to make sure no details I don't want leaked are uploaded. Right now I'm expected to trust that you will do this correctly.


----------



## ralphbsz (May 31, 2020)

While your suggestions are going in the right direction, they still leave many concerns unresolved.


Jose said:


> One runs the probe and generates a plain text report.


That's still thousands of lines of code that need to be run as root. I will simply not download code to be downloaded as root from a source that's not 110% trustworthy. Sure, I need to download the kernel ... but I trust Kirk and his friends. Why? Because I've known Kirk and his friends for decades, and I think I know what makes them tick, what rewards they work for. In this case, I get code from an opaque source, without clear chain of trust.

And: How are you going to make sure that this first program really only makes a list of installed hardware? It could clandestinely transmit information. It could install trojan horses. It could damage data. It could ... do so many things. Really large organizations might be able to validate that this software is "clean" by going a line-by-line code inspection. For 10K lines, that's going to take hundreds of many hours.



> Another uploads the report to your servers. Better still, allow me to upload the report using a standard tool like curl(1).
> 
> This would give me the chance to review and edit the report to make sure no details I don't want leaked are uploaded. Right now I'm expected to trust that you will do this correctly.


The OP had said that the report will be anonymized, by having the top few words of SHA-encoded data. You will not be able to understand those in the hex dump. It won't say "Intel FooLake CPU, 512gig RAM, and LSI disk controller model 12345", it will say "0xABCD, 0x4321, and 0x3579". And if we report the hardware in cleartext (like I did above), it will give away many secrets. Imagine you find that company X is buying all its disk drives from Seagate, in spite of the fact that it claims in press releases to have entered into an exclusive agreement with WD. That would be fascinating, or perhaps lawsuit worthy. There are so many more examples about information leakage from this preventing it, not even worth discussing.


----------



## aponomarenko (May 31, 2020)

First of all, thanks for such a great review. I agree with everything you said.



ralphbsz said:


> There is a giant fundamental problem: you ask people to take a large piece of code (~18K lines of pretty dense perl), and run it as root, knowing full well that the results will be uploaded. That requires a level of trust that is unrealistic. Matter-of-fact, if I did this on any computer owned by any of my employers (current or past), I would get fired - on the spot. Matter-of-fact, I strongly suspect that running this script at work would cause my phone to ring within a few minutes, because my employers typically have reasonable privacy mechanisms. So your results will be biased towards (a) people who don't care about security, and (b) computers that are either privately owned, or installed in places where they don't care.



This is why we remove ALL private info from collected logs, that can identify you or your hardware devices. The BETA version is released with the purpose to fix all possible leaks of sensitive information.



ralphbsz said:


> On the other hand, I understand why you are doing this, and it makes perfect sense. What you need to do is to get trust, and demonstrate to the user a chain of trust. Let's begin with: Associate yourself with an organization that has a sterling reputation ... which github isn't (anything can be stored on github). On the web site and in the script, identify very clearly who you are (names, locations), and make an argument that you are trustworthy. As far as I can see, the web site has no identifying information whatsoever. By using whois, all I can see that it is registered to an anonymous organization in Russa, not a country that has a sterling reputation in computer security circles. Looking at the author of the script, I see that they are employed by an organization that's incorporated in Switzerland, headquartered in Singapore, and staffed with Russians, again something that does not create a warm and fuzzy feeling. Now I happen to know what your employer A... is (and I don't have any reason to believe that they are evil), and I think you are trustworthy, and there is in reality nothing wrong with Switzerland, Singapore, or Russia, but: Guilt by association. You need to do a much better job demonstrating that you are of good will to people, since otherwise, all the alarm bells will go off.



This project is not related to any organization. It's just my personal contribution to the growth of Linux and BSD operating systems.

I can't note my current employer to avoid erroneous associations of the employer with the project. All I can do is list my well-known open-source projects created in the past. Done this on the contacts page on the site and in the script.



ralphbsz said:


> Having said that ... let's test the script (I know how to sandbox a machine):
> 
> ```
> > mkdir foo
> ...



No ideas yet. Looks lite this can't be fixed in the script itself.



ralphbsz said:


> So try again as root:
> 
> ```
> # ./hw-probe.pl -save foo
> ...



Fixed in master.



ralphbsz said:


> So try it with -probe:
> 
> ```
> ./hw-probe.pl -probe
> ...



This is what differs hw-probe from bsdstats: collecting maximum details about the computer hardware configuration. Base system doesn't include enough utilities to do this.

I've added experimental `-nodeps` option to skip missed dependencies. However `curl` is still required to upload data securely.


----------



## Jose (May 31, 2020)

ralphbsz said:


> That's still thousands of lines of code that need to be run as root. I will simply not download code to be downloaded as root from a source that's not 110% trustworthy. Sure, I need to download the kernel ... but I trust Kirk and his friends. Why? Because I've known Kirk and his friends for decades, and I think I know what makes them tick, what rewards they work for. In this case, I get code from an opaque source, without clear chain of trust.


I don't know them and don't care. What I know is tens (hundreds?) of thousands of people have used their software, and many thousands have examined and modified the source code. It would be very difficult (but not impossible) that any malicious code in it has gone unnoticed. This project lacks that pedigree, and I'm not going to run it on any machine I consider production in any role.



ralphbsz said:


> And: How are you going to make sure that this first program really only makes a list of installed hardware? It could clandestinely transmit information. It could install trojan horses. It could damage data. It could ... do so many things. Really large organizations might be able to validate that this software is "clean" by going a line-by-line code inspection. For 10K lines, that's going to take hundreds of many hours.


There are ways to detect this short of code inspection, but yeah, it would be very laborious. I'd be willing to help, but am not going to try to do it all myself.


----------



## kpedersen (May 31, 2020)

I do appreciate the idea behind this tool. I think we should be making better notes about supported hardware (especially since my beloved Thinkpads are getting worse quality with each revision; I will likely need to jump ship one day and will be pretty lost).

However, why does this need to be run as root? Perhaps it was just quick 'n easy "web dev" instructions that overlooked correct security?

Strip the code out that requires root privileges and more people would be willing to risk it.


----------



## jmos (May 31, 2020)

After all, this is a script. I can read it.

Fact is: Every computer (also: Smartphone) is just as secure as the least trustworthy software on it. And today there is a lot of software on such a computer… And absolutely every single software that I start as a user can read my mails and send itself home, garnished with my SSH keys (!) etc.; Whether root is at work or not, the exciting information in my practice cannot be found in the system configuration, but can be reached as user; And so I see less of a problem with root - running this as root is not more than a stupid feeling. And even on my servers: root is only for administration, everything else is unprivileged (and that's exactly how it should be).

I install a lot of packages in binary form - no idea what I'm getting…: Was this mass of code really checked by others, or simply compiled with a "cool, works!" and offered to me as a package? But still: This "hw-probe" is open source. And that is what I finally decided to trust. If it weren't, yes: I would be skeptical, too.


----------



## ralphbsz (May 31, 2020)

Good question. In a moment, I will ask a deep question, namely "what is really sensible to figure out". I wonder how far one could get from just userspace. Let's see: "pciconf -l" works great. That gives us a list of the graphics devices (which are the #1 through #10 problems in compatibility!), and the disk interfaces. Just doing this would give us 80% of the gain for 1% of the effort, and it works from user space. The next one: "usbconfig list" does not work, needs to be root. But how does it help to know that something is attached in hardware, since we don't know whether it actually works? Next one: dmesg works great from user space. That tells us which disks are attached, a lot of detail of the CPU and memory, network drivers. More importantly, it tells us whether the devices that were found have drivers attached.

The tool the OP posted goes much much further though: it tries to identify the CPU chip exactly (which really doesn't matter, 99.99% of all CPUs are either Intel, AMD or Arm, and the support situation for those is crystal clear), measure the amount of memory (same argument, memory works, we don't need memtest to know whether the memory is fully functional), we don't need smartctl to get details of disks given that we already have the manufacturer, model and revision of the disks in dmesg, and so on. So here is one criticism of the OP's tool: It tries to get to 99% completeness, which requires things that are complex and dangerous and socially not acceptable, when it could have gotten 90% completeness with much less problems.

But the real fundamental problem is: What does it mean for something to be supported? For example, my machine at home contains 4gig of physical RAM (it is a 32-bit CPU). Is that RAM supported? Well, kinda sorta. It functions. But only 3gig of it is actually used on a standard 32-bit machine, which sort of bugs me (not enough to do something about, I could play around with some physical address extension, but I don't have time for that). So I would claim my RAM is not fully supported out of the box. Similar question: I used to have a PCIe attached WiFi card. It was "supported" in the sense that I could make my machine a WiFi client. But I really wanted my machine to be a WiFi AP ... and while the FreeBSD software does actually allow that, it was so buggy at the time that I gave up and switched to an external commercial AP (15 minute trip to the Apple store, done). So is the WiFi card "supported"? Hell no, and I'm actually reasonably upset at the low quality standards of FreeBSD WiFi which allowed half-broken software to get out into the field (it's OK Sam ... I don't mean you, I mean the absence of testers and bug fixers, you did what you could). But the OP's tool would have found it and claimed that it was supported.

So, if the OP's tool wants to know whether something is "supported", it really needs to ask the human user or administator: Are you satisfied with this device, does it meet your expectations? That's something that is not scriptable. Just because a device is physically attached to the computer doesn't mean it is actually working. The definition of "supported" has to be a negotiation process, and is intersubjective and transactional. There is no single correct answer ... the same hardware that I find to be unsupported (the WiFi card, see example above) others will be happy with.

And: the tool goes far beyond listing hardware attributes. It looks at flatpacks, docker, apps, CPU frequencies. This is all very dangerously close to industrial espionage. I'm not claiming that the OP is working for a clandestine source, but I think in his zeal for completeness and correctness (which is, see above, unachievable) he has gone too far in ignoring privacy and security.


----------



## ralphbsz (May 31, 2020)

jmos said:


> After all, this is a script. I can read it.


Try it. It's 17,895 lines long. And it relies on extensive Perl libraries (another hundreds of thousands of lines of code), and installation of about a dozen or two dozen packages (another few million lines of code). With unknown side effects. For example, it tries to run smartctl. While on the surface that might seem sensible (you can learn more about disk drives), it is also dangerous: A few FreeBSD releases back, running smartctl on my boot SSD would cause the system to immediately crash. Known bug in the SATA layer.

So no, you can't read it. A team of several dozen experienced engineers would be able to read it, and its dependencies, if they had a month. If a tool such as this were to be used in a commercial setting (large computer company), that's exactly what would happen: before we are allowed to use it, a team of security people would audit it, line by line.



> Fact is: Every computer (also: Smartphone) is just as secure as the least trustworthy software on it. And today there is a lot of software on such a computer… And absolutely every single software that I start as a user can read my mails and send itself home, garnished with my SSH keys (!) etc.


Absolutely true. And that's why trust is necessary when installing software. Trust is mostly not a technical attribute, but a social one. For example, I trust the small group of people who write FreeBSD, because I've known them for a long time (I know a certain fraction of them personally, and many of them from experience with their work product). As a corporate software customer, I can trust certain vendors (such as RedHat), because I can reason about what their goals and methods are, and see that they have no incentive to do certain nasty things. And even if I don't have trust in them, my lawyers can create contracts that make it very unlikely that such vendors will do bad things ... because if they do, they will go bankrupt, to jail, or both. This doesn't work for software written by an unknown individual, in a jurisdiction where the long arm of the law doesn't reach. Which is why I won't install their software.

(Actually, I did: I did run the OP's hw-probe.pl script several times yesterday. But that's because I know what I'm doing, I know reasonably well how to sandbox things, and I did inspect the OPs background first to get a good level of confidence that he's not a bad person.)


----------



## Jose (May 31, 2020)

ralphbsz said:


> So, if the OP's tool wants to know whether something is "supported", it really needs to ask the human user or administator: Are you satisfied with this device, does it meet your expectations? That's something that is not scriptable. Just because a device is physically attached to the computer doesn't mean it is actually working. The definition of "supported" has to be a negotiation process, and is intersubjective and transactional. There is no single correct answer ... the same hardware that I find to be unsupported (the WiFi card, see example above) others will be happy with.


But then you'll have to trust the user to not deliberately include devices that are not well supported, maybe just for kicks. Maybe you work for the device manufacturer.



ralphbsz said:


> And: the tool goes far beyond listing hardware attributes. It looks at flatpacks, docker, apps, CPU frequencies. This is all very dangerously close to industrial espionage. I'm not claiming that the OP is working for a clandestine source, but I think in his zeal for completeness and correctness (which is, see above, unachievable) he has gone too far in ignoring privacy and security.


There are legitimate reasons to want to know CPU frequencies and model, and things like amount of RAM available. The Steam hardware survey comes to mind. Game developers like to have an idea of how large of an potential audience they're likely to get for a given set of hardware requirements. I can think of use cases beyond games too. Maybe you're working on a video codec or a novel compression algorithm.


----------



## ralphbsz (Jun 1, 2020)

Yes, I agree with your reasoning: When serving individuals, who are using their computers for human interface tasks, things like CPU speed and RAM probably make sense.

Now take this tool to a commercial setting. You know how much computer company A (say IBM, Apple, Microsoft) would want to know how many CPUs and at what speed computer company B (say HP, Amazon, Google) has? They would really want to know! Imagine how interesting it would be to the CIO of Lehman Brothers to get a complete list of the CPU speeds and disk capacities that Smith-Barney is using? (I deliberately used two tanks that no longer exist, so no feelings get hurt). And now do the same thing with the NSA and the FSB, and it stops being funny.

This tool is useful, and also extremely dangerous. The industrial and political espionage aspects are just scary. To get around a lot of those worries, dialing it down would really help: Just doing statistics on desktop users of FreeBSD, and only model numbers of video cards and motherboards.

There are also lots of techniques for correctly anonymizing information that the medical data science community has developed. I think the author might want to look at those technologies, before making this generally available.


----------



## Mjölnir (Jun 6, 2020)

_Next one: dmesg works great from user space. _
In general, yes, but it's advisable to have `security.bsd.unprivileged_read_msgbuf=1` if you're concerned about security.
So I think it's reasonable for such a tool to be run with root privileges.  Nevertheless, all the points you mentioned are _very_ reasonable as well.


----------



## diego (Jul 3, 2020)

Agreed. The tool is just amazing...... I cannot believe how much information it takes from your computer.
Maybe its too much information to share in a database, for security reasons
Questions: Where is located the database? Who have access? Permissions?


----------



## aponomarenko (Jul 3, 2020)

No sensitive information is collected. If you take two different probes of the same computer model made by different people, you will not be able to distinguish or identify them.


----------



## aponomarenko (Jul 3, 2020)

diego said:


> Where is located the database?



The main database is located in the GitHub project in order not to lose the collected data in the future. The https://bsd-hardware.info/ site is just a proxy to the main database.


----------



## jamie (Aug 13, 2020)

I haven't had a look at the script yet, but I'd too prefer if it wasn't run as root - it's not just your code, it's the dependencies, even perl itself may have weird bugs (I had one program that would send perl into an endless loop if it didn't like the locale.. If it was running as root, it may have had access to more resources)

Anyway, as mentioned, dmesg. And also mentioned, dmesg could be disabled.

At default, /var/run/dmesg.boot is readable by all, and would be more useful for device probing that running dmesg, as the boottime messages will expire as the buffer gets full.

Even if this file is locked down, it could be made accessable via ACLs or group membership.

usbcontrol - I have a homegrown script that needs raw usbaccess. Even then, I don't run it as root, I have it run under a user in group "usbaccess", and then:


```
jamie@thompson% cat /etc/devfs.rules
# To give usbconfig(8) and libusb(3) enabled applications permission to all usb
# devices for their owner and the “usb” group, the following rule may be used.
# The first line declares and starts a new ruleset, with the name
# "thompson_local" and the number 999:
[thompson_local=999]
add path 'usb/*' mode 0660 group usbaccess
```
Even then it could be tightened down further.

Ok, I realise these things would require more installation configuration, but how about separating all the privilege-required commands from the script?

e.g., you could have

`25 03 * * * root /sbin/usbconfig list > /var/db/hd-probe/usbconfig.list`

where /var/db/hd-probe would just have read access, so something like:

`drwxr-x---   2  root hd-probe 512 13 Aug 15:13 /var/db/hd-probe/`

Rather than individual cron lines, you could have a minimal script to do this, that really contains nothing but a list of fully-pathed commands that produce the output you require.

Of course, for one-off runs, the documentation could mention to call the minimal script as root, and the non-minimal script as its own user (not any other user!)

Just a few thoughts,

cheers, Jamie


----------



## neilulrich (Dec 23, 2020)

made ssd and hd to plug in to send system hardware info
not taking chance makes us all less secure
binary system design BSD parts plus parts=SYSTEM


----------



## aponomarenko (Dec 23, 2020)

neilulrich said:


> made ssd and hd to plug in to send system hardware info
> not taking chance makes us all less secure
> binary system design BSD parts plus parts=SYSTEM


Yep. Sending report from a LiveCD/LiveUSB (e.g. NomadBSD distribution) is secure too.

BTW Currently, the anonymization script in 1.6.b2 version of the tool is strong enough to remove all private info (if any) from collected logs.


----------



## neilulrich (Dec 23, 2020)

aponomarenko said:


> Yep. Sending report from a LiveCD/LiveUSB (e.g. NomadBSD distribution) is secure too.
> 
> BTW Currently, the anonymization script in 1.6.b2 version of the tool is strong enough to remove all private info (if any) from collected logs.


security had to unplug drives anyway


----------



## jb_fvwm2 (Mar 6, 2021)

BTW running the command at the OP linked site froze the mouse here
til Xorg was restarted... so if more than one tab is open, or it is not easy
to restart Xorg, be prepared to shutdown [ sometimes w/o visible 
input, typing into a blank screen... ]  and restart...

Not to dissuade from contributing.


----------



## grahamperrin@ (Sep 2, 2021)

<https://cgit.freebsd.org/ports/tree/sysutils/hw-probe/Makefile> `d1de28b` (at the time of writing) relates to a 2020 commit:

<https://github.com/linuxhw/hw-probe/commit/d1de28b1ba983cfa64b12e3a7098c8eb257ed9ef>



grahamperrin said:


> FreeBSD bug 252282 – sysutils/hw-probe runtime dependencies



– updated with reference to part of a June 2021 commit: 

<https://github.com/linuxhw/hw-probe...df4e7f5379c70143088da7a585faff98d7ff35c62R290>


----------



## SirDice (Sep 7, 2021)

In case you're wondering why some posts have gone missing, a bunch of them have been trashed.


----------



## grahamperrin@ (Dec 19, 2021)

aponomarenko please, what's the meaning of the new green _Classic_ badges?


----------



## grahamperrin@ (Mar 6, 2022)

sysutils/hw-probe failed upload to upload probe – ~/HW_PROBE/LATEST/hw.info/logs/sysctl is unusually large

In recent weeks, I'm unable to upload the result of any probe from my usual computer. 

Has anyone else encountered the symptom below? 

`ERROR: the probe is larger than 500K

ERROR: failed to upload probe`

The offending file: 


```
% ls -hl /root/HW_PROBE/LATEST/hw.info/logs/sysctl
-rw-r--r--  1 root  wheel   2.0M  6 Mar 12:35 /root/HW_PROBE/LATEST/hw.info/logs/sysctl
% du -hs /root/HW_PROBE/LATEST/hw.info/logs/sysctl
521K    /root/HW_PROBE/LATEST/hw.info/logs/sysctl
%
```



Spoiler: Context





```
root@mowa219-gjp4-8570p-freebsd:~ # date ; uname -aKU
Sun Mar  6 12:21:27 GMT 2022
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #5 main-n253627-25375b1415f-dirty: Sat Mar  5 14:21:40 GMT 2022     root@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 1400053 1400053
root@mowa219-gjp4-8570p-freebsd:~ # time hw-probe -all -upload
Probe for hardware ... Ok
Reading logs ... Ok
ERROR: the probe is larger than 500K

ERROR: failed to upload probe
14.783u 3.050s 0:29.44 60.5%    47+169k 920+388io 506pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # time hw-probe -all
Probe for hardware ... Ok
Reading logs ... Ok
Local probe path: /root/HW_PROBE/LATEST/hw.info
14.373u 2.797s 0:22.72 75.5%    49+169k 311+389io 35pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # du -hs ~/HW_PROBE/
788K    /root/HW_PROBE/
root@mowa219-gjp4-8570p-freebsd:~ # du -h ~/HW_PROBE/
773K    /root/HW_PROBE/LATEST/hw.info/logs
782K    /root/HW_PROBE/LATEST/hw.info
783K    /root/HW_PROBE/LATEST
788K    /root/HW_PROBE/
root@mowa219-gjp4-8570p-freebsd:~ # cd ~/HW_PROBE/LATEST/hw.info/logs
root@mowa219-gjp4-8570p-freebsd:~/HW_PROBE/LATEST/hw.info/logs # bfs . -size +100k -ls
  4153348      9 -rw-r--r--   1 root     wheel      116186 Mar  6 12:35 ./usbconfig
  4153253    521 -rw-r--r--   1 root     wheel     2097152 Mar  6 12:35 ./sysctl
  4153347     25 -rw-r--r--   1 root     wheel      164544 Mar  6 12:35 ./lsusb
root@mowa219-gjp4-8570p-freebsd:~/HW_PROBE/LATEST/hw.info/logs # less sysctl
root@mowa219-gjp4-8570p-freebsd:~/HW_PROBE/LATEST/hw.info/logs # head -n 25 sysctl
kern.ostype: FreeBSD
kern.osrelease: 14.0-CURRENT
kern.osrevision: 199506
kern.version: FreeBSD 14.0-CURRENT #5 main-n253627-25375b1415f-dirty: Sat Mar  5 14:21:40 GMT 2022
    XXX@XXX:/usr/obj/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

kernXXaxvnodes: 348278
kernXXaxproc: 21636
kernXXaxfiles: 520033
kern.argmax: 524288
kern.securelevel: -1
kern.hostname: ...
kern.hostid: 33473F351CBFA5B64728CB222EA22970
kern.clockrate: { hz = 1000, tick = 1000, profhz = 8128, stathz = 127 }
kern.posix1version: 200112
kern.ngroups: 1023
kern.job_control: 1
kern.saved_ids: 0
kern.boottime: { sec = 1646543691, usec = 595130 } Sun Mar  6 05:14:51 2022
kern.domainname:
kern.osreldate: 1400053
kern.bootfile: /boot/kernel/kernel
kernXXaxfilesperproc: 468027
kernXXaxprocperuid: 19472
kern.ipcXXaxsockbuf: 2097152
root@mowa219-gjp4-8570p-freebsd:~/HW_PROBE/LATEST/hw.info/logs #
```


----------



## Geezer (Mar 6, 2022)

grahamperrin said:


> `mowa219-gjp4-8570p-freebsd`



Interesting name for a box.


----------



## grahamperrin@ (Mar 12, 2022)

excessive probing and logging · Issue #123 · linuxhw/hw-probe
					

From hw-probe -help: -probe Probe for hardware. Collect only hardware related logs. Expected With general option -probe, logs not related to hardware must not be collected. Actual result Inappropri...




					github.com
				












						log level ambiguities · Issue #124 · linuxhw/hw-probe
					

Observation A From hw-probe -help: -minimal|-min Collect minimal number of logs. Equal to --log-level=min. -maximal|-max Collect maximal number of logs. Equal to --log-level=max. Issue There is no ...




					github.com
				






grahamperrin said:


> please, what's the meaning of the new green _Classic_ badges?











						Classic · Issue #125 · linuxhw/hw-probe
					

From https://forums.freebsd.org/posts/547382: I can't find an explanation for the green Classic badging that's seen in areas such as https://bsd-hardware.info/?computer=6fbb1f806232




					github.com
				






grahamperrin said:


> Has anyone else encountered the symptom below?
> 
> `ERROR: the probe is larger than 500K …`



Maybe no-one else, but I took the opportunity to raise an issue: 









						extraordinarily large /root/HW_PROBE/LATEST/hw.info/logs/sysctl preventing upload · Issue #126 · linuxhw/hw-probe
					

https://forums.freebsd.org/posts/558998 a few days ago: sysutils/hw-probe failed upload to upload probe … In recent weeks, I'm unable to upload the result of any probe from my usual computer. H...




					github.com


----------



## Geezer (Mar 12, 2022)

I don't get that error.


```
Executing hw-probe -all -upload

Probe for hardware ... Ok
Reading logs ... Ok
WARNING: failed to detect EDID
Uploaded to DB, Thank you!

Probe URL: https://bsd-hardware.info/?probe=abcdefgh
```


----------



## grahamperrin@ (Mar 12, 2022)

Geezer said:


> I don't get that error.



Thanks.

I can disable (disallow) the excessively large file, for example:



Spoiler: hw-probe -all -disable sysctl





```
root@mowa219-gjp4-8570p-freebsd:~ # rm -r /root/HW_PROBE/LATEST
root@mowa219-gjp4-8570p-freebsd:~ # hw-probe -all -disable sysctl
Probe for hardware ... Ok
Reading logs ... Ok
Local probe path: /root/HW_PROBE/LATEST/hw.info
root@mowa219-gjp4-8570p-freebsd:~ # du -hs /root/HW_PROBE/LATEST/hw.info
262K    /root/HW_PROBE/LATEST/hw.info
root@mowa219-gjp4-8570p-freebsd:~ # file /root/HW_PROBE/LATEST/hw.info/logs/sysctl
/root/HW_PROBE/LATEST/hw.info/logs/sysctl: cannot open `/root/HW_PROBE/LATEST/hw.info/logs/sysctl' (No such file or directory)
root@mowa219-gjp4-8570p-freebsd:~ #
```




I'm puzzled by the file's content, here, when it _is_ allowed. 

`tail -n 100 /root/HW_PROBE/LATEST/hw.info/logs/sysctl`

Do you see anything unusual? 

Compare with mine at <https://github.com/linuxhw/hw-probe/issues/126#issue-1167209962>.


----------



## Geezer (Mar 12, 2022)

On the off, what does `hw-probe -version` give you?


```
Hardware Probe 1.6
A tool to probe for hardware, check operability and find drivers
License: LGPL-2.1-or-later OR BSD-4-Clause
```


----------



## grahamperrin@ (Mar 12, 2022)

Geezer said:


> what does `hw-probe -version` give you?



The same as you, but the version of the port/package, seen in the issue, is slightly different.


----------



## grahamperrin@ (Mar 20, 2022)

grahamperrin said:


> … Has anyone else encountered the symptom below? …



The likely explanation: boot profiling with a *TSLOG-enabled kernel*. <https://github.com/linuxhw/hw-probe/issues/126#issuecomment-1073269062>


----------



## achix (Jun 16, 2022)

so this site died right?


----------



## jbo (Jun 16, 2022)

achix said:


> so this site died right?


No, still alive: https://bsd-hardware.info

A lot of users just choose not to run the probe utility due to aforementioned "problems" (lots of non-audited code running as root).


----------

