# Some questions about FreeBSD-SA-19:07.mds



## zirias@ (May 15, 2019)

I'm having a few doubts here. First about the microcode:


> Ensure that the BIOS update or devcpu-data package is dated after 2014-05-14.


Is this a typo? Or does it mean there is no microcode update addressing this specific problem yet, but the mitigation needs some instruction introduced much earlier?



> Mitigation Configuration
> 
> Systems with users, processes, or virtual machines in different trust
> domains should disable Hyper-Threading
> ...


What exactly does this mitigation do? Why is there the recommendation to still disable HT? Does that mean the mitigation is known not to address all possible problems or does it mean it's just unknown whether there's still a risk left with HT?


----------



## zirias@ (May 15, 2019)

Zirias said:


> Is this a typo?


It obviously was, the date now reads "2019-05-14" 

Anyways, I'm still curious about the background of the mitigation and the recommendation to disable HT ...


----------



## SirDice (May 15, 2019)

Does this help?









						Intel’s New Spectre-Like Flaw Affects Chips Made Since 2008
					

Intel revealed a new speculative execution attack that would allow malicious actors to obtain sensitive information that would otherwise be protected by the processor. The flaw could affect system performance.




					www.tomshardware.com


----------



## zirias@ (May 15, 2019)

SirDice said:


> Does this help?


Not so much, unfortunately.

Without knowing what FreeBSD's mitigation does, it's hard to assess whether you should disable HT. Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?

For example, I have a virtual machine with Windows 7 running on my server, it's for testing purposes and only has internet access -- no way to reach anything in my local network. So, if this machine catches some malware, it's not a big deal -- once noticed, just reset it to an earlier ZFS snapshot. Now, without disabling HT, do I have to worry that malware on this machine could somehow spy on processes running on my FreeBSD host? If for example the mitigation would make sure that bhyve processes will never share a core with processes of the host or of other bhyve VMs, I'd say it's fine for me to keep HT enabled. But is this the case?


----------



## SirDice (May 15, 2019)

Zirias said:


> Without knowing what FreeBSD's mitigation does, it's hard to assess whether you should disable HT. Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?


That question is probably best asked on the freebsd-security  mailing list. I'm not sure if we have forum members with that level of intimate knowledge.


----------



## PMc (May 15, 2019)

Zirias said:


> Not so much, unfortunately.
> 
> Without knowing what FreeBSD's mitigation does



Looking at the code, it does bring in some assembler code differenciated for various CPU types from intel core gen2 upwards, and/or some use of an assember CPU instruction `verw` and a sysctl(8) switch `hw.mds_disable`.

Concerning that instruction, I could find this snippet:
_The Intel *CPU instruction* called "*VERW*" is enhanced by the new *CPU* Microcode such that it flushes all buffers and ports. The *VERW* *instruction* will be called during task switch or VM switch by the patched kernels and hypervisors._
This explanation comes from some Linux, but one might assume it's the same here.



> , it's hard to assess whether you should disable HT.



It is indeed, but that gets someway clear from the description of the thing in some of the papers floating around:
They say there are basically two different attack methods. One is to obtain unaccessible data when a thread transites from kernel to userland (probably difficult to implement such an attack), the other is that two threads are running on the same core in hyperthreading mode, and one obtains data owned by the other (the more common and demonstrated way).
It further says that cleaning the microarch structures does not fully help against the second scheme, because this cleaning happens within one thread, while the other could still obtain something.
Therefore they recommend to swich off HT in cases where a core could be shared by hostile threads.



> Does FreeBSD's mitigation somehow classify processes in "security domains"? If so, how?



Sorry, I have no idea.



> For example, I have a virtual machine with Windows 7 running on my server, it's for testing purposes and only has internet access -- no way to reach anything in my local network. So, if this machine catches some malware, it's not a big deal -- once noticed, just reset it to an earlier ZFS snapshot. Now, without disabling HT, do I have to worry that malware on this machine could somehow spy on processes running on my FreeBSD host? If for example the mitigation would make sure that bhyve processes will never share a core with processes of the host or of other bhyve VMs, I'd say it's fine for me to keep HT enabled. But is this the case?



Wouldn't it be possible to dedicate or reserve specific cores (and only complete cores) to each installation? On FreeBSD this could be done with cpuset(1) (I don't touch windows).


----------



## zirias@ (May 16, 2019)

PMc said:


> It further says that cleaning the microarch structures does not fully help against the second scheme, because this cleaning happens within one thread, while the other could still obtain something.
> Therefore they recommend to swich off HT in cases where a core could be shared by hostile threads.


Thanks, this already explains a lot. So probably, FreeBSD's mitigation just makes sure to issue a VERW at appropriate times, but doesn't change scheduling so that bhyve and host processes don't share the same core...
That's bad news for me, and I guess I'll have to live with the risk here.


PMc said:


> Wouldn't it be possible to dedicate or reserve specific cores (and only complete cores) to each installation?


Just pinning bhyve to a specific core wouldn't help, I'd also have to make sure any other process doesn't use that core. As my server only has 4 cores, doing that (possibly with several virtual machines) would be even worse than disabling HT. A good solution for my specific usecase would be if the scheduler would block the core as long as a bhyve process is active on it...


----------



## hsw (May 16, 2019)

Just hit this problem system was really unstable random memory corruption random segfaults
because pkg repo does not have up-to-date devcpu-data (v1.21) had to build from ports (v1.22)
should have realised somthing was wrong when "sysctl hw.mds_disable_state" showed "Software ..." and not "VERW"


----------



## PMc (May 17, 2019)

Yeah, here it illuminates only half of the monitor anymore...


----------



## CyberCr33p (May 17, 2019)

In mds advisory they say:


```
Perform one of the following:

Update CPU microcode, upgrade your vulnerable system to a supported FreeBSD
stable or release / security branch (releng) dated after the correction date,
evaluate mitigation and Hyper Threading controls, and reboot the system.
```

So we either need to use updated microcode or upgrade to 12.0-RELEASE-p5?

I just upgrade to p5 and I see:


```
sysctl hw.mds_disable_state
hw.mds_disable_state: inactive
```

So it's disabled by default and we have to set hw.mds_disable_state during boot.

They recommend to use hw.mds_disable=3 to autoselect if the microcode or the code will be used.

Is the microcode update better for performance?


----------



## PMc (May 17, 2019)

CyberCr33p said:


> In mds advisory they say:
> 
> 
> ```
> ...



NO! Both!

This advisory is horrible. It uses a terrible language, a creepy formatting, and needs a lawyer to understand what it says. 

The stance "_Perform one of the following_" does not apply to the directly following paragraph!
It does apply to the numbered list that follows much later.



> So it's disabled by default and we have to set hw.mds_disable_state during boot.
> 
> They recommend to use hw.mds_disable=3 to autoselect if the microcode or the code will be used.



Yepp.



> Is the microcode update better for performance?



I might suppose so, but I don't know for certain.

I have set 

```
pmc@disp:501:1~$ grep mds /etc/sysctl.conf 
hw.mds_disable=3
pmc@disp:502:1~$
```

and now it shows


```
pmc@disp:504:1~$ /sbin/sysctl -a | grep mds_
hw.mds_disable: 3
hw.mds_disable_state: software IvyBridge
```

I have not installed the microcode - if there is any for IvyBridge - because nobody did yet bother to put it into the `2019Q2` ports branch.


----------



## noodlefling (May 24, 2019)

I'm still confused by the whole thing.

Are they saying we should try to get a firmware upgrade first, and then maybe we don't need to make these tweaks?  How do I even go about figuring out which processor I've got in each machine? I've updated the BIOS on many computers before, but the vendor tools all required Windows.


```
Systems with users, processes, or virtual machines in different trust
domains should disable Hyper-Threading by setting the
machdep.hyperthreading_allowed tunable to 0:
```
What's a "different trust domain" in this context?  If I don't have those sorts of things, does this mean I can ignore the mitigation? Does the mitigation significantly impact performance?

When did you guys start experiencing hardware problems?  After the regular `freebsd-update`, or after applying the "Mitigation Configuration"?

Given that these instructions are usually extremely explicit, this one seems awfully muddy.


----------



## zirias@ (May 24, 2019)

noodlefling said:


> Are they saying we should try to get a firmware upgrade first, and then maybe we don't need to make these tweaks?


No. The firmware upgrade enables a "better" tweak using a new CPU instruction introduced in the microcode (VERW). So you should do both. Full protection means disabling HT altogether 


noodlefling said:


> How do I even go about figuring out which processor I've got in each machine?


`sysctl hw.model`


----------



## noodlefling (May 24, 2019)

Sorry, I was editing my message when you responded.



Zirias said:


> Full protection means disabling HT altogether



Is this a significant performance hit?

Also thanks for that `sysctl`command, that's cool.


----------



## zirias@ (May 24, 2019)

noodlefling said:


> Is this a significant performance hit?


Depending on your workload, you'll loose almost 50%. With HT, you can have two jobs on each core.

My suggestion would be:

On a desktop, don't bother and don't enable any mitigation. There are attack vectors much more likely than this processor bug.
On a server, it really depends on the scenario. What's critical is running virtual machines with "untrusted" software*. These might spy on processes outside the vm. Setting `hw.mds_disable` to 1 or 3 is a partial mitigation and costs performance (I have no figures here, might be around 30%). Disabling HT altogether is the full mitigation, but -- see above.
* of course, just having processes with vastly different trust levels is already a risk, you don't want some malware to spy on your web browser, for example... how likely and how grave this is is up to you to assess.


----------



## PMc (May 24, 2019)

noodlefling said:


> When did you guys start experiencing hardware problems?  After the regular `freebsd-update`, or after applying the "Mitigation Configuration"?



There aren't; that was a troll.

Concerning performance hits - my CPU cannot do hyperthreading, and I didn't measure/compare speed. In any case, the thing can be switched off again.


----------



## noodlefling (May 24, 2019)

OK, I've read through this thread and some links a few more times and I feel like I understand a tiny bit more.

The root of the problem is processes being able to see what's going on in other processes (I think).  And specifically that processes inside a virtual machine would be able to see what's going on outside the sandbox, which is very bad.

But what about if all activity on the server is taking place in the same space (ie, no virtual machines)?  Is that better or worse or makes no difference?

All of this seems to require access to an account that can run processes, and if a bad guy's got that, we're already in trouble.  I suppose we could have someone who _should _have access spying on stuff he shouldn't be able to see, but I'm fairly comfortable that none of our users would be able to figure something like that out, even if they were willing.

But let's say the server is running apache.  Are we talking about outside web users being able to exploit this bug without some other ability to exploit apache?  That is, can anyone who can't otherwise gain privileges exploit this bug?

If I'm going to cut efficiency in half or even by a third, I need a pretty good reason.

It does say you should do the `freebsd-update` before mitigation, so does that on its own actually improve the situation, or does it only give you the tools to "mitigate" and without doing so, you're no safer than before the update?

Sorry to be so newbish.  I think my brain's gotten gotten lazy after years of "to get your system updated, just type in these _exact _magic words and you'll be fine".


----------



## oddnoc (Jun 10, 2019)

I've been seeing segmentation faults (error 11) on cloud-based systems where I've enabled the mitigations. I'll say that it happens "under load" because it does not seem to happen on systems with light activity. I've not seen any definitive guidance from the FreeBSD Project, Intel, or cloud vendors as to whether one can rely on the host system to implement the mitigations (meaning that they can be left disabled on guest systems). Because crashes of mission-critical processes (I'm focused on the nginx, mariadb/mysql, php stack) are unacceptable, it seems like there's not much choice but to run without the mitigations on production cloud systems.

I'm sure someone knows better than I do, and I'm eager to hear from them.


----------

