# VMware ESXi best practices?



## spork (Aug 4, 2015)

I suppose this is a silly time to ask this now that I have about a dozen virtualized FreeBSD instances running, but after having one panic this morning in the VMware memory control kernel module, I'm reminded there are probably a few things I've not kept up with since virtualizing a bunch of hosts a few years ago.

I'll start with a few things that I believe are correct based on past Googling:

• Do not let the VMware tools set the guest's time, use NTP.
• If using VMware's included tools, ensure that the emulators/compat6x port is installed.
• Some older 5.x versions of ESXi have a bug in the HPET timer that will cause your guest to stop moving forward in time (yay PF states that never expire because time has stopped!).  http://kb.vmware.com/kb/2032586

And that's about all I remember. 

So on to questions:

• Use VMware's included tools or emulators/open-vm-tools?  And what is the difference?
• Use em(4) or the if_vmx(4) kernel module for networking?
• Also, regrading the guest tools, is it OK to be using tools that are ahead of the ESXi host version?

And finally, just any best practices you know of if you run lots of VMware-virtualized FreeBSD.


----------



## robroy (Aug 5, 2015)

Good day spork,

I'll comment on two of your questions, though my thoughts may be "old news" to you.



spork said:


> Use em(4) or the if_vmx(4) kernel module for networking?



The VMXNET3 interface is preferred when it's available, since its performance is supposed to be better than the virtual Intel NIC (em(4)).  I've used both types with FreeBSD guests in the past, and can't remember having had problems with either type, but I never seriously stress-tested them.



spork said:


> Also, regrading the guest tools, is it OK to be using tools that are ahead of the ESXi host version?



Ideally, the VMware Tools version should match what's distributed with your ESX version; Tools shouldn't be newer than your ESX version.

When I worked at VMware (2006 to 2009), nearly all of the testing was done with Tools running in the guests, and with the Tools version matching the ESX version.  That was the "beaten path" configuration that was least likely to yield surprises.  This said, very little emphasis was put on testing ESX with guests other than Windows and Linux at the time; 'not sure about today.


----------



## wisdown (Aug 12, 2015)

I am on same boat like you spork.

On my scenario I have ESXi free version, no extras features like vMotion, etc...

One thing I can share and have read in a lot of tutorials and blogs, is about not using thin disks with ZFS, but no one can explain why...

On my setup I make the datasets by hand, so after use GPT on disk, usually I do:

`dd if=/dev/random of=/dev/adaX bs=4096`

For every pool, this avoid the messages about error using encryption the checksums will be ready to use) and will make the thin disk get full...
On my scenario, I use thin to make an vmdk of 2TB, but making the partitions for ZFS I set an low space (around 256 GB). This is for in future. If I need more room, the vmdk is already ready for it (remaining unpartitioned space don't will spend room)...

About vmware-tools; If you try install them on new Linux versions will get a warning about the indication for use the emulators/open-vm-tools and then setup exit (on FreeBSD and ESXi 6 this not happen), In my opinion after few tests (cannot be exhaustive because is free version) the vmware-tools are better (installed on old Linux version then did upgrade with tools installed).


P.S.: Im using ESXi 6


----------



## rootwyrm (Aug 16, 2015)

robroy said:


> Ideally, the VMware Tools version should match what's distributed with your ESX version; Tools shouldn't be newer than your ESX version.
> 
> When I worked at VMware (2006 to 2009), nearly all of the testing was done with Tools running in the guests, and with the Tools version matching the ESX version.  That was the "beaten path" configuration that was least likely to yield surprises.  This said, very little emphasis was put on testing ESX with guests other than Windows and Linux at the time; 'not sure about today.



Apparently, I was finally successful in yelling at them to addressing the tools deficiencies with FreeBSD 10. As of 5.5U2+ you should *only *use the actual VMware Tools and basically _never_ use emulators/open-vm-tools. In fact, emulators/open-vm-tools will make your life miserable these days and little else. Of course, official still requires misc/compat6x (seriously guys? Sigh.) But baby steps, right?

Do note that there's a rough patch around 10.1-RELEASE-p8 to p14 or thereabouts where instead of rebooting, the system will hang until watchdog hits. Probably related to a known FreeBSD bug, but it's really unpleasant to deal with in an Ent+ cluster. Resolved in p15+ for sure, never showed up in 9.x. Note that you may still hit it if you reboot from the OS instead of having tools initiate the reboot. I'm not sure why that is, but I haven't had time to investigate further. Tools initiated works perfectly every time though.

Other than that? Don't use timesync *ever*, bad things will happen. Otherwise, you really don't need to do anything other than use VMware Tools and VMXNET3 interfaces. I recommend using the LSI SAS for SCSI, but it really doesn't matter. Both will throw the same error if your disks go away unexpectedly. Don't need to worry about setting your clock device any more either.

Do note that if you load crypto(4), you will break migration unless your cluster is in EVC mode or identical hardware. Loading the module is what triggers it; if you do not load the module, you can still migrate from AESNI to non-AESNI processors.


----------



## wisdown (Aug 17, 2015)

Just adding a found:

Testing about think HDs on one datastore and thick provision eager zeroed in another dedicated datastore, maybe I got the point about the recommendation about not use think.

On thick vmdks if you rename the vmdk for something else, seems the occult and not documented index by vmware is lost, then esxi start make an flat full file, or in practice, the index works like an point to an compressed file, if index is lost, esxi start decompress the file...

Replicating the issue:
Make one virtual Machine with 1 TB think HD.
Detach the think HD.
Go to directory of the dettached think HD and rename it for other think example:
test.vmdk -> test.vmdk.other
Refresh the folder and then you can see esxi making an full file, from previous example:
test-flat.vmdk (1TB uncompressed)

Now think about 2 compression working all time...
First the ones from thick vmdk, second the other from ZFS if you set it, and then this is why there performance issues...

I'm migrating all my thick to an smallest thick provision zeroed, the virtual machines after this migration are running more faster.


----------



## urbanriot (Nov 14, 2015)

rootwyrm said:


> Do note that there's a rough patch around 10.1-RELEASE-p8 to p14 or thereabouts where instead of rebooting, the system will hang until watchdog hits. Probably related to a known FreeBSD bug, but it's really unpleasant to deal with in an Ent+ cluster. Resolved in p15+ for sure, never showed up in 9.x. Note that you may still hit it if you reboot from the OS instead of having tools initiate the reboot. I'm not sure why that is, but I haven't had time to investigate further. Tools initiated works perfectly every time though.



We're considering the idea of moving a number of LAN facing servers, primarily Squid proxies, and internet facing servers, primarily Apache HP web hosts, and were wondering if anyone has any updated warnings or words of advice concerning FreeBSD 10.2 with VMware ESXi 6.0 Update 1 or newer? Or alternatively if FreeBSD 9.3 is a better route to go?


----------



## rootwyrm (Nov 15, 2015)

urbanriot said:


> We're considering the idea of moving a number of LAN facing servers, primarily Squid proxies, and internet facing servers, primarily Apache HP web hosts, and were wondering if anyone has any updated warnings or words of advice concerning FreeBSD 10.2 with VMware ESXi 6.0 Update 1 or newer? Or alternatively if FreeBSD 9.3 is a better route to go?



I am putting together some new servers (LittleDragon III's) for VMware 6.0 here, but that's not done yet, so afraid I can't provide any deep insights. Some rough testing on another 6.0U1 install didn't reveal any issues at all for a guest migrated from 5.5. The biggest catch you'll need to watch for is if you use Workstation to prep. Every time I've tried going Workstation 10 to ESX6 it has utterly failed. I don't have a Workstation 12 Pro license, so I can't test that.


----------



## glocke (Nov 24, 2015)

rootwyrm said:


> Apparently, I was finally successful in yelling at them to addressing the tools deficiencies with FreeBSD 10. As of 5.5U2+ you should *only *use the actual VMware Tools and basically _never_ use emulators/open-vm-tools. In fact, emulators/open-vm-tools will make your life miserable these days and little else. Of course, official still requires misc/compat6x (seriously guys? Sigh.) But baby steps, right?


Could you elaborate on that please? Any sources / KB entries / links? What problems did you encounter? What was your setup (running X11 or headless). Thanks in advance


----------



## hruodr (May 23, 2019)

What storage adapter is working with FreeBSD? I have problems with VMWare Paravirtual in OpenBSD.


----------



## hruodr (May 24, 2019)

The answer to my above question is here:






						VMware Compatibility Guide - Guest/Host Search
					






					www.vmware.com


----------

