# Solaris 11 in bhyve



## absduser (Jan 28, 2016)

What is the current state of development towards running Solaris in a bhyve? Now, I know it's not officially supported... but I wonder how close this is to working? Would it even boot? Is it horribly broken? What is the difference in getting Solaris to run in a bhyve vs. Ubuntu, for example?

If Solaris in bhyve is unworkable or a long way off, might the following rat's nest have a chance of working?:

FreeBSD -> bhyve -> Ubuntu -> Xen -> Solaris 11


----------



## grehan@ (Jan 29, 2016)

Recent Illumos-derived distros (e.g. SmartOS) should work fine - http://docs.FreeBSD.org/cgi/mid.cgi?56118B2B.2040101

Was this for Solaris 11.3 ?


----------



## absduser (Feb 2, 2016)

Thank you! I see that I would need FreeBSD 11 release r288524 or greater as I'm running something a bit older on 10.2 AMD which apparently doesn't support it:


```
bhyve: invalid lpc device configuration 'bootrom,BHYVE_UEFI_CSM_20151002.fd'
```

I'd prefer not to reinstall my OS...would be it possible to copy in /usr/src/usr.sbin/bhyve/* (from FreeBSD 11) and build it on my 10.2 box to get the (UEFI) support I need to get this working?


----------



## SirDice (Feb 3, 2016)

absduser said:


> I'd prefer not to reinstall my OS...would be it possible to copy in /usr/src/usr.sbin/bhyve/* (from FreeBSD 11) and build it on my 10.2 box to get the (UEFI) support I need to get this working?


I very much doubt this will work as things are closely tied to the vmm kernel module. So you'd need to import that too. And that will cause problems because of differences between 10 and 11. 

You may want to give 10-STABLE a shot. And there's no need to 'reinstall' the entire OS. You can simply do a source upgrade from 10.2-RELEASE to 10-STABLE or 11-CURRENT.


----------



## grehan@ (Feb 3, 2016)

The relevant changes were backported to 10-STABLE with r295124 so as SirDice mentioned a  source-upgrade from 10.2 to 10-STABLE should allow you to get the required features.


----------



## absduser (Feb 4, 2016)

Thanks for your advice. I'm not sure SmartOS will work as we need some features that only Solaris ZFS supports. So, to my original question, is there a chance of getting any modern version (not necessarily the latest) of Solaris working in a bhyve?


----------



## grehan@ (Feb 5, 2016)

I've been able to install Solaris 11.3 using UEFI (not CSM) firmware, but there's no virtio network driver support and also haven't been able to get a clean boot post-install.


----------



## absduser (Dec 12, 2016)

Is there any new information regarding a Solaris bhyve running on FreeBSD 10.3-RELEASE ?

I was under the impression that this was supposed to be working in 10.3 because of new UEFI support ... has anyone successfully gotten this to work ?

Thanks.


----------



## fnoyanisi (Dec 12, 2016)

You may want to flick an email to virtualization mailing list


----------



## grehan@ (Dec 12, 2016)

FreeNAS has a template for it - https://github.com/freenas/vm-templates/blob/master/solaris-11.3/template.json

Looks like the e1000 NIC emulation is needed (to get around the missing virtio drivers mentioned above), but this hasn't yet been MFCd to 10-stable.


----------



## free-and-bsd (Dec 14, 2016)

absduser said:


> Thanks for your advice. I'm not sure SmartOS will work as we need some features that only Solaris ZFS supports.


I'm not sure SmartOS _has_ any such features in its version of ZFS. Whatever stems from OpenSolaris, as does SmartOS, is shared by FreeBSD ZFS driver as well, is it not?
And all the unsupported Solaris ZFS features are in Oracle Solaris where they appeared after the closure of the code, are they not?


----------



## absduser (Jan 11, 2017)

OK, I've got Solaris 11 running inside smartos (which is running inside a bhyve) per instructions here:
https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-October/003833.html

I thought I'd try an experiment and try to run Solaris in a bhyve in the same way per:

`bhyve -c 2 -s 3,ahci-cd,sol-11_3-text-x86.iso -s 4,ahci-hd,vm.img -s 5,virtio-net,tap1 -s 31,lpc -l com1,stdio -l com2,/dev/nmdm1A -l bootrom,BHYVE_UEFI_CSM_20151002.fd -m 4G -H solaris`

And it seems to work- the installer runs, it sees the disk, but it does not see any NICs. This is no surprise since I needed to use e1000 within SmartOS to get Solaris to see the NIC in that install. 

A little poking around reveals e1000 support was added for bhyve:

https://svnweb.freebsd.org/base?view=revision&sortby=date&revision=302504

So, on my 10.3-RELEASE system I patched:

```
/usr/src/usr.sbin/bhyve/Makefile
/usr/src/usr.sbin/bhyve/bhyve.8
/usr/src/usr.sbin/bhyve/pci_e82545.c (added)
```
and rebuilt bhyve and when I run:

`bhyve -c 2 -s 3,ahci-cd,sol-11_3-text-x86.iso -s 4,ahci-hd,vm.img -s 5,e1000,tap1 -s 31,lpc -l com1,stdio -l com2,/dev/nmdm1A -l bootrom,BHYVE_UEFI_CSM_20151002.fd -m 4G -H solaris`

I get:



```
*SNIP*

LegacyGetDataOrTable - ID: 0, Success
  Table - BE86D000, Size - 124, Location - BE622035, Alignment - 0
Table ID 0 length insufficient
LegacyGetDataOrTable - ID: 1, Unsupported
  Table - BE86D000, Size - 124, Location - BE622035, Alignment - 0
LegacyGetDataOrTable - ID: 2, Unsupported
  Table - BE86D000, Size - 124, Location - BE622035, Alignment - 0
LegacyGetDataOrTable - ID: 3, Unsupported
  Table - BE86D000, Size - 124, Location - BE622035, Alignment - 0
Bhyve-HandleProtocol: Copying DevPath: PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0x0,0x0) [32]
Legacy INT19 Boot...
vm_run error -1, errno 14

#
```

Of note, the patch to the Makefile for e1000 was not matching up nicely with the Makefile in 10.3 so here's my manually patched Makefile (in case that's the source of the issue):


```
#
# $FreeBSD: releng/10.3/usr.sbin/bhyve/Makefile 295124 2016-02-01 14:56:11Z grehan $
#

PROG=   bhyve

DEBUG_FLAGS= -g -O0

MAN=    bhyve.8

SRCS=   \
        atkbdc.c                \
        acpi.c                  \
        bhyverun.c              \
        block_if.c              \
        bootrom.c               \
        consport.c              \
        dbgport.c               \
        fwctl.c                 \
        inout.c                 \
        ioapic.c                \
        mem.c                   \
        mevent.c                \
        mptbl.c                 \
        pci_ahci.c              \
        pci_e82545.c            \
        pci_emul.c              \
        pci_hostbridge.c        \
        pci_irq.c               \
        pci_lpc.c               \
        pci_passthru.c          \
        pci_virtio_block.c      \
        pci_virtio_net.c        \
        pci_virtio_rnd.c        \
        pci_uart.c              \
        pm.c                    \
        post.c                  \
        rtc.c                   \
        smbiostbl.c             \
        task_switch.c           \
        uart_emul.c             \
        virtio.c                \
        xmsr.c                  \
        spinup_ap.c

.PATH:  ${.CURDIR}/../../sys/amd64/vmm
SRCS+=  vmm_instruction_emul.c

DPADD=  ${LIBVMMAPI} ${LIBMD} ${LIBUTIL} ${LIBPTHREAD}
LDADD=  -lvmmapi -lmd -lutil -lpthread -lz
CFLAGS+= -I${BHYVE_SYSDIR}/sys/dev/e1000
CFLAGS+= -I${BHYVE_SYSDIR}/sys/dev/mii

WARNS?= 2

.include <bsd.prog.mk>
```

So, what have I missed, why doesn't e1000 work? 

Are there some other dependencies which I should also patch? 

Is this a fool's errand- will this (Solaris in a bhyve) ultimately not work for some other reason?


----------



## grehan@ (Jan 11, 2017)

You'll most likely need a `-s 0,hostbridge` option. Not sure if that will help or not.

FreeNAS has a template for Solaris 11.3 so assuming someone has it working.
https://github.com/freenas/vm-templates/blob/master/solaris-11.3/template.json


----------



## absduser (Jan 12, 2017)

I think FreeNAS 10 is based on FreeBSD 11 so that's not super applicable to my situation on 10.3

Unfortunately, adding hostbridge didn't help:


```
Failed to emulate instruction [0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf 0xaf] at 0xbea43fe9
```


----------

