# Lock order reversal on boot



## benshort (Aug 17, 2011)

I'm running FreeBSD 9 BETA1 and got the following when booting my system. Should I report this as a bug?


```
Aug 17 09:14:25 storeageserver1 kernel: Trying to mount root from zfs:zsystem []...
Aug 17 09:18:38 storeageserver1 kernel: lock order reversal:
Aug 17 09:18:38 storeageserver1 kernel: 1st 0xfffffe004626b9f8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:828
Aug 17 09:18:38 storeageserver1 kernel: 2nd 0xfffffe0046ed5278 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2134
Aug 17 09:18:38 storeageserver1 kernel: KDB: stack backtrace:
Aug 17 09:18:38 storeageserver1 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Aug 17 09:18:38 storeageserver1 kernel: kdb_backtrace() at kdb_backtrace+0x37
Aug 17 09:18:38 storeageserver1 kernel: _witness_debugger() at _witness_debugger+0x2e
Aug 17 09:18:38 storeageserver1 kernel: witness_checkorder() at witness_checkorder+0x807
Aug 17 09:18:38 storeageserver1 kernel: __lockmgr_args() at __lockmgr_args+0xd42
Aug 17 09:18:38 storeageserver1 kernel: vop_stdlock() at vop_stdlock+0x39
Aug 17 09:18:38 storeageserver1 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Aug 17 09:18:38 storeageserver1 kernel: _vn_lock() at _vn_lock+0x47
Aug 17 09:18:38 storeageserver1 kernel: vget() at vget+0x7b
Aug 17 09:18:38 storeageserver1 kernel: devfs_allocv() at devfs_allocv+0x101
Aug 17 09:18:38 storeageserver1 kernel: devfs_root() at devfs_root+0x4d
Aug 17 09:18:38 storeageserver1 kernel: vfs_donmount() at vfs_donmount+0x988
Aug 17 09:18:38 storeageserver1 kernel: nmount() at nmount+0x63
Aug 17 09:18:38 storeageserver1 kernel: syscallenter() at syscallenter+0x1aa
Aug 17 09:18:38 storeageserver1 kernel: syscall() at syscall+0x4c
Aug 17 09:18:38 storeageserver1 kernel: Xfast_syscall() at Xfast_syscall+0xdd
Aug 17 09:18:38 storeageserver1 kernel: --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800ab43cc, rsp = 0x7fffffffca68, rbp = 0x801009048 ---
Aug 17 09:19:26 storeageserver1 kernel: pid 7213 (try), uid 0: exited on signal 10 (core dumped)
Aug 17 10:49:40 storeageserver1 kernel: pid 19881 (try), uid 0: exited on signal 10 (core dumped)
Aug 17 12:13:10 storeageserver1 kernel: lock order reversal:
Aug 17 12:13:10 storeageserver1 kernel: 1st 0xfffffe000b5f3db8 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1734
Aug 17 12:13:10 storeageserver1 kernel: 2nd 0xfffffe019f945458 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2134
Aug 17 12:13:10 storeageserver1 kernel: KDB: stack backtrace:
Aug 17 12:13:10 storeageserver1 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Aug 17 12:13:10 storeageserver1 kernel: kdb_backtrace() at kdb_backtrace+0x37
Aug 17 12:13:10 storeageserver1 kernel: _witness_debugger() at _witness_debugger+0x2e
Aug 17 12:13:10 storeageserver1 kernel: witness_checkorder() at witness_checkorder+0x807
Aug 17 12:13:10 storeageserver1 kernel: __lockmgr_args() at __lockmgr_args+0xd42
Aug 17 12:13:10 storeageserver1 kernel: vop_stdlock() at vop_stdlock+0x39
Aug 17 12:13:10 storeageserver1 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Aug 17 12:13:10 storeageserver1 kernel: _vn_lock() at _vn_lock+0x47
Aug 17 12:13:10 storeageserver1 kernel: vget() at vget+0x7b
Aug 17 12:13:10 storeageserver1 kernel: vfs_msync() at vfs_msync+0xa5
Aug 17 12:13:10 storeageserver1 kernel: sync_fsync() at sync_fsync+0x12a
Aug 17 12:13:10 storeageserver1 kernel: sync_vnode() at sync_vnode+0x157
Aug 17 12:13:10 storeageserver1 kernel: sched_sync() at sched_sync+0x1d1
Aug 17 12:13:10 storeageserver1 kernel: fork_exit() at fork_exit+0x135
Aug 17 12:13:10 storeageserver1 kernel: fork_trampoline() at fork_trampoline+0xe
Aug 17 12:13:10 storeageserver1 kernel: --- trap 0, rip = 0, rsp = 0xffffff8348c61d00, rbp = 0 ---
```


----------



## benshort (Aug 17, 2011)

I have attached the full output from /var/log/messages

The output from zfs list


```
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
zdata                                   5.54G  3.56T    33K  /zdata
zdata/jira                               414M  3.56T    32K  /zdata/jira
zdata/jira/jira_home                     232M  3.56T   232M  /usr/jails/jira/usr/local/jira/jira_home
zdata/jira/jira_install                  143M  3.56T   143M  /usr/jails/jira/usr/local/jira/jira_install
zdata/jira/pgsql                        39.2M  3.56T    31K  /zdata/jira/pgsql
zdata/jira/pgsql/data                   39.2M  3.56T  39.2M  /usr/jails/jira/usr/local/pgsql/data
zdata/versioncontrol                    5.12G  3.56T    32K  /zdata/versioncontrol
zdata/versioncontrol/respositories      5.12G  3.56T    31K  /zdata/versioncontrol/respositories
zdata/versioncontrol/respositories/git    31K  3.56T    31K  /usr/jails/versioncontrol/repositories/git
zdata/versioncontrol/respositories/svn  5.12G  3.56T  5.12G  /usr/jails/versioncontrol/repositories/svn
zsystem                                 15.4G   101G  5.53G  legacy
zsystem/tmp                             67.4M   101G  67.4M  /tmp
zsystem/usr                             9.79G   101G  2.47G  /usr
zsystem/usr/home                          31K   101G    31K  /usr/home
zsystem/usr/jails                       5.15G   101G   241M  /usr/jails
zsystem/usr/jails/backups               1021M   101G  1021M  /usr/jails/backups
zsystem/usr/jails/jira                  2.04G   101G  2.04G  /usr/jails/jira
zsystem/usr/jails/nfs                    991M   101G   991M  /usr/jails/nfs
zsystem/usr/jails/versioncontrol         934M   101G   934M  /usr/jails/versioncontrol
zsystem/usr/ports                       1.83G   101G  1.54G  /usr/ports
zsystem/usr/ports/distfiles              298M   101G   298M  /usr/ports/distfiles
zsystem/usr/ports/packages                31K   101G    31K  /usr/ports/packages
zsystem/usr/src                          345M   101G   345M  /usr/src
zsystem/var                             2.55M   101G   646K  /var
zsystem/var/crash                       31.5K   101G  31.5K  /var/crash
zsystem/var/db                          1.61M   101G   137K  /var/db
zsystem/var/db/pkg                      1.48M   101G  1.48M  /var/db/pkg
zsystem/var/empty                         31K   101G    31K  /var/empty
zsystem/var/log                          126K   101G   126K  /var/log
zsystem/var/mail                          33K   101G    33K  /var/mail
zsystem/var/run                           64K   101G    64K  /var/run
zsystem/var/tmp                           32K   101G    32K  /var/tmp
```

And the output from zpool status


```
pool: zdata
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	zdata       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0

errors: No known data errors

  pool: zsystem
 state: ONLINE
 scan: none requested
config:

	NAME           STATE     READ WRITE CKSUM
	zsystem        ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0

errors: No known data errors
```


----------



## DutchDaemon (Aug 17, 2011)

Please consider using pastebin.com or pastie.org for pasting large amounts of output. People typically don't want to download and uncompress .tgz type files.


----------



## SirDice (Aug 17, 2011)

FWIW, I just installed 9.0-BETA on a virtualbox VM and got the same sort of lock-order messages while running portsnap. I didn't use ZFS, just the standard GPT/UFS automatic configuration.


----------



## poh-poh (Aug 18, 2011)

Have you seen LOR page, e.g. #280, #260?


----------



## Slurp (Aug 24, 2011)

Got the problem after adding a standalone Nvidia GPU.

```
login: lock order reversal
1st 0xfffffe0006ff3848 filedesc structure (filedesc structure) @ /pcbsd-build90/fbsd-source/9.0/sys/kern/kern_descrip.c:1193
2nd 0xfffffe008b9b5278 zfs (zfs) @ /pcbsd-build90/fbsd-source/9.0/sys/kern/vfs_subr.c:4212
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
_witness_debugger() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
__lockmgr_args() at __lockmgr_args+0xdc6
vop_stdlock() at ..+0x39
VOP_LOCK1_APV() at ..+0x9b
_vn_lock() at ..+0x47
knlist_remove_kq() at ..+0x67
knote_fdclose() at ..+0x177
kern_close() at ..+0xe9
syscallenter() at ..+0x1aa
syscall() at ..0x4c
Xfast_syscall() at ..+0xdd
--- syscall (6,FreeBSD ELF64, close), rip=0x8015972bc, rsp = 0x7fffffffd868, rbp=0x801807ad0
```


----------

