# Weird errors after upgrading to 10.2-STABLE



## xy16644 (Aug 22, 2015)

Hi All

I have just upgraded my 10.1-STABLE server to 10.2-STABLE. All daemons are running and everything appears to be working...except for some strange boot up errors I get (which I didn't before).

If I look in /var/log/console.log I get the following errors:


```
Aug 22 13:52:25 mail kernel: Clearing /tmp.
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/conftest3824: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/.ICE-unix: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_3.png: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_3_err.png: Read-only file syste
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_2_err.png: Read-only file syste
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_2.png: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_0_err.png: Read-only file syste
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_1.png: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_1_err.png: Read-only file syste
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi/mailgraph_0.png: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph/,cgi: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/mailgraph: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22/apcaccess.core: Read-only file system
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot/2015.08.22: Operation not supported
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/snapshot: Operation not supported
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs/shares: Operation not supported
Aug 22 13:52:25 mail kernel: rm: /tmp/./.zfs: File exists
```

I've always cleared my /tmp in the past successfully so why now the problem?

After these errors I get the following:

```
Aug 22 13:52:32 mail kernel: Starting clamav_clamd.
Aug 22 13:52:32 mail kernel: ERROR: /var/log/clamav/clamd.log is locked by another process
Aug 22 13:52:32 mail kernel: ERROR: Can't initialize the internal logger
Aug 22 13:52:32 mail kernel: /etc/rc: WARNING: failed to start clamav_clamd
Aug 22 13:52:32 mail kernel: Starting clamav_freshclam.
Aug 22 13:52:32 mail kernel: Waiting for clamd socket.. 60..
Aug 22 13:52:34 mail kernel: Starting clamav_milter.
Aug 22 13:52:34 mail kernel: Waiting for clamav-milter socket..
Aug 22 13:52:34 mail kernel: Starting miltergreylist.
Aug 22 13:52:34 mail kernel: Starting milteropendkim.
Aug 22 13:52:34 mail kernel: Starting mysql.
Aug 22 13:52:36 mail kernel: Starting dovecot.
Aug 22 13:52:36 mail kernel: Fatal: Dovecot is already running with PID 1111 (read from /var/run/dovecot/master.pid)
Aug 22 13:52:36 mail kernel: /etc/rc: WARNING: failed to start dovecot
Aug 22 13:52:36 mail kernel: postfix/postfix-script: fatal: the Postfix mail system is already running
Aug 22 13:52:36 mail kernel: Performing sanity check on openssh configuration.
Aug 22 13:52:36 mail kernel: Starting openssh.
Aug 22 13:52:36 mail kernel: Starting munin_node.
Aug 22 13:52:36 mail kernel: Pid_file already exists for running process (1138)... aborting
Aug 22 13:52:36 mail kernel: /etc/rc: WARNING: failed to start munin_node
Aug 22 13:52:37 mail kernel: Starting cron.
Aug 22 13:52:37 mail kernel: cron: cron already running, pid: 1093
Aug 22 13:52:37 mail kernel: /etc/rc: WARNING: failed to start cron
```

and I just noticed these too:


```
Aug 22 13:52:17 mail kernel: Starting file system checks:
Aug 22 13:52:17 mail kernel: Mounting local file systems:.
Aug 22 13:52:17 mail kernel: rm: apcupsd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: clamav-milter.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: clamd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: clamd.sock: Read-only file system
Aug 22 13:52:17 mail kernel: rm: clmilter.sock: Read-only file system
Aug 22 13:52:17 mail kernel: rm: freshclam.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: cron.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: devd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: devd.pipe: Read-only file system
Aug 22 13:52:17 mail kernel: rm: devd.seqpacket.pipe: Read-only file system
Aug 22 13:52:17 mail kernel: rm: dkim-filter.sock: Read-only file system
Aug 22 13:52:17 mail kernel: rm: dmesg.boot: Read-only file system
Aug 22 13:52:17 mail kernel: rm: anvil: Read-only file system
Aug 22 13:52:17 mail kernel: rm: anvil-auth-penalty: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-client: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-login: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-master: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-token-secret.dat: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-userdb: Read-only file system
Aug 22 13:52:17 mail kernel: rm: auth-worker: Read-only file system
Aug 22 13:52:17 mail kernel: rm: config: Read-only file system
Aug 22 13:52:17 mail kernel: rm: dict: Read-only file system
Aug 22 13:52:17 mail kernel: rm: director-admin: Read-only file system
Aug 22 13:52:17 mail kernel: rm: dns-client: Read-only file system
Aug 22 13:52:17 mail kernel: rm: doveadm-server: Read-only file system
Aug 22 13:52:17 mail kernel: rm: dovecot.conf: Read-only file system
Aug 22 13:52:17 mail kernel: rm: imap-urlauth: Read-only file system
Aug 22 13:52:17 mail kernel: rm: imap-urlauth-worker: Read-only file system
Aug 22 13:52:17 mail kernel: rm: indexer: Read-only file system
Aug 22 13:52:17 mail kernel: rm: indexer-worker: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ipc: Read-only file system
Aug 22 13:52:17 mail kernel: rm: lmtp: Read-only file system
Aug 22 13:52:17 mail kernel: rm: log-errors: Read-only file system
Aug 22 13:52:17 mail kernel: rm: imap: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ipc-proxy: Read-only file system
Aug 22 13:52:17 mail kernel: rm: login: Read-only file system
Aug 22 13:52:17 mail kernel: rm: sieve: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-params: Read-only file system
Aug 22 13:52:17 mail kernel: rm: master.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replication-notify: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replication-notify-fifo: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replicator: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-params: Read-only file system
Aug 22 13:52:17 mail kernel: rm: master.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replication-notify: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replication-notify-fifo: Read-only file system
Aug 22 13:52:17 mail kernel: rm: replicator: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-params: Read-only file system
Aug 22 13:52:17 mail kernel: rm: stats: Read-only file system
Aug 22 13:52:17 mail kernel: rm: stats-mail: Read-only file system
Aug 22 13:52:17 mail kernel: rm: imap-urlauth: Read-only file system
Aug 22 13:52:17 mail kernel: rm: tokenlogin: Read-only file system
Aug 22 13:52:17 mail kernel: rm: httpd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ld-elf.so.hints: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ld-elf32.so.hints: Read-only file system
Aug 22 13:52:17 mail kernel: rm: milter-greylist.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: monit.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: monitorix.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: moused.ums0.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: mpm-accept.24388: Read-only file system
Aug 22 13:52:17 mail kernel: rm: mrtg.ok: Read-only file system
Aug 22 13:52:17 mail kernel: rm: munin-node.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ntpd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: rewrite-map.24388: Read-only file system
Aug 22 13:52:17 mail kernel: rm: spamass-milter.sock: Read-only file system
Aug 22 13:52:17 mail kernel: rm: spamd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: sshd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: sshguard.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-cache.24387: Read-only file system
Aug 22 13:52:17 mail kernel: rm: rewrite-map.24388: Read-only file system
Aug 22 13:52:17 mail kernel: rm: spamass-milter.sock: Read-only file system
Aug 22 13:52:17 mail kernel: rm: spamd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: sshd.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: sshguard.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-cache.24387: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-stapling-refresh.24387: Read-only file system
Aug 22 13:52:17 mail kernel: rm: ssl-stapling.24387: Read-only file system
Aug 22 13:52:17 mail kernel: rm: syslog.pid: Read-only file system
Aug 22 13:52:17 mail kernel: rm: syslogd.sockets: Read-only file system
Aug 22 13:52:17 mail kernel: rm: utx.active: Read-only file system
```

Again, why do these daemons fail to start when booting up when they always started up just fine with 10.1?

Because I run Monit all the daemons auto started themselves after the server was booted up and are running fine but I would like to understand why the above errors are occurring and what I can do to fix them.

Can anyone help please?

Thanks!

Edit: Interestingly, if I do an `ls /tmp` I get:


```
(/support)$ ls /tmp/
total 160
drwxrwxrwt  9 root  wheel  16B Aug 22 16:20 ./
drwxr-xr-x  20 root  wheel  28B Aug 22 14:52 ../
drwxrwxrwt  2 root  wheel  2B Aug 22 13:52 .ICE-unix/
drwxrwxrwt  2 root  wheel  2B Aug 22 13:52 .X11-unix/
drwxrwxrwt  2 root  wheel  2B Aug 22 13:52 .XIM-unix/
drwxrwxrwt  2 root  wheel  2B Aug 22 13:52 .font-unix/
ls: /tmp//.zfs: Operation not supported
dr-xr-xr-x  4 root  wheel  4B Jan  1  2014 .zfs/
drwxr-xr-x  3 www  wheel  3B Aug 22 13:54 mailgraph/
srwxrwxrwx  1 mysql  wheel  0B Aug 22 13:52 mysql.sock=
-rw-------  1 www  wheel  0B Aug 22 13:54 sess_1rs9qug6iltjhh87mhu5v8hf77
-rw-------  1 www  wheel  0B Aug 22 13:53 sess_8h91gg58ov45035ch631ii1v77
-rw-------  1 www  wheel  0B Aug 22 15:53 sess_a3fbn636q0i8qhv561kpjnaur7
-rw-------  1 www  wheel  0B Aug 22 13:59 sess_d1n8hrafcp7srlja4pg0hfgcd2
-rw-------  1 www  wheel  0B Aug 22 13:59 sess_f6jbeshmk5t4eiqeqalsusldp0
-rw-------  1 www  wheel  0B Aug 22 14:18 sess_jql0a07r688r50mp295ner9h05
-rw-------  1 www  wheel  0B Aug 22 15:52 sess_upilqbqn9rctdc9rjmfa55brq5
drwx------  2 xy16644  wheel  3B Aug 22 14:29 ssh-2VmxJYy4Up/
```


----------



## xy16644 (Aug 24, 2015)

Anyone? ;-)


----------



## Kiiski (Aug 25, 2015)

Just a long shot, but assuming you did not use `freebsd-update`, did you remember to do this during build world process:

`# zfs set readonly=off zroot
# zfs mount -a`

as described here:

https://www.freebsd.org/doc/handbook/makeworld.html


----------



## xy16644 (Aug 25, 2015)

Kiiski said:


> Just a long shot, but assuming you did not use `freebsd-update`, did you remember to do this during build world process:
> 
> `# zfs set readonly=off zroot
> # zfs mount -a`
> ...



I did not use `freebsd-update` as I run STABLE. The exact steps I followed when upgrading is as follows:


```
rm -rf /usr/src

svn co svn://svn.freebsd.org/base/stable/10/usr/src

cd /usr/src
make -j `sysctl -n hw.ncpu` buildworld
make -j `sysctl -n hw.ncpu` buildkernel
make installkernel
shutdown -r now
cd /usr/src
mergemaster -p
make installworld
mergemaster -iUF
yes | make delete-old
yes | make delete-old-libs
cd /usr/obj && chflags -R noschg * && rm -rf *
shutdown –r now
```

These are the steps I have been using to upgrade from 9.3 to 10.0 to 10.1 and now to 10.2 (all STABLE) and I have never had any issues except when upgrading to 10.2. Why is it when I do an `ls` on root or /tmp that I get:


```
ls: /tmp//.zfs: Operation not supported
```

I know I have the .zfs directories enabled but I have never had an issue viewing them before when using `ls`.

I also can't understand all the errors when booting up? I don't have a read-only file system on zroot (my ZFS on root) so whats going on here?

Just really baffled by all this!


----------



## monwarez (Aug 26, 2015)

Maybe you should do `make installworld` in single user mode. I hear that it was recommended to do it in single user mode.


----------



## kpa (Aug 26, 2015)

monwarez said:


> Maybe you should do `make installworld` in single user mode. I hear that it was recommended to do it in single user mode.



It won't make a difference unless you're using securelevel(7) to secure your system. In that case it's impossible to write over files that are protected with schg fileflags and you have to boot to single user mode to perform `make installworld`.


----------



## xy16644 (Aug 26, 2015)

Thanks for the replies. I have never used single user mode when upgrading FreeBSD. I've also been using ZFS for a few years now so I am baffled still as to WHY I am getting errors about a read-only file system? I also don't understand the "Operation not supported" warning?


----------

