# Possibly silent data corruption happens on standby server with HAST setup?



## belon_cfy (Sep 4, 2013)

Hi,

I'm just curious: will silent data corruption occur on the standby server when running a HAST setup? We can do weekly scrubbing on the active server, however we can't even import the ZFS pool on the secondary server due to the ZFS locking, otherwise bad things may happen after failover. 

Theoretically the standby server will suffer from silent data corruption due to lack of regularly scrubbing, so you may get the corruption data after failover to it, am I right?

If so, is there any way to perform integrity checks on the standby server?


----------



## vanessa (Sep 10, 2013)

Well, theoretically, without scrubbing - yes, silent data corruption might occur.  And if there was any other way for ZFS integrity checking, why would Sun come up with `scrub` ...

Why not just take the servers offline for a couple of minutes every month for scrubbing?


----------



## junovitch@ (Sep 11, 2013)

It would probably be a good idea to periodically shift the HAST primary over to the standby so you can periodically do security patches on the primary as well as know the disaster recovery setup works as planned.  Scheduling a scrub schedule to coincide would be reasonable.  Better to plan for maintenance than have it catch you off guard.  There is a bunch of info about doing a failover with import/export with ZFS on the HAST wiki.  https://wiki.freebsd.org/HAST


----------



## da1 (Sep 12, 2013)

Just one thing. If you scrub the pool on the primary machine and then you do a failover to the secondary machine, wouldn't the same, already scrubbed, pool be imported?


----------



## usdmatt (Sep 12, 2013)

Interesting question



> Why not just take the servers offline for a couple of minutes every month for scrubbing?



I'm guessing you've never done a scrub on any reasonable sized pool of data? It can easily take a day for a scrub to finish on a few TB of data.



> Just one thing. If you scrub the pool on the primary machine and then you do a failover to the secondary machine, wouldn't the same, already scrubbed, pool be imported?



No, when you scrub on the live system, it'll read all the ZFS records from the local disks and compare them against their checksum. There's no guarantee that the copies of those blocks that were written to the slave disks are still intact.

I think the best suggestion here is to look at the possibility of swapping the master/slave  every now and then run a periodic scrub on whichever server happens to be the master at the time. As mentioned this also allows you to make sure the secondary actually works and you can do regular updates/maintenance on both servers while they're not primary.


----------



## vanessa (Sep 12, 2013)

usdmatt said:
			
		

> I'm guessing you've never done a scrub on any reasonable sized pool of data? It can easily take a day for a scrub to finish on a few TB of data.



No, we don't scrub on a few TB, so yes, I don't have that practical experience.



			
				usdmatt said:
			
		

> I think the best suggestion here is to look at the possibility of swapping the master/slave  every now and then run a periodic scrub on whichever server happens to be the master at the time. As mentioned this also allows you to make sure the secondary actually works and you can do regular updates/maintenance on both servers while they're not primary.



If we take this further, wouldn't it be better to always hook the users to the secondary server while scrubbing, making updates and maintenance on the primary server? This way users would get the full performance out of the machine. I suppose that scrubbing, besides time, also needs considerable system resources (CPU, RAM, disk I/O).


----------



## usdmatt (Sep 12, 2013)

I was about to say what a good idea that was, until I realised that the pool is only available on the 'live' machine. Neither HAST or ZFS are designed to handle accesses to the devices on both systems at the same time. You could probably read from the raw disk devices on the server currently acting as slave, but importing the pool so you could scrub would most likely cause havok.


----------



## junovitch@ (Sep 12, 2013)

Let's prove it.  I spun up two VMs with FreeBSD 9.1.  Host vm20 has address 10.100.82.20 and host vm10 has address 10.100.82.10.

On both:  Create some disks.

```
dd if=/dev/zero of=/root/disk1 bs=1M count=100
dd if=/dev/zero of=/root/disk2 bs=1M count=100
```

On both:  A quick /etc/hast.conf

```
resource disk1 {
        on vm20 {
                local /root/disk1
                remote 10.100.82.10
        }
        on vm10 {
                local /root/disk1
                remote 10.100.82.20
        }
}
resource disk2 {
        on vm20 {
                local /root/disk2
                remote 10.100.82.10
        }
        on vm10 {
                local /root/disk2
                remote 10.100.82.20
        }
}
```

On both: Create the HAST disks and start the service

```
hastctl create disk1
hastctl create disk2
service hastd onestart
```

On the primary:  Initialize as the primary role

```
hastctl role primary disk1
hastctl role primary disk2
```

On the secondary:  Initialize as the secondary role

```
hastctl role secondary disk1
hastctl role secondary disk2
```

Back to the primary:  Create a zpool, put some files on it, and scrub it.

```
zpool create zmirror mirror /dev/hast/disk1 /dev/hast/disk2
cp -R /usr/sbin/ /zmirror/
zpool scrub zmirror
zpool status
```

Obviously we are fine at this point, so let's break it.

```
dd if=/dev/zero of=/root/disk1 bs=1M count=75
zpool scrub zmirror
zpool status
... output omitted
    scan: scrub repaired 43.7M in 0h0m with 0 errors on Thu Sep 12 18:26:01 2013
... output omitted
zpool clear zmirror
```

On the secondary:

```
dd if=/dev/zero of=/root/disk1 bs=1M count=75
```

On the primary:

```
zpool scrub zmirror
zpool status
... output omitted
    scan: scrub repaired 0 in 0h0m with 0 errors on Thu Sep 12 18:27:11 2013
```

So no error even though we damaged the secondary's disk.  Lets switch roles.

On the primary:

```
zpool export zmirror
hastctl role secondary disk1
hastctl role secondary disk2
```

On the secondary (soon to be primary)

```
hastctl role primary disk1
hastctl role primary disk2
zpool import zmirror
zpool status
... output omitted
    state: DEGRADED
... output omitted
	    13192269359050368896  UNAVAIL      0     0     0  was /dev/hast/disk1
```

After the import it didn't seem to like the disk1 anymore.  I would guess the loss of HAST metadata since I damaged such a large swath of a small disk had something to do with it.  Let's fix it.

```
hastctl create disk1
hastctl role primary disk1
zpool scrub zmirror
zpool status
... output omitted
    state: ONLINE
    scan: scrub repaired 44.5K in 0h0m with 0 errors on Thu Sep 12 18:33:36 2013
```

Something to keep in mind from https://wiki.freebsd.org/HAST:


> HAST can be compared to a RAID1 (mirror) where one of the components is local disk (on primary node) and second component is disk on remote machine (secondary node). Every write, delete or flush operation (BIO_WRITE, BIO_DELETE, BIO_FLUSH) is sent to local disk and to remote disk over TCP connection (if secondary node is available). Every read operation (BIO_READ) is served from local disk, unless local disk isn't up-to-date or an I/O error occurs, then read operation is sent to secondary node (if it is, of course, available).



In summary, it would appear that since HAST reads only go to the local disk then there is is nothing a `zpool scrub` can do.  Having planned maintenance to regularly switch roles and do a scrub seems required.


----------



## belon_cfy (Sep 13, 2013)

Ok, so to mitigate the only solution is regular switches in between the primary and standby server for scrubbing. 

Thanks for all of your comments and suggestion.


----------

