Here I will go over some suggestions for getting better performance out of ZFS. Some of these suggestions are based solely on other articles, while others are based on personal experience. This article may be edited from time to time based on my increasing experience with ZFS.
One of the advantages that ZFS has is it allows you to have different record sizes in each volume you create. If, for example, you are using a database such as mysql or postgres a record size of 16/8K would likely be optimal as databases tend to writes records of 16/8K. Virtual machines tend to have 4K block sizes and so a 4K record size in ZFS would usually be optimal.
Using ZFS for virtual machines
For performance management purposes and independent rollbacks, I recommend using separate ZFS volumes for each VM and at the very least keep your VM disk images separate from the rest of your system (don't mix things like VMs and databases in the same volume). Assuming that we are dealing with a 4K sector size for your VM's hard disk image, we will create a ZFS volume with a record size of 4K.
# zfs create -o recordsize=4K -o mountpoint=/your/vm/mount pool/vm1
Be careful when using a ZFS swap when using the same pool for your VM. In my experience this will cause your VM performance to be extremely poor. I've experienced that some graphical VMs have load times to the login screen 50-100x slower. This occurred with the swappiness turned down. Use swap with caution.
Using ZFS for databases
When using databases such as mysql or postgres it is ideal to limit the number of costly rewrites to disk. If your record size on you databases Zvol is larger than the page size your database is using, you could end up having a higher IO than is necessary. For mysql the default page size is 16K and postrges is 8K. So, before you install your database, create your volume with the appropriate record size.
Mysql
# zfs create -o recordsize=16K -o mountpoint=/var/lib/mysql pool/mysql
If you are doing this on a fresh mysql install, start mysql and have fun.
If you already have installed and started using mysql then change the mount point to another location to start. I will use /mnt/mysql. With that mounted and mysql off, copy your database directory to the mounted Zvol with cp or rsync. I prefer rsync. Once you finish copying that and you are sure that the copy is the same, delete the contents of the original mysql folder. Finally, change the mount on the Zvol.
# rsync -rp /var/lib/mysql/* /mnt/mysql
# rm -rf /var/lib/mysql/*
# zfs set mountpoint=/var/lib/mysql pool/mysql
Start mysql
For postgres make the recordzise=8K instead.