I'M seeing strange performance issues in ZFS transfers between pools

Question

I'M seeing strange performance issues in ZFS transfers between pools

of similar transfer speeds (roughly 300MB/s sequential, bottlenecked by the network) and bursts of expected 100MB/s followed by serious and long dips into KB/s speeds. I think that on one of the ends there might be caching involved. Is it possible to bypass such a cache in ZFS and transfer more synchronously? As far as I'm aware ZFS has an intent log for this (there are no SLOG devices on either end though), could this be related? I'm also seeing this issue only on some pools but not others so maybe it'd be the drives after all? Or fragmentation? I honestly have no idea where to start looking for this, only that it's not the first time this has been happening. For larger transfers like this one it causes them to take a lot longer than they should. It also seems to happen only on large transfers - up to 1GB or so the issue doesn't appear anywhere.
Edit: FWIW, each "burst" seems to be around 400MB in size.

0

20.12.2019