# Throttling TAR disk IO



## PacketMan (Oct 25, 2020)

When I run tar(1) to read files on one disk, and write the .tar to another disk the IO is pretty much 100%. If I wanted to throttle that, say for example 50%, how would I do it? I am using tar(1) inside a script.  Any particular URLs someone can refer me to?

Thanks again everyone.


----------



## jb_fvwm2 (Oct 25, 2020)

I'd switch to rsync:  --bwlimit=700 [ slow background ] ... --bwlimit=7000 [ almost full speed ]


----------



## PacketMan (Oct 25, 2020)

Yeah I was looking at it, scrolling down through its various options, but I not considering using it yet. Looks like TAR(1) doesn't have any options to control IO usage.


----------



## T-Daemon (Oct 25, 2020)

There is this article, using sysutils/pv to slow down tar:





__





						Slowing down (limiting) tar, mysqldump or other processes to save IO bandwidth - Closer to Code
					

Sometimes we want to perform some sort of tasks that consume whole available IO bandwidth. This may lead to some unexpected behaviours from our OS. OS might even kill the given process due to resource lack. Lets take an example. We want to tar.gz a huge directory with a lot of files in it. Our […]




					mensfeld.pl


----------



## ralphbsz (Oct 25, 2020)

I don't think you can. Individual programs can limit how much IO they do; jb_fvwm2 gave an example for rsync. There is no OS-wide facility for limiting/throttling the IO of a particular process, nor is there an IO priority system and scheduler. For CPU usage, there are such things as priorities, the nice command, the whole mechanism behind it, not for IO.

But let me ask another question: Why do you care? What's wrong with tar using 100% of the disk? Matter-of-fact, you have to remember that spinning disks are one of the few things in the universe that get more efficient the more overloaded they are. From an overall throughput standpoint, you really want your tar to run as fast as possible, even if that's an inconvenience for other processes that use the same disk.


----------



## T-Daemon (Oct 25, 2020)

T-Daemon said:


> using sysutils/pv to slow down tar:


Here is another one: misc/cstream , cstream(1).









						Backup: bandwidth throttling with cstream (#4524) · Issues · ISPConfig / ISPConfig 3 · GitLab
					

I noticed last night that my disk IO usage was 100% and server was hardly working. I saw the ispconfig backup is to blame. To fix...




					git.ispconfig.org


----------



## PacketMan (Oct 25, 2020)

ralphbsz said:


> But let me ask another question: Why do you care? What's wrong with tar using 100% of the disk?


Well, perhaps I have other files on one of those disks and using those files become sluggish. Yes I would normally run TAR after hours, but sometimes I don't or can't.


----------



## eldaemon (Oct 25, 2020)

Sure, you can throttle tar. Put something in a pipe with it.

`tar cf - . | slowitdown > foo.tar`

`slowitdown` could be `xz` or an actual pipe throttler. You can use `pv` to show the speed and verify if something is indeed slowing it down. But note that you would want to put `pv` before the compressor and not after.


----------



## eldaemon (Oct 25, 2020)

T-Daemon said:


> There is this article, using sysutils/pv to slow down tar:
> 
> 
> 
> ...


Ah, nice! I had no idea `pv` supported that. Definitely the way to go for a pipe-based solution.


----------



## PacketMan (Oct 25, 2020)

Thanks everyone.


----------

