Investigate O_SYNC / fsync effects; update counters during buffer flush

a-j-wood commented

2022-05-29 12:25:52 +00:00

(Migrated from github.com)

See what happens if we fsync() and/or use O_SYNC, which would be useful when writing to USB devices or other things that tend to hang for ages after we've finished.

a-j-wood commented

2022-05-29 14:10:05 +00:00

(Migrated from github.com)

Correspondence from 2018:

From: Andrew Wood
To: Kalin KOZHUHAROV me.kalin@gmail.com
Subject: Re: Message from Contact Form (PV)

Thanks for your email. Sorry for the late reply.

PV is not aware of what's happening when you see this, because in this case
the operating system is buffering the data, so PV thinks it's finished. So
PV will be writing to the output, but it's not really going direct to the
disk, it's going to an area of memory that the operating system maintains.
The operating system then handles the slow writing of data out of that area
to the actual disk. When PV goes to close its output, when it thinks it's
finished, the operating system makes PV wait until the operating system's
write buffer is empty.

One way around this would be to add an option to PV which tells it to set
the output to bypass all operating system buffering. This would decrease
performance, but increase accuracy because none of the activity described
above would happen any more.

This is on the TODO list as "option to enable O_DIRECT" so it might appear
in a future version.

Sent from: Kalin KOZHUHAROV

Hello,
I have been using pv for hm.. 10 years? for various things, mostly processing large files. Great!
One thing that sometimes bother me is that with slow output (e.g. redirecting to a file on slow disk), the counters reach 100% and stop while the output is still "in flight".
Will it be possible to continue updating the counters (you can leave the 100% out) while the pv is waiting to exit?
Initially I thought of simply issuing sync(1) and waiting, but there is no guarantee the next in pipeline is doing diskio.

Correspondence from 2018: From: Andrew Wood To: Kalin KOZHUHAROV <me.kalin@gmail.com> Subject: Re: Message from Contact Form (PV) Thanks for your email. Sorry for the late reply. PV is not aware of what's happening when you see this, because in this case the operating system is buffering the data, so PV thinks it's finished. So PV will be writing to the output, but it's not really going direct to the disk, it's going to an area of memory that the operating system maintains. The operating system then handles the slow writing of data out of that area to the actual disk. When PV goes to close its output, when it thinks it's finished, the operating system makes PV wait until the operating system's write buffer is empty. One way around this would be to add an option to PV which tells it to set the output to bypass all operating system buffering. This would decrease performance, but increase accuracy because none of the activity described above would happen any more. This is on the TODO list as "option to enable O_DIRECT" so it might appear in a future version. > Sent from: Kalin KOZHUHAROV > > Hello, > I have been using pv for hm.. 10 years? for various things, mostly processing large files. Great! > One thing that sometimes bother me is that with slow output (e.g. redirecting to a file on slow disk), the counters reach 100% and stop while the output is still "in flight". > Will it be possible to continue updating the counters (you can leave the 100% out) while the pv is waiting to exit? > Initially I thought of simply issuing sync(1) and waiting, but there is no guarantee the next in pipeline is doing diskio.

a-j-wood commented

2022-05-29 14:38:58 +00:00

(Migrated from github.com)

Additional correspondence - May 2022

Sent from: Andrew

When I use pv to write an iso file to USB, it goes to 100% in 2 seconds followed by many minutes of waiting before the process finishes and I return to my terminal prompt.

sudo sh -c "pv /mnt/somepath/to/the/image.iso > /dev/sda"
1.19GiB 0:00:02 [ 535MiB/s] [=====================================================================>] 100%

arch linux 5.17.7

Additional correspondence - May 2022 Sent from: Andrew When I use pv to write an iso file to USB, it goes to 100% in 2 seconds followed by many minutes of waiting before the process finishes and I return to my terminal prompt. sudo sh -c "pv /mnt/somepath/to/the/image.iso > /dev/sda" 1.19GiB 0:00:02 [ 535MiB/s] [=====================================================================>] 100% pv --version pv 1.6.20 - Copyright 2015 Andrew Wood <andrew.wood@ivarch.com> arch linux 5.17.7

a-j-wood commented

2023-07-30 16:36:26 +00:00

(Migrated from github.com)

This is partially covered by the new "--sync" and "--direct-io" options introduced in 1.7.17, but that will still lock up pv's display if the disk write takes a long time.

This is partially covered by the new "`--sync`" and "`--direct-io`" options introduced in 1.7.17, but that will still lock up `pv`'s display if the disk write takes a long time.

Investigate O_SYNC / fsync effects; update counters during buffer flush #6