Re: Efficient and effective fsync during commit

Stefan Fuhrmann <stefan.fuhrmann_at_wandisco.com> writes:

> In the future, we should implement step 1 as simple
> non-fsync'ing file operations. Then explicitly sync every
> file, and on POSIX the folders, once. Step 2 does not
> have any atomicity requirements. Finally, do the 'current'
> rename. This also only requires a single fsync b/c
> the temp file will be in the same folder.
>
> On top of that, all operations in step 2 can be run
> concurrently. I did that for FSX on Linux using aio_fsync
> and it got 3x as fast. Windows can do something similar.
> I wrapped that functionality into a "batch_fsync" object
> with a few methods on it. You simply push paths into it,
> it drops duplicates, and finally you ask it to fsync all.

fsync() works on file descriptors rather than files, do we need to keep
the original file descriptors open in order to fsync()? The POSIX
description is

http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html

  The fsync() function shall request that all data for the open file
  descriptor named by fildes is to be transferred to the storage device
  associated with the file described by fildes. The nature of the
  transfer is implementation-defined.

It probably is feasible to keep the file descriptors open, provided we
don't accumulate too many.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*