On Wed, Jun 17, 2015 at 3:46 PM, Bert Huijben <bert_at_qqmail.nl> wrote:
>
>
> > -----Original Message-----
> > From: stefan2_at_apache.org [mailto:stefan2_at_apache.org]
> > Sent: woensdag 17 juni 2015 12:09
> > To: commits_at_subversion.apache.org
> > Subject: svn commit: r1685985 - in
> /subversion/branches/fsx-1.10/subversion:
> > include/private/svn_mutex.h libsvn_fs_x/batch_fsync.c
> > libsvn_fs_x/batch_fsync.h libsvn_fs_x/fs.c libsvn_subr/mutex.c
> > tests/libsvn_fs_x/fs-x-pack-test.c
> >
> > Author: stefan2
> > Date: Wed Jun 17 10:09:12 2015
> > New Revision: 1685985
> >
> > URL: http://svn.apache.org/r1685985
> > Log:
> > On the fsx-1.10 branch:
> > Introduce a new infrastructure to FSX that allows us to do efficient
> fsyncs.
> >
> > It basically uses a thread pool to execute multiple fsyncs concurrently.
> > Interestingly, this generic implementation is faster on Linux than even
> the
> > POSIX-provided aio_fsync functionality on the same system. As a
> centralized
> > mechanism for scheduling fsyncs it also takes care of preventing
> redundant
> > flushes.
> >
> > With this commit, FSX does not actually use the new capabilities. That
> will
> > be in the following commits.
>
> Would it be possible to implement this on filehandles on Windows, instead
> of just on filenames.
>
That is actually what I'm doing. It may not be obvious
from the interface but the usage is such that files will
be opened only once and fsync'ed through that handle.
The only exception so far is the 'current' file which needs
to be fsync'ed twice (before and after the rename). But
I plan to eliminate the extra fsync even there.
> Reopening a file that has just been closed is typically not fast on
> Windows, as virusscanners and file indexers are often fighting to use the
> same files. The next open operation then sometimes has to wait (when using
> OPLOCKS) or has to be retried (when the other process opens the file in a
> way that concurrent writes are denied).
>
I may not have answered your post when you wrote
that a couple of weeks ago but I heard you and used
it as input for the batch fsync design ;)
> All of this can be avoided by just flushing the filehandle we already had
> when the file was open for writing. By closing and re-opening we are doing
> a lot of unneeded work and add a lot of potential race conditions.
>
One key design element is that the batch fsync
container owns the open files and will open them
only once. So, it can not only guarantee that we
don't need to reopen files for fsync but it can even
prevent reopening files in cases where different
functions were to open & close the same as part
of some bigger functionality.
That said, this is all very much work-in-progress
and the final logic may be quite different again.
-- Stefan^2.
Received on 2015-06-18 21:58:43 CEST