AW: failure of creating mutex file on pending delete

Thank you very much for the exhaustive response. Very interesting to get to know the background.

My repository is already sharded. A „svnadmin pack“ command is excecuted in the post-commit-hook.

I publish the repositories under two apache locations. One for internal users with unlimited access. One for external users with per directory access. I observed the error under the access fort he internal unlimited users.

My directories are not that large. Most are very small, maximum is about 100 entries.

Files are nigthly backuped + every view minutes a svn synch.

The repository is already in 1.8 format. I converted it when 1.8 came out.

I disabled revprop caching this gave tortoisesvn repo browser a performance boost. (on the first open of repo browser, no need to reopen it)

Excellent! Thank you very much.

Matthias

Von: Stefan Fuhrmann [mailto:stefan.fuhrmann_at_wandisco.com]
Gesendet: Montag, 10. November 2014 15:11
An: Matthias Ludwig
Cc: Subversion Development
Betreff: Re: failure of creating mutex file on pending delete

On Sat, Nov 8, 2014 at 1:32 AM, Matthias Ludwig <matthias-ludwig_at_gmx.net <mailto:matthias-ludwig_at_gmx.net> > wrote:

He there,

I don't have any clue oft he development of svn, nevertheless I try to make some assumptions. Please correct me if I'm wrong.

I might obverved a problem:

I'm usging Apache mod_dav_svn 1.8.10 on windows 7.

In my apache error log files I found the errors:

(20014)Internal error: -Can't open file 'C:\\ServerData\\svn\\repos\\db\\rev-prop-atomics.mutex': access denied
(20014)Internal error: Revprop caching for 'C:/ServerData/svn/repos/db' disabled because SHM infrastructure for revprop caching failed to initialize.

Yes, that is a known issue on Windows. It results in slightly
degraded performance further down the road. Disable revprop
caching on the server side and the error message will disappear.
 

They appear when I open TortoiseSvn-Repo-Browser (which means that a lot of request are send tot he server. (maybe even parallel? I don't know)

The problem occurs when multiple connections are being opened
at the same time that need access to time stamps and user names.
This happens e.g. during checkout via serf or with TSVN's repo
browser because it, too, opens multiple connections.

The apache run under local System account. I've checked the file permissions on all repo-files for user SYSTEM. Everything is ok.

Then I used "Process Monitor" from Sysinternals to check the file access on the file above (http://technet.microsoft.com/en-us/sysinternals/bb842062.aspx)
I saw hundreds of failures of file creation, with the result "DELETE PENDING"

20:08:43,2589409 httpd.exe 3728 CreateFile C:\ServerData\svn\repos\db\rev-prop-atomics.mutex DELETE PENDING Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-

Googling around I found out that an attempt to create a file with a "Pending Delete" can lead to an "access denied" result. (e.g. http://stackoverflow.com/questions/3764072/c-win32-how-to-wait-for-a-pending-delete-to-complete)

Is there a relation to the error-Log-Message? The timing said so.

I suppose the create/lock/unlock/delete command on that file are used for synchronization.

It is a Windows-specific initialization race.

What are the performance costs of synchronization with the file? When I I see the error file creation failure I get the feeling there is a performance loss there. The Tortoise-SVN-Repo browser in our setting is quit slow. Is there a relation?

The repo browser runs several small requests ('svn ls', 'svn pg'),

hence the init failure may incur a large extra cost. The penalty
after the init failure should only become visible for directories with

several 1000 entries in them.

Here is what you should do. Disable revprop caching. It is broken
in 1.8, particularly on Windows. Once that is done, open the repo
browser, open a few folders in it, close the repo browser, reopen
it and open to the same folders again.

If the second run is still slow, it may be due to path-based access
control, i.e. not all users have access to all parts of the repo.

If the second run is significantly faster, the repositories may be

fragmented. In that case,

* Make sure your backups are up-to-date. You probably won't

  need them but this is a good time to check on them.

* Make sure all servers, tools and scripts that will access the repo
  can handle svn 1.8 repositories.
* Check whether it is "sharded". Under %repo%/db/revs, there

  must be subfolders such as "0", "1000", ... etc. or "0.pack" etc.

  If those don't exist and all revision files are flat directly in db/revs,

  your best option is to dump & load the repo and then continue

  with the next step.

* Upgrade the repository to 1.8 with 'svnadmin upgrade'.

* Pack the repository with 'svnadmin pack'.
 

Maybe a named mutex of he Win32-API would be the better choice for synchonization because they are made for that reason. (http://msdn.microsoft.com/en-us/library/windows/desktop/ms682411(v=vs.85).aspx)

There is a fix for the init race but then we discovered that the
inherent restrictions (repositories not shared between server
machines, all server processes using the consistent revprop
cache settings) have not been communicated. Moreover, we

decided that those restrictions make the feature too fragile.

As a result, it will (very likely) be disabled in 1.8.11 and 1.9.

A revised implementation has been proposed for 1.10. In 1.9,
the code has been tuned to minimize the performance penalty

of not having revprop caching.

-- Stefan^2.

 
Received on 2014-11-10 15:49:39 CET