Re: Race in svn_atomic_namespace__create

Stefan Fuhrmann <stefan.fuhrmann_at_wandisco.com> writes:

> Maybe, there should be a regression test that
> tries concurrent initialization.

There are two objects here: the named file and the shared memory
segment. The usual sequence for a single process is: create file,
create segment, delete segment, delete file. There are two windows when
the file exists and the segment does not and at that point a second
process can neither attach nor create. The create window is protected
by the inter-process lock but I think the delete window is not protected
as it is triggered by a pool cleanup. We need to find a way to either
take the inter-process lock or introduce some sort of delay/retry.

However I think there is a bigger problem: I don't think this use of APR
named memory allows processes to start and stop in an unordered fashion.
I've built APR 1.4.x and configure chooses:

  apr.h:#define APR_USE_SHMEM_MMAP_TMP 0
  apr.h:#define APR_USE_SHMEM_MMAP_SHM 0
  apr.h:#define APR_USE_SHMEM_MMAP_ZERO 0
  apr.h:#define APR_USE_SHMEM_SHMGET_ANON 0
  apr.h:#define APR_USE_SHMEM_SHMGET 1
  apr.h:#define APR_USE_SHMEM_MMAP_ANON 1
  apr.h:#define APR_USE_SHMEM_BEOS 0

Start a process, svnadmin in my case, and apr_shm_attach fails as this
is the first process so apr_shm_create is run. This creates the named
file and the shared memory segment. The apr call installs a cleanup
handler shm_cleanup_owner. At this point I see the shared memory
segment:

$ ipcs -m | grep 7340039
0x0101e244 7340039 pm 600 65536 1

and the process using it:

$ lsof | grep 7340039
lt-svnadm 24614 pm DEL REG 0,4 7340039 /SYSV0101e244

Start a second process, this time the apr_shm_attach works and the two
processes use the same shared memory segment. At this point I see two
processes using the shared memory segment:

$ lsof | grep 7340039
lt-svnadm 24614 pm DEL REG 0,4 7340039 /SYSV0101e244
lt-svnadm 24623 pm DEL REG 0,4 7340039 /SYSV0101e244

Allow the first process to exit. The cleanup handler detaches from the
shared memory removes the named file. The second process still uses the
segment but the named file is deleted. At this point the shared memory
segment key has changed:

$ ipcs -m | grep 7340039
0x00000000 7340039 pm 600 65536 1 dest

but the second process is still using it:

lt-svnadm 24623 pm DEL REG 0,4 7340039 /SYSV0101e244

Start a third process, the apr_shm_attach fails because the file is
removed. So this process creates a new named file and a second shared
memory segment. At this point I see two shared memory segments:

$ ipcs -m | egrep '(7340039|7372808)'
0x00000000 7340039 pm 600 65536 1 dest
0x0101e245 7372808 pm 600 65536 1

and the second and third processes are using different segments:

$ lsof | egrep '(7340039|7372808)'
lt-svnadm 24623 pm DEL REG 0,4 7340039 /SYSV0101e244
lt-svnadm 24637 pm DEL REG 0,4 7372808 /SYSV0101e245

That's a serious problem, the two processes are not longer using the
same shared memory segment to keep in sync. Changes made by one process
won't be visible to another.

-- 
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download