Hi all,
we have an Oracle Solaris 11.3 x86_64-based server with the following components:
- Apache 2.4.12(-0.175.3.0.0.30.0)
- SVN 1.7.20(-5.12.0.0.0.91.0)
SVN is connected to Apache with "mod_dav_svn" and I have a repository with a "post-commit" hook.
The Apache package is the original package with comes with Solaris 11.3, while SVN is compiled on a Solaris build host by ourselves because Oracle missed to include 64 bit binaries in Solaris 11.3 and they are needed when one tries to include "mod_dav_svn.so" into a 64 bit version of the Apache web server, of course. But this shouldn't be the cause on any problem.
My problem is: The whole configuration is running absolutely fine as long as Apache is configured in the "prefork" MPM. As soon as I configure it to the "worker" mode (i.e. multi-threaded mode), the Apache process hangs whenever I want so commit a file to one of my repositories, but this problem only occurs as long as a "post-commit" hook is defined. Maybe other kinds of hooks are affected as well, I haven't tried them, yet.
It seems for my that "mod_dav_svn" itself might be able to work in a multi-threaded environment, but the connection to the hooks doesn't.
I've used the tool "truss" to capture the syscalls which are called by Apache in both situations:
a) MPM Prefork
| 25769: stat("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit", 0xFFFF80F48B165B40) = 0
| 25769: open("/dev/null", O_WRONLY|O_CLOEXEC) = 24
| 25769: fcntl(24, F_DUPFD, 0x00000000) = 25
| 25769: fcntl(25, F_GETFD, 0x7FFA9304E144) = 0
| 25769: fcntl(25, F_SETFD, 0x00000000) = 0
| 25769: pipe() = 26 [27]
| 25769: fcntl(26, F_GETFD, 0x99427F850) = 0
| 25769: fcntl(26, F_SETFD, 0x00000001) = 0
| 25769: forkx(0) = 25782
| 25769: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25769: close(25) = 0
| 25769: close(27) = 0
| 25782: forkx() (returning as child ...) = 25769
| 25782: getpid() = 25782 [25769]
| 25782: munmap(0x7FFA91DC0000, 262144) = 0
| 25782: lwp_self() = 1
| 25782: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25782: close(23) = 0
| 25782: close(22) = 0
| 25782: close(21) = 0
| 25782: schedctl() = 0x7FFA92243000
| 25782: munmap(0x7FFA91A80000, 288862) = 0
| 25782: munmap(0x7FFA91AD7000, 7936) = 0
| 25782: munmap(0x7FFA91A60000, 29785) = 0
| 25782: munmap(0x7FFA91A78000, 2792) = 0
| 25782: close(5) = 0
| 25782: close(3) = 0
| 25782: close(6) = 0
| 25782: close(4) = 0
| 25782: close(14) = 0
| 25782: close(13) = 0
| 25782: close(12) = 0
| 25782: close(11) = 0
| 25782: close(10) = 0
| 25782: close(9) = 0
| 25782: close(8) = 0
| 25782: close(20) = 0
| 25782: close(26) = 0
| 25782: close(24) = 0
| 25782: fcntl(25, F_DUP2FD, 0x00000001) = 1
| 25782: close(25) = 0
| 25782: fcntl(27, F_DUP2FD, 0x00000002) = 2
| 25782: close(27) = 0
| 25782: sigaction(SIGCLD, 0xFFFF80F48B165A90, 0xFFFF80F48B165B20) = 0
| 25782: chdir(".") = 0
| 25782: execve("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit", 0x9942814C8, 0xFFFF80F48B165B78) argc = 4
| 25782: sysinfo(SI_MACHINE, "i86pc", 257) = 6
| 25782: mmap(0x00000000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFF80FFBF760000
| 25782: memcntl(0xFFFF80FFBF792000, 110504, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
| 25782: memcntl(0x00400000, 172672, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
| 25782: resolvepath("/usr/bin/amd64/ksh93", "/usr/bin/amd64/ksh93", 1023) = 20
| 25782: resolvepath("/usr/lib/amd64/ld.so.1", "/lib/amd64/ld.so.1", 1023) = 18
| 25782: stat("/usr/bin/amd64/ksh93", 0xFFFF80FFBFFFFA50) = 0
| 25782: open("/var/ld/64/ld.config", O_RDONLY) = 3
| 25782: fstat(3, 0xFFFF80FFBFFFF7E0) = 0
| 25782: mmap(0x00000000, 156, PROT_READ, MAP_SHARED, 3, 0) = 0xFFFF80FFBF750000
b) MPM worker:
| 25747/27: stat("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit", 0x7FF86F7DC980) = 0
| 25747/27: open("/dev/null", O_WRONLY|O_CLOEXEC) = 24
| 25747/27: fcntl(24, F_DUPFD, 0x00000000) = 25
| 25747/27: fcntl(25, F_GETFD, 0x7FF87D69E144) = 0
| 25747/27: fcntl(25, F_SETFD, 0x00000000) = 0
| 25747/27: pipe() = 26 [27]
| 25747/27: fcntl(26, F_GETFD, 0xB075736D0) = 0
| 25747/27: fcntl(26, F_SETFD, 0x00000001) = 0
| 25747/27: lwp_suspend(28) = 0
| 25747/27: lwp_suspend(1) = 0
| 25747/27: lwp_suspend(3) = 0
| 25747/27: lwp_suspend(4) = 0
| 25747/27: lwp_suspend(5) = 0
| 25747/27: lwp_suspend(6) = 0
| 25747/27: lwp_suspend(7) = 0
| 25747/27: lwp_suspend(8) = 0
| 25747/27: lwp_suspend(9) = 0
| 25747/27: lwp_suspend(10) = 0
| 25747/27: lwp_suspend(11) = 0
| 25747/27: lwp_suspend(12) = 0
| 25747/27: lwp_suspend(13) = 0
| 25747/27: lwp_suspend(14) = 0
| 25747/27: lwp_suspend(15) = 0
| 25747/27: lwp_suspend(16) = 0
| 25747/27: lwp_suspend(17) = 0
| 25747/27: lwp_suspend(18) = 0
| 25747/27: lwp_suspend(19) = 0
| 25747/27: lwp_suspend(20) = 0
| 25747/27: lwp_suspend(21) = 0
| 25747/27: lwp_suspend(22) = 0
| 25747/27: lwp_suspend(23) = 0
| 25747/27: lwp_suspend(24) = 0
| 25747/27: lwp_suspend(25) = 0
| 25747/27: lwp_suspend(26) = 0
| 25747/27: forkx(0) = 25753
| 25747/27: lwp_continue(28) = 0
| 25747/27: lwp_continue(1) = 0
| 25747/27: lwp_continue(3) = 0
| 25753: forkx() (returning as child ...) = 25747
| 25747/27: lwp_continue(4) = 0
| 25747/27: lwp_continue(5) = 0
| 25747/27: lwp_continue(6) = 0
| 25747/27: lwp_continue(7) = 0
| 25753: getpid() = 25753 [25747]
| 25747/27: lwp_continue(8) = 0
| 25747/27: lwp_continue(9) = 0
| 25747/27: lwp_continue(10) = 0
| 25747/27: lwp_continue(11) = 0
| 25747/27: lwp_continue(12) = 0
| 25753: schedctl() = 0x7FF87C885000
| 25747/27: lwp_continue(13) = 0
| 25747/27: lwp_continue(14) = 0
| 25753: munmap(0x7FF87C0D0000, 262144) = 0
| 25747/27: lwp_continue(15) = 0
| 25747/27: lwp_continue(16) = 0
| 25747/27: lwp_continue(17) = 0
| 25747/27: lwp_continue(18) = 0
| 25753: lwp_self() = 27
| 25747/27: lwp_continue(19) = 0
| 25747/27: lwp_continue(20) = 0
| 25747/27: lwp_continue(21) = 0
| 25747/27: lwp_continue(22) = 0
| 25747/27: lwp_continue(23) = 0
| 25747/27: lwp_continue(24) = 0
| 25747/27: lwp_continue(25) = 0
| 25747/27: lwp_continue(26) = 0
| 25747/27: lwp_sigmask(SIG_SETMASK, 0xFFBE6007, 0xFFFFFFF7, 0x000000FF, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25753: munmap(0x7FF87B800000, 8245248) = 0
| 25753: munmap(0x7FF86E800000, 8245248) = 0
| 25753: munmap(0x7FF87B000000, 8245248) = 0
| 25747/27: close(25) = 0
| 25747/27: close(27) = 0
| 25753: munmap(0x7FF87A800000, 8245248) = 0
| 25753: munmap(0x7FF87A000000, 8245248) = 0
| 25753: munmap(0x7FF879800000, 8245248) = 0
| 25753: munmap(0x7FF879000000, 8245248) = 0
| 25753: munmap(0x7FF878800000, 8245248) = 0
| 25753: munmap(0x7FF878000000, 8245248) = 0
| 25753: munmap(0x7FF877800000, 8245248) = 0
| 25753: munmap(0x7FF877000000, 8245248) = 0
| 25753: munmap(0x7FF876800000, 8245248) = 0
| 25753: munmap(0x7FF876000000, 8245248) = 0
| 25753: munmap(0x7FF875800000, 8245248) = 0
| 25753: munmap(0x7FF875000000, 8245248) = 0
| 25753: munmap(0x7FF874800000, 8245248) = 0
| 25753: munmap(0x7FF874000000, 8245248) = 0
| 25753: munmap(0x7FF873800000, 8245248) = 0
| 25753: munmap(0x7FF873000000, 8245248) = 0
| 25753: munmap(0x7FF872800000, 8245248) = 0
| 25753: munmap(0x7FF872000000, 8245248) = 0
| 25753: munmap(0x7FF871800000, 8245248) = 0
| 25753: munmap(0x7FF871000000, 8245248) = 0
| 25753: munmap(0x7FF870800000, 8245248) = 0
| 25753: munmap(0x7FF870000000, 8245248) = 0
| 25753: munmap(0x7FF86F800000, 8245248) = 0
| 25753: lwp_sigmask(SIG_SETMASK, 0xFFBE6007, 0xFFFFFFF7, 0x000000FF, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25753: close(23) = 0
| 25753: close(22) = 0
| 25753: close(21) = 0
| 25747/28: lwp_mutex_timedlock(0x7FF87C180000, 0x00000000, 0x7FF87C19DA40) (sleeping...)
| 25749/28: port_getn(19, 0xB06513E50, 2, 1, 0x00000000) (sleeping...)
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50, WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25747/27: read(26, 0xB075CB9B8, 16384) (sleeping...)
| 25753: lwp_park(0x00000000, 0) (sleeping...)
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50, WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50, WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50, WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) (sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50, WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
As you can see, in "prefork" mode the hook is actually executed using the syscall "execve()". All is running fine then. In the "worker" mode, the process 25753 is created by the "forkx(0)" call, bit it will go to the state "lwp_park(0x00000000, 0) (sleeping...)" were it never awakes from anymore. I actually have to kill the hanging process usind a SIGKILL. The SVN client which committed to the repository via https will also hang forever. The file(s) I want to commit are actually committed, i.e. the whole problem occurs after commit - which is quite feasible because it is a "post-commit" hook. As the "post-commit" script file is never executed, it doesn't matter what the contents of the "post-commit" script are.
So my questions to you are:
1) Are there any known multi-threading problems with the combination Apache 2.4 MPM Worker, SVN 1.7.x, MOD_DAV_SVN and hooks?
2) Is there any alternative to access an SVN repository via Apache besides using the MOD_DAV_SVN module in Apache? For example, can I possibly use "Fast CGI" to link SVN to Apache? (This is, for example, a solution to run the non-thread-safe PHP in a multi-threaded Worker MPM).
3) What is the recommended way to solve the problem? Do I really have to go without a multi-threaded web server?
Any help or information to solve my problem is highly appreciated! Thank you very much in advance!
Kind regards,
Steffen
--
------------------------------------------------------------------------
Dipl.-Inf. Steffen Moser
School of Advanced Professional Studies Room: O27/317
Ulm University Tel: +49.731.50-24179
Albert-Einstein-Allee 11 Fax: +49.731.50-24182
89081 Ulm, Germany http://saps.uni-ulm.de/
Received on 2016-02-11 07:51:00 CET