SVN 1.1.3 Massive CPU load at updates using mod_dav_svn

Dear Subversion developers,

I've been running load testing on Svn updates through httpd (apache /
mod_dav_svn).

I am updating 100 clients similtanuously who are one revision a way from
the svn head; I am using fsfs for svn back end
to achieve pure-readonly updates; and to make it faster for svn to
deliver the changeset (rev) just by handing the related sections in the
db/revs/rev_num file. (where rev_num stands for the latest revision number)

I checked the content of the xdelta tag in the xml response and found it
to identically match sections of the db/revs/rev_num file.

My assumption was that the mod_dav_svn(along with the backend svn
libraries), once receiving a one-version difference update request would
only care to grab that data and pass it back to the clients; apparently
this is not the case; apache ends up consuming massive cpu load
(reaching load average to 40 sometimes).

Whats even more confusing was when trying to do a fresh checkout or
direct grab of files over svn/http; cpu usage was almost negligible.
This is confusing because, theoretically, this scenario should be the
most cpu intensive because mod_dav_svn needs to reconstruct the latest
files over the different previous revisions.

After facing this un-justifiable cpu-load and consulting on #svn irc, I
ended up writing to you the detailed secanrio here below.

Note that I also ran this test against a quad-proc xeon debian server
and was able to reproduce that issue as well.

Thank you in advance,

I must say that I feel truly grateful to the people who made this
outstanding software possible.

Let me know if you need any further details. Kindly respond to my email
(kefah@freesoft.jo) as I am not subscribed to your mailing list.

Best regards,
Kefah T. Issa

P.S. You can find attached the helper bash scripts I used to conduct my
tests.

# Hardware : 550 MHz PIII box
# Software: Fedora Core 2 (updated using Atrpm repository with
subversion 1.1.3)
# ( RPMS : subversion-1.1.3-0_14.rhfc2.at,
mod_dav_svn-1.1.3-0_14.rhfc2.at)
# Svn using mod_dav_svn / Apache 2.0.51

# Svn server lives in separete box than the svn clients running the load.

# On svn server box; prepare ...
# login as root, we are also using a predefined user named esam.
ssh root@mytestbox

# Create Http configuration file for our test svn repo.
cat > /etc/httpd/conf.d/svn_loadtest.conf << EOF

<Location /svnlt>
  DAV "svn"
  SVNPath "/home/esam/www/svnlt"
  AllowOverride None
</Location>

EOF

# Create the needed svn repository
su - esam
mkdir www
cd www
svnadmin create --fs-type fsfs svnlt

# exit user esam and return to root
chown -R apache.apache /home/esam/www/svnlt
chmod o+rx /home/esam/www
chmod o+rx /home/esam

# On the other box;from we run svn clients try checkouts and updates.

# Run 100 concurrent checkouts for this 0-rev repo on another box using
the attached create_working_copies.sh shell script.
# Note that config.sh should be properly configured; as its using by the
other shell scripts.
# server loadaverage before : 0,0,0
./create_working_copies.sh
# server load average after : 0.01,0,0

# On another working copy, Create and add a 100 MB bin (empty / zero)
file :
cd myotherworkingcopy/svnlt
dd if=/dev/zero of=onemb.bin count=1 bs=1MB
svn add onemb.bin
svn commit -m """ onemb.bin

# Conclusion : Checking out empty (rev 0) repository consume negligible cpu.

# Remove all checked out folder
rm -fR wc_*

# Run the create_working_copies
./create_working_copies.sh
# load average on the server after finishing : 0.02, 0.01, 0.00
# Conclusion : Checking out 1MB repository consumes negligible cpu.

# From another working copy, add another 1 MB content to the original
file and commt.
cd myotherworkingcopy/svnlt
dd if=/dev/zero of=onemb.bin count=1 bs=1MB seek=1
svn commit -m """ onemb.bin

# Update from rev1 to rev2
./update_working_copies.sh

# Using top, monitor load average when that is taking place
# serverside load average : 27,14,5
# Conclusion : Updating a 1MB file diff consumes massive cpu resources.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org