[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

[PATCH]: Speed up deletion of multiple files

From: Paul Burba <ptburba_at_gmail.com>
Date: Wed, 23 May 2012 18:42:43 -0400

On one CollabNet's forums a user reported that a single delete command
of 2000+ WC files took well over three hours to complete with 1.7.5
(see http://subversion.open.collab.net/ds/viewMessage.do?dsForumId=4&dsMessageId=456214).

I'm able to replicate similar behavior with a partial checkout of
^/subversion/tags

My test WC:

  WC Size: 437 MB 21,012 Files, 2,717 Folders
  wc.db Size: 18,108,000 bytes
  Deletion Targets: 2642 files

Using 1.7.5 this takes almost 27 minutes on my machine:

  C:\SVN\sandbox\subversion-tags>timethis svn delete -q --targets
del-target.2463.txt

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Fri May 18 11:27:04 2012

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Fri May 18 11:27:04 2012
  TimeThis : End Time : Fri May 18 11:53:58 2012
  TimeThis : Elapsed Time : 00:26:53.758

trunk_at_1341851 is significantly faster, taking only 16 minutes:

  C:\SVN\sandbox\subversion-tags>timethis svn delete -q --targets
del-target.2463.txt

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Wed May 23 10:56:41 2012

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Wed May 23 10:56:41 2012
  TimeThis : End Time : Wed May 23 11:13:17 2012
  TimeThis : Elapsed Time : 00:16:35.873

Using optimizations similar to what Bert used in r1341848 and creating
a new , the attach patch cuts this down to just over 4 seconds:

  C:\SVN\sandbox\subversion-tags>timethis svn delete -q --targets
del-target.2463.txt

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Wed May 23 17:41:12 2012

  TimeThis : Command Line : svn delete -q --targets del-target.2463.txt
  TimeThis : Start Time : Wed May 23 17:41:12 2012
  TimeThis : End Time : Wed May 23 17:41:16 2012
  TimeThis : Elapsed Time : 00:00:04.116

...So, WCNG gurus, does this look ok?

[[[
Speed up WC deletions.

* subversion/libsvn_wc/wc-queries.sql
  (STMT_DELETE_NODES_ABOVE_DEPTH_RECURSIVE,
   STMT_INSERT_DELETE_FROM_NODE_RECURSIVE): Make the OR operation the
   outer operation by duplicating some cheap tests.

  (STMT_INSERT_DELETE_LIST_RECURSIVE,
   STMT_INSERT_DELETE_LIST): Split old STMT_INSERT_DELETE_LIST into two
   new versions, one recursive, and one not.

* subversion/libsvn_wc/wc_db.c
  (delete_node): Use the faster non-recursive query when operating on a file.
]]]

-- 
Paul T. Burba
CollabNet, Inc. -- www.collab.net -- Enterprise Cloud Development
Skype: ptburba

Received on 2012-05-24 00:43:15 CEST

This is an archived mail posted to the Subversion Dev mailing list.