On Tue, Jun 14, 2011 at 6:47 PM, Paul Burba <ptburba_at_gmail.com> wrote:
> On Tue, Jun 14, 2011 at 11:45 AM, Paul Burba <ptburba_at_gmail.com> wrote:
>> On Tue, Jun 14, 2011 at 3:41 AM, Philip Martin
>> <philip.martin_at_wandisco.com> wrote:
>>> Paul Burba <ptburba_at_gmail.com> writes:
>>>
>>>> No surprise that your checkouts are faster than mine given you are
>>>> using a local mirror. What's puzzling me is why my upgrades are so
>>>> much slower than yours.
>>>>
>>>> Running an upgrade of a trunk WC on my machine under xperf takes
>>>> 00:03:46.351 elapsed and 11.44s CPU time using my primary drive (320
>>>> GB, SATA-II, 7200 rpm, 16 MB Cache, NTFS). Subversion spends 50s
>>>> total disk service time (46.8s of that is read service time).
>>>
>>> Does "total disk service time" represent all the time waiting for disk
>>> IO? 11s CPU plus 50s disk still leaves 165s unaccounted.
>>
>> I've only just started using xperf (it's part of the Windows
>> Performance Toolkit) and can't find an exact definition of their
>> terminology. I can say that xperf adds lot of overhead since upgrades
>> run under it reliably take about twice as long.
>>
>> The 165s of unaccounted time is largely made up of 143s of disk
>> service time by the system process...but now that I dig a little
>> further I find that the system process' disk utilization is touching
>> every directory in the working copy as well as writing the new
>> .svn\pristine\ stuffs. Previously I was only looking at the disk
>> utilization by the svn.exe process itself.
>>
>> ~~~~~
>>
>> Not sure what to make of this: I tried an upgrade of a trunk WC using
>> trunk_at_1135642 on my HDD and then my SSD (on the same box)
>>
>> Elapsed time for the HDD: 00:01:12.450
>> For the SSD: 00:00:09.267
>>
>> Not sure if this points to anything other than SSDs are faster (duh!),
>> but the degree of improvement was a unexpected.
>>
>>> Upgrade
>>> prints a notification line for each directory, is there a significant
>>> delay before the first line, or after the last line, or is the delay
>>> spread among the lines?
>>
>> In all my testing I used the --quiet option. I just now tried it
>> without and the delay is definitely spread among the output.
>>
>>> If it is a Subversion problem rather than your machine there are two
>>> areas that may be worth investigating (but it's hard to say when most of
>>> the time is unaccounted).
>>>
>>> The property migration currently invokes multiple transactions
>>> per-directory. Moving to a single transaction per-directory will
>>> probably help.
>>>
>>> Upgrade currently copies all the text-bases, I did experiment with
>>> creating workqueue items to move them instead but it wasn't any faster
>>> on my Linux box. However it may help on Windows.
>>
>> A bit of cut-and-paste from IRC this morning for the benefit of others
>> reading this thread:
>>
>> <philipm> Bert, pburba: I suppose we could do the whole upgrade as a
>> single transaction.
>> <philipm> The database will never be accessed by more than one
>> thread/process during upgrade.
>> <philipm> And "partial" upgrades are already thrown away.
>>
>> <Bert> philipm: Agree
>>
>> <hwright> philipm: we have nested txns in sqlite, so you should be
>> able to just wrap the entire upgrade process in a txn
>>
>> <Bert> philipm: Last week I tried to reduce the number of stats in the
>> property load code by just fetching all the entries in the property
>> dirs, but the perf difference was not measurable. I think making it a
>> single transaction would have the most impact.
>>
>> <pburba> philipm, Bert, hwright : Have we done anything similar to
>> that already (wrapping the whole upgrade in a single transaction)? I'd
>> like to tackle it, but looking at similar work to provide some
>> understanding/traction would be very helpful
>>
>> <hwright> pburba: we don't expose txns outside of wc_db
>>
>> <philipm> pburba: What you want is upgrade_working_copy as a callback
>> from svn_wc__db_with_txn
>>
>> <hwright> I'm not sure how much of the work is done in wc_db (and how
>> much without), but that's the first thing to consider
>>
>> <philipm> We would have to move lots of code, or expose the txn to
>> upgrade.c in some form.
>>
>> <hwright> upgrade is such a special case, it makes sense to
>> potentially expose it in a limited fashion
>>
>> <Bert> svn_sqlite__with_lock()
>>
>> <philipm> upgrade.c already access the sqlite db directly
>>
>> <hwright> philipm: in that case, just do the txn in upgrade.c, and
>> wc_db.c should be fine due to txn nesting, yes?
>>
>> <philipm> Yes. Putting svn_sqlite__with_lock around
>> upgrade_working_copy in upgrade.c will probably be OK
>>
>> <hwright> (if it isn't, you'll know pretty quick. :) )
>>
>> <philipm> Upgrade uses a txn to write entries, per-dir.
>> <philipm> But if the whole upgrade is a txn that could be removed,
>> it's not used for anything other than upgrade.
>> <philipm> With nested txns that will not be necessary, but avoiding
>> nested txns may have a performance benefit?
>> <philipm> I don't know. For a first attempt just put
>> upgrade_working_copy in svn_sqlite__with_lock.
>>
>> <pburba> philipm: Ok, I'll give that a try then.
>
> Here's a stab at it. In my ad-hoc testing, the attached patch
> improved the elapsed upgrade time of a ^/subversion/trunk WC from
> 00:02:00 to 00:01:34 (3 runs). It improved an upgrade of a
> ^/subversion/tags/ebcdic WC from 00:11:44 to 00:06:52 (single run).
Committed r1136525
> [[[
> Improve 'svn upgrade' performance by moving more of the upgrade process into
> a single sqlite transaction.
>
> * subversion/libsvn_wc/entries.c
>
> (entries_write_baton,
> entries_write_new_cb): Remove.
>
> (svn_wc__write_upgraded_entries): Don't run this part of the upgrade
> operation in a dedicated sqlite transaction, but rather embed it in a
> larger transaction. Also use the existing iterpool when creating
> tmp_entry_abspath argument to write_entry.
>
> * subversion/libsvn_wc/upgrade.c
>
> (upgrade_to_wcng): Don't create the new DB here, move that to svn_wc_upgrade.
>
> (upgrade_working_copy_baton_t,
> upgrade_working_copy_txn): New baton and transaction wrapper for
> upgrade_working_copy.
>
> (svn_wc_upgrade): Create the new wcng DB here and wrap the call to
> upgrade_working_copy() in a sqlite transaction.
> ]]]
>
> Paul
>
Received on 2011-06-16 18:20:42 CEST