On Tue, Jun 14, 2011 at 3:41 AM, Philip Martin
<philip.martin_at_wandisco.com> wrote:
> Paul Burba <ptburba_at_gmail.com> writes:
>
>> No surprise that your checkouts are faster than mine given you are
>> using a local mirror. What's puzzling me is why my upgrades are so
>> much slower than yours.
>>
>> Running an upgrade of a trunk WC on my machine under xperf takes
>> 00:03:46.351 elapsed and 11.44s CPU time using my primary drive (320
>> GB, SATA-II, 7200 rpm, 16 MB Cache, NTFS). Subversion spends 50s
>> total disk service time (46.8s of that is read service time).
>
> Does "total disk service time" represent all the time waiting for disk
> IO? 11s CPU plus 50s disk still leaves 165s unaccounted.
I've only just started using xperf (it's part of the Windows
Performance Toolkit) and can't find an exact definition of their
terminology. I can say that xperf adds lot of overhead since upgrades
run under it reliably take about twice as long.
The 165s of unaccounted time is largely made up of 143s of disk
service time by the system process...but now that I dig a little
further I find that the system process' disk utilization is touching
every directory in the working copy as well as writing the new
.svn\pristine\ stuffs. Previously I was only looking at the disk
utilization by the svn.exe process itself.
~~~~~
Not sure what to make of this: I tried an upgrade of a trunk WC using
trunk_at_1135642 on my HDD and then my SSD (on the same box)
Elapsed time for the HDD: 00:01:12.450
For the SSD: 00:00:09.267
Not sure if this points to anything other than SSDs are faster (duh!),
but the degree of improvement was a unexpected.
> Upgrade
> prints a notification line for each directory, is there a significant
> delay before the first line, or after the last line, or is the delay
> spread among the lines?
In all my testing I used the --quiet option. I just now tried it
without and the delay is definitely spread among the output.
> If it is a Subversion problem rather than your machine there are two
> areas that may be worth investigating (but it's hard to say when most of
> the time is unaccounted).
>
> The property migration currently invokes multiple transactions
> per-directory. Moving to a single transaction per-directory will
> probably help.
>
> Upgrade currently copies all the text-bases, I did experiment with
> creating workqueue items to move them instead but it wasn't any faster
> on my Linux box. However it may help on Windows.
A bit of cut-and-paste from IRC this morning for the benefit of others
reading this thread:
<philipm> Bert, pburba: I suppose we could do the whole upgrade as a
single transaction.
<philipm> The database will never be accessed by more than one
thread/process during upgrade.
<philipm> And "partial" upgrades are already thrown away.
<Bert> philipm: Agree
<hwright> philipm: we have nested txns in sqlite, so you should be
able to just wrap the entire upgrade process in a txn
<Bert> philipm: Last week I tried to reduce the number of stats in the
property load code by just fetching all the entries in the property
dirs, but the perf difference was not measurable. I think making it a
single transaction would have the most impact.
<pburba> philipm, Bert, hwright : Have we done anything similar to
that already (wrapping the whole upgrade in a single transaction)? I'd
like to tackle it, but looking at similar work to provide some
understanding/traction would be very helpful
<hwright> pburba: we don't expose txns outside of wc_db
<philipm> pburba: What you want is upgrade_working_copy as a callback
from svn_wc__db_with_txn
<hwright> I'm not sure how much of the work is done in wc_db (and how
much without), but that's the first thing to consider
<philipm> We would have to move lots of code, or expose the txn to
upgrade.c in some form.
<hwright> upgrade is such a special case, it makes sense to
potentially expose it in a limited fashion
<Bert> svn_sqlite__with_lock()
<philipm> upgrade.c already access the sqlite db directly
<hwright> philipm: in that case, just do the txn in upgrade.c, and
wc_db.c should be fine due to txn nesting, yes?
<philipm> Yes. Putting svn_sqlite__with_lock around
upgrade_working_copy in upgrade.c will probably be OK
<hwright> (if it isn't, you'll know pretty quick. :) )
<philipm> Upgrade uses a txn to write entries, per-dir.
<philipm> But if the whole upgrade is a txn that could be removed,
it's not used for anything other than upgrade.
<philipm> With nested txns that will not be necessary, but avoiding
nested txns may have a performance benefit?
<philipm> I don't know. For a first attempt just put
upgrade_working_copy in svn_sqlite__with_lock.
<pburba> philipm: Ok, I'll give that a try then.
Received on 2011-06-14 17:45:56 CEST