On Tue, May 22, 2018 at 8:30 PM, Davor Josipovic <davorj_at_live.com> wrote:
>> Now that is interesting. 40k doesn't seem to be such a large amount of
>> data for modern computers. Very slow and fragmented hard drive? Or perhaps
>> there's something else going on that is manifesting this way?
>
> The HDD is indeed on the slowside, and together with low memory...
>
> But I think this also show how I/O intensive SVN is. On the client side, for
> each committed file, one copy is placed in .svn folder, and an other copy in
> a temporary folder (which is deleted after file transfer in v1.9). So for
> each file committed, a double copy is made client-side. This temporary copy
> is really necessary?
>
> Server-side, I see similar disk bashing. For each committed file, max 2 (?)
> copies are made in the transaction directory.
>
> So any way to reduce the I/O?
A couple of improvements have been made in 1.10. It would be
interesting to test this same scenario with a 1.10 client (to verify
the client-side improvements) and if possible also with a 1.10 server,
with a dumped+loaded FSFS back-end (to get an idea of any server-side
improvements).
Client-side there were some changes to make the (http) client stream
svndiff deltas without creating temporary files (during 'commit' and
'import'):
https://svn.apache.org/r1803143
https://svn.apache.org/r1803144
https://svn.apache.org/r1804819
Server-side, LZ4 compression was added as the default compression
algorithm in the FSFS back-end, and on the wire (not sure if that
helps in your case ... it won't impact the number of files that are
created temporarily I suppose):
https://subversion.apache.org/docs/release-notes/1.10.html#lz4
Also, maybe a commit such as this one helps too (though it seems to
only talk about reads):
https://svn.apache.org/r1759399
As always, probably further improvements can be made I guess, but
someone will need to dig into it.
If you'd be able to test with 1.10 both client and server-side, that
would give some more data points about where we are with the latest
state of things ...
Finally, one last tip: this (rather old) performance tuning page
suggests putting the "transactions" folder on a separate (faster)
filesystem to speed up commits:
https://www.orcaware.com/svn/wiki/Server_performance_tuning_for_Linux_and_Unix#Reducing_the_number_of_writes
I don't know how realistic that is in your setup (plus you need to
make sure that transactions filesystem is large enough for your
commits).
--
Johan
Received on 2018-05-25 16:29:21 CEST