Re: Merging parallel-put to /trunk

Greg Stein <gstein_at_gmail.com> writes:

> Oh, give me a break. "not officially supported" ... This is a public list.
> I'll be nice.
>
> Both of those components have H/2 support now.

I might have misinterpreted the warning banner in httpd documentation [1]
saying that mod_http2 is experimental (or maybe this statement is obsolete):

  [Warning]

  This module is experimental. Its behaviors, directives, and defaults are
  subject to more change from release to release relative to other standard
  modules.

>> (2) Is making parallel PUTs the proper way to speed up commits?
>
> Absolutely. Typically, the PUT is a delta against a prior revision, it may
> be compressed, and it may be encrypted. ALL of the work to handle those
> aspects can be handled by a different core in the server.
>
> I seem to recall an inspection of our server time, that showed MD5 handling
> was a significant component. By separating PUTs, we can get multiple server
> cores working for us.

I think that local commits are usually fast enough. But committing over
a high-latency network, e.g., with a transatlantic RTT of 150ms, can be
painfully slow — see below.

>> As far as I know, squashing everything into a single POST would make
>> the commit up to 10-20 times faster, depending on the amount of changes.
>
> Pfft.

I attached a dirty patch that does that. The aim is to measure what we
can expect from this approach. Please note that the patch is just a proof
of concept, and has many various issues — the wire format, only supporting
short-circuit authz, etc.

Results without and with the patch are:

  Importing 10000 files, 150 ms RTT 1887.582 s → 46.363 s (40.7x faster)
  Importing 10000 files, 50 ms RTT 660.897 s → 35.267 s (18.7x faster)
  Committing 25 files, 150 ms RTT 9.198 s → 1.739 s (5.3x faster)
  Committing 25 files, 50 ms RTT 3.357 s → 0.714 s (4.7x faster)
  Committing 2 files, 150 ms RTT 1.102 s → 0.787 s (1.4x faster)
  Committing 2 files, 50 ms RTT 0.589 s → 0.406 s (1.4x faster)

Sending everything in a single request is probably less appealing than having
parallel PUT requests. But in the meanwhile, I don't think that parallel
PUTs can provide us with the same performance improvement — at least,
for the high-latency networks.

[1] https://httpd.apache.org/docs/2.4/mod/mod_http2.html

Regards,
Evgeny Kotkov