[Sidebar] - Interesting discussion on impact of "simple" HTTP requests

Some interesting discussion going on this week at Eclipse.org (none of
this is specifically related to SVN):

http://eclipsewebmaster.blogspot.ca/2012/06/problems-of-scale.html

Which spawned:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=381598
https://bugs.eclipse.org/bugs/show_bug.cgi?id=381620

Among others. There could be some Serf relevance here, although to be
honest I think the way Neon works with HTTPv1 in many ways may be even
more relevant.

The net effect is just the impact that a seemingly small HTTP request
can have on a highly trafficked site where those requests start to add
up. The main problem here is that the Eclipse update process looks
for a small index file that is currently always a HTTP 404. This
results in 1.5GB of network packets per day which basically amounts to
152 minutes per day of their available bandwidth. The 404 itself is
not the problem, the bandwidth usage would be worse if the file
existed. The issue is more the design of the protocol and the
discovery it does each time it runs. This is probably similar to all
of the HEAD and PROPFIND requests we do in our protocol. They are
seeing similar bandwidth issues resulting from the usage of HEAD in
their protocol. It looks like the webmaster would like them to
explore doing more with a single request. Someone that understands
how our REPORT requests work might want to throw some commentary into:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=381620

To explain how a single HTTP request could stream back multiple files
to the client. It looks like this would mean adopting WebDAV on their
servers and clients though.

I am not sure if there is anything for us to learn, but I suspect
there is. If nothing else, it is interesting to see the impact that a
seemingly small and simple request can have when it is multiplied by
millions of users. If there are ways we can continue the work we
started with HTTPv2 and eliminate more HTTP requests there would be
benefits. For example, we still issue a lot of HEAD and PROPFIND
requests during update and checkout. Maybe we could include more
information in the REPORT response to eliminate the need for those
additional requests.

-- 
Thanks
Mark Phippard
http://markphip.blogspot.com/