SvnAnt/Subclipse command-line ability to handle large quantities of data

On Sun, 2005-07-03 at 09:07 -0400, Mark Phippard wrote:
> You can go ahead and move this to the cc: list.

Thanks. Here's a link to the issue for everyone's reference:

http://subclipse.tigris.org/issues/show_bug.cgi?id=321

Committers, please read the issue and comment on this scalability issue.

> I am inclined to just go with your idea, but the one thing I was wondering
> is why the function doesn't just return the InputStream it is already
> receiving back? I would assume there was a problem, perhaps a deadlock,
> and that is why it was reading it into a ByteArray to begin with.

Yes, that was my concern as well, and why I haven't already changed the
code. I ran both 'svn log' and 'svn blame' last week to attempt to
locate when that code was introduced, and _why_. I found no trace of it
in the log, and blame shows it changed during an (apparently accidental)
line ending conversion. :-(

Is anyone familiar with this code, which uses a java.lang.Process object
to wrap the svn command-line client, ands read its output byte-by-byte
into a Vector in CmdLineClientAdapter.streamToByteArray(), then converts
the data read into that Vector into a ByteArrayInputStream?

> In general I would agree that it is a reasonable assumption to make that
> the InputStream will be read in a timely manner by anyone using the API.

A hybrid approach might also be reasonable, where a BufferedInputStream
extension class is used which attempts to stream the data to disk when
the buffer is filled and more data is available to be read (e.g.
threshold is exceeded), but can gracefully revert to normal
BufferedInputStream behavior, blocking if the disk is full or not
writable.

This approach is closer to preserving the previous semantics while still
scaling, but is not as performant as is my preference.

- Dan
Received on Wed Jul 6 04:12:24 2005