On Sat, Aug 29, 2015 at 3:57 PM, Evgeny Kotkov <evgeny.kotkov_at_visualsvn.com>
wrote:
> Hi Stefan,
>
> Stefan Fuhrmann <stefan2_at_apache.org> writes:
>
> > Introduce a stream wrapper object that adds mark/seek support to any
> > readable stream. Use it on the stdin streams in our CL tools.
>
> [...]
>
> What happens if someone calls svn_stream_mark(stream, &mark, pool) on the
> stream and then — perhaps, much later — reads a huge amount of data from
> the
> stream? Say, the total amount of data to read is 8 GiB, and it's
> processed in
> small chunks. Is it true that in this case all 8 GiB of data are going to
> be
> buffered by the stream and kept in memory?
>
Yes. This is exactly why we can only use it when we have
reasonable control over the stream's usage, i.e. we can
use it in our CL tools because all the code that will be run
is under our control. But we cannot make e.g.
svn_stream_for_stdin() use it by default.
> Here is an quick example:
>
> SVN_ERR(svn_stream_mark(buffered_stream, &mark, pool));
> ...
>
> while (!eof)
> {
> svn_stringbuf_t *str;
>
> svn_pool_clear(iterpool);
>
> SVN_ERR(svn_stream_readline(buffered_stream, &str, "\n", &eof,
> iterpool));
> ...
> }
>
And this example is even nicer than say our dump stream
parser code where some pools are relatively long-lived.
Not a problem for revprop-only dumps (one cleanup per
revision is enough there) but for generic dumps.
The best solution seems to be to allow for explicit resource
management as we do with other potentially "expensive"
objects. r1700305 implements that.
-- Stefan^2.
Received on 2015-08-31 21:34:15 CEST