Re: svn commit: r1358022 - in /subversion/trunk: LICENSE NOTICE subversion/include/svn_utf.h subversion/libsvn_subr/utf_width.c subversion/svn/file-merge.c

On Mon, Jul 09, 2012 at 02:47:25PM +0200, Bert Huijben wrote:
> How do you check if the file you are merging is valid utf-8?

See the merge_chunks() function.

We convert data to UTF-8 from the native (locale) encoding.
This cannot fail (every encoding can be represented in UTF-8)
but the result might look funny in case the file uses some other encoding
than the native one. But that's OK -- this conversion happens only for
display purposes, data in the actual file is never changed, so you can
still edit individual chunks in their original form.

> I assumed that we currently just passed files to the console mostly unmodified to allow the terminal to do the hard work.

That works fine as long as you don't care about the width of the
line you're printing.

For the side-by-side display we make an effort to make it look nice.
If that doesn't work, the side-by-side display might look strange
because lines appear with varying lengths. That is the fallback mode
which assumes width=1 and one-byte-per-character for all characters.
 
> I'm pretty sure that we can assume at least many (if not most) text files stored in Subversion are *not* utf-8 and will fail when tested for utf-8 validness.
>
> How does this library handle non-utf8 strings?

You mean the svn_utf_cstring_utf8_width() function? It will return
an error for invalid UTF-8.

In our usage of this API, the UTF-8 validness check in is performed
on data that the merge tool has converted to UTF-8. The API must fail for
invalid UTF-8 input since it cannot convert such input to UTF-32 in
order to run mk_wcwidth() on it.

Again, this is in-memory data which we're going to display to the user
in a formatted way to so we need to know its width.
None of this has anything to do with any versioned data in files.
Received on 2012-07-09 15:31:14 CEST