Yasuhito FUTATSUKI wrote on Tue, Dec 11, 2018 at 01:18:19 +0900:
> Hi,
> I hear that property values may not be vaild UTF-8 string if they are
> set with skipping validation.
Let's be more precise.
In general, a property's name must satisfy the conditions documented in
svn_prop_name_is_valid()'s docstring and a property's value is an opaque
binary blob, just like file contents (when svn:eol-style and svn:keywords
are unset).
However, some specific properties have additional requirements on their
values. If a property passes svn_prop_is_boolean() then its value MUST
be "*" and if a property is an svn:* property then its value MUST be
UTF-8 with LF line endings — the latter is enforced by svn_repos__validate_prop().
> (https://twitter.com/jun66j5/status/1067295499907084288 (written in
> Japanese language); https://trac.edgewall.org/ticket/4321)
>
> However, current swig-py3 typemaps to return those values use
> PyStr_FromStringAndSize() to convert svn_string_t into Python's str object,
> which raise UnicodeDecodeError for invalid UTF-8 octet sequence, so there
> is no way to get those strict value.
>
> To resolve this issue, there seems to be some some options:
>
> (1). Those API always return str (Unicode) with 'strict' conversion.
> if error occured, abandon to get these values. (current implementation)
> (2). Those API always return str with 'surrogateescape' conversion.
> if applicatin want to detect irregular data, search \u+dc00-\u+dcff
> character in values.
> (3). Those API always return bytes. if applications want to handle as
> str, decode them in application side.
> (4). Those API return str for valid data, and return bytes for invalid data
> to avoid missing way to get data.
> (5). other (I have no idea..., though)
>
> I think (2) or (3) is appropriate, but I don't have confidence.
> Any ideas?
Generic APIs that work on any property should return bytes. More
specific APIs that work on properties that have further restrictions
(svn:needs-lock, svn:date), or even a structured value (svn:externals,
svn:mergeinfo), may use appropriately more specific data types.
In general, if an API has an input of type bytes and needs to return
str, I would have it throw an exception if the conversion fails, so the
caller is forced to deal with the failure mode explicitly. It'd be fine
to use surrogateescape or return bytes *if the caller has explicitly
requested that*, but the default postcondition should be as simple as
possible: "This function either returns str or raises an exception".
Makes sense?
Cheers,
Daniel
Received on 2018-12-11 09:12:43 CET