Yasuhito FUTATSUKI wrote on Wed, Jan 08, 2020 at 00:26:39 +0900:
> On 2020/01/07 9:41, Yasuhito FUTATSUKI wrote:
> > On 2020/01/07 6:52, Yasuhito FUTATSUKI wrote:
> >> By the way, it seems another issue about truncate_subject that current
> >> implementation of truncate_subject may break utf-8 multi-bytes character
> >> sequence, but I didn't reproduce it(because I always use ascii
> >> characters only for file names...).
>
> I could reproduce this problem.
⋮
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3-4: invalid continuation byte
>
> > Probably it needs something like this (but it doesn't support conbining
> > characters, and I didn't any test...):
> > [[[
> > Index: tools/hook-scripts/mailer/mailer.py
> > ===================================================================
> > --- tools/hook-scripts/mailer/mailer.py (revision 1872398)
> > +++ tools/hook-scripts/mailer/mailer.py (working copy)
> > @@ -159,7 +159,13 @@
> > truncate_subject = 0
> >
> > if truncate_subject and len(subject) > truncate_subject:
> > - subject = subject[:(truncate_subject - 3)] + "..."
> > + # To avoid breaking utf-8 multi-bytes character sequence, we should
> > + # search the top of the sequence if the byte of the truncate point is
> > + # secound or later part of multi-bytes character sequence.
> > + idx = truncate_subject - 3
> > + while 0x80 <= ord(subject[idx]) <= 0xbf:
> > + idx -= 1
> > + subject = subject[:idx] + "..."
> > return subject
> >
> > def start(self, group, params):
> > ]]]
>
> After this patch applied, the script above runs without error.
>
> However, this produces Subject line below.
>
> [[[
> Subject: r1 - =?utf-8?b?44CH44CH44CH5LiA?= =?utf-8?b?44CH44CH44CH5LiJ?= =?utf-8?b?44CH44CH44CH5LqM?= =?utf-8?b?44CH44CH44CH5LqU?= =?utf-8?b?44CH44CH44CH5YWt?= =?utf-8?b?44CHLi4u?=^M
> ]]]
>
> and decoded Results is
>
> "Subject: r1 - 〇〇〇一〇〇〇三〇〇〇二〇〇〇五〇〇〇六〇..."
>
> because white space(s) between encoded words are ignored.
> I think this is not what we want.
We shouldn't be handling a UTF-8 string bytewise. If it needs truncating, then
we should truncate it characterwise or wordwise, not bytewise.
We shouldn't be doing the MIME-encoding ourselves. We should just provide the
subject line to the 'email' module and let it worry about MIME encoding, line
folding, and everything else. (This is a preëxisting problem.)
Makes sense?
Cheers,
Daniel
Received on 2020-01-07 18:19:11 CET