Re: Python 3 Bindings Query

Troy Curtis Jr wrote on Tue, Oct 17, 2017 at 03:33:29 +0000:
> On Mon, Oct 16, 2017 at 3:36 PM Daniel Shahaf <d.s_at_daniel.shahaf.name>
> wrote:
> > I take it that of all these, you're interested in the swig-py bindings?
> > Or are the build system and test suite also within your scope?
> >
> >
> Well I certainly don't want to half way do it, so while my initial target
> was the swig bindings, I'll take a look at the full set.

Wonderful.

> > You said the patch was going to be a largeish one. How large/invasive
> > are we talking about? (This affects how easy it'd be for us to
> > review/apply it)
> >
>
> Yes, I suspect the effort will touch a fair number of code, but much of it
> will be direct substitution, and thus should be reasonable to review.
> Plus, it'll certainly be broken up into smaller, more manageable commits.
> I'll also be planning to simultaneously build python3 and python2 bindings,
> since I'll be eyeing future Fedora (and presumably other distro) packaging,
> which will touch a few front end build pieces.
>
> In fact, to get the conversation going I've attached a patch which gives a
> sense of the road ahead. This is where I got to yesterday before deciding

Thanks for posting that. Is that patch ready for commit, or is it
intended just as a proof of concept / basis for discussions?

> that I should probably start talking to the dev team about desires and
> direction. I believe that it should consist mostly of replacing various
> functions deprecated/removed in Py3 with wrappers to consolidate all the
> conditional compiling into a common location. Then substituting the use of
> those functions.
>

Makes sense.

> My initial assessment is there there are really only a handful of the

[snip long & detailed analysis that was helpfully summarised to:]

> The questions to decide on are:
> 1. Are you generally comfortable with the build changes necessary to
> (optionally) build Py2 and Py3 bindings at the same time?

Do you mean that a single 'make install' would install two versions of
libsvn_swig_py.so, one compiled against CPython 2 and one compiled
against CPython 3?

Or do you mean, having a (say) configure-time knob that controls whether
bindings would be built against py2 or against py3?

I think the latter would be uncontroversial. As to the former, it would
involve more moving pieces, so I would be concerned about maintenance
overhead of the new build system code. (We have few active developers
nowadays.)

> 2. Do you want to pick up 'py3c' as a new dep, or implement the handful of
> necessary wrappers?

I don't know the swig-py bindings well enough to answer that. I guess
it depends on how much a "handful" would be.

Datapoints: We use 73 distinct CPython API symbols (see attached grep
results). Of these, only three(?) entry points (all of them PyInt_*)
appear in py3c/compat.h.

> 3. Is the assumption of utf8 encoding sufficiently reasonable?

In Subversion, we assume that argv and stdout are encoded in APR's
"system encoding", but basically everything else — including nearly
every single string parameter to every public API — is required to be in
UTF-8. svn_cmdline.h contains some of the exceptions.

I'm not sure whether that information answers your question, though. In
what concrete cases do the bindings have to convert between str and
bytes? What are the compatibility considerations for user code
(consumers of the swig-py2 bindings that want to upgrade to py3)?

> My general plan of attack will be:
> 1. Replace deprecated Py2 swig functions and syntax with Py2/Py3
> cross-compatible versions or wrappers.
> 2. Ensure existing full Py2 support works correctly.
> 3. Update build to build both, Py2 and Py3, and get Py3 working.
> 4. Look at the remainder of the python usage to ensure Py3 compliance.

At some point we'll want to switch one of the buildbot builders to
use python3. (See http://subversion.apache.org/buildbot/all)

Could you explain how you — and we — test patches written in step #1,
given that "update the build to support py3" only comes later, in #3?
Is it currently possible to build the bindings against py3 (well, to
_run_ the build against py3 and get compiler errors about missing
symbols)?

Thanks for the very detailed email!

Daniel

P.S. Feel free to join #svn-dev on freenode.