Re: Fix make -j for SWIG bindings (was: swig: srcdir vs. blddir)

On 12/17/05, Max Bowsher <maxb1@ukf.net> wrote:
> David James wrote:
> > On 12/15/05, Philip Martin <philip@codematters.co.uk> wrote:
> >
> >>A more annoying problem is that the 'make -j4 swig-py' parallel build
> >>is broken, it sometimes fails with
> >>
> >>/usr/local/swig-1.3.24/bin/swig -I/home/pm/sw/subversion/obj/../svn/subversion/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/include -I/home/pm/sw/subversion/obj/../svn/subversion/bindings/swig/proxy -I/home/pm/sw/subversion/obj/subversion/bindings/swig/proxy -I/usr/local/apache2/include -I/usr/local/apache2/include -python -classic -w451 -w305 -o ../svn/subversion/bindings/swig/python/svn_client.c ../svn/subversion/bindings/swig/svn_client.i
> >>Traceback (most recent call last):
> >> File "build/generator/swig/external_runtime.py", line 107, in ?
> >> gen.write()
> >> File "build/generator/swig/external_runtime.py", line 99, in write
> >> self.write_long_long_fix()
> >> File "build/generator/swig/external_runtime.py", line 82, in write_long_long_fix
> >> self.checkout("python","python.swg")
> >> File "/home/pm/sw/subversion/svn/build/generator/swig/__init__.py", line 56, in checkout
> >> shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
> >> File "/usr/lib/python2.3/shutil.py", line 73, in copy
> >> copymode(src, dst)
> >> File "/usr/lib/python2.3/shutil.py", line 52, in copymode
> >> os.chmod(dst, mode)
> >>OSError: [Errno 2] No such file or directory: 'subversion/bindings/swig/proxy/python.swg'
> >
> >
> > Philip, does the attached patch solve the issue with make -j?
> >
> > [[[
> >
> > * build/generator/gen_make.py
> > Ensure that external_runtime.py is only called once. This fixes
> > "make -j", which crashes when multiple instances of
> > external_runtime.py try to create the same files.
> >
> > Found by: philip
> >
> > ]]]
> >
> > Index: build/generator/gen_make.py
> > ===================================================================
> > --- build/generator/gen_make.py (revision 17814)
> > +++ build/generator/gen_make.py (working copy)
> > @@ -148,10 +148,16 @@
> > fname = '%s/swig_%s_external_runtime.swg' % (self.swig.proxy_dir, lang)
> > swig_runtime_fnames.append(fname)
> > if not self.release_mode:
> > + # external_runtime.py generates the SWIG runtimes for all languages
> > + # the first time it is called
> > + first_runtime = swig_runtime_fnames[0]
> > + other_runtimes = " ".join(swig_runtime_fnames[1:])
> > self.ofile.write(
> > - '%s:\n' % " ".join(swig_runtime_fnames) +
> > - '\tcd $(top_srcdir) && $(PYTHON)' +
> > + '%s: %s\n'
> > + '%s:\n'
> > + '\tcd $(top_srcdir) && $(PYTHON)'
> > ' build/generator/swig/external_runtime.py build.conf "$(SWIG)"\n\n'
> > + % (other_runtimes, first_runtime, first_runtime)
> > )
> >
> > self.ofile.write(
>
>
> Ouch! That's kind of inelegant. -0.9
>
> Exactly what is going on here?
>
> Surely it is a bug in 'make -j' if it executes the same make stages in
> multiple threads of control?

Here's a test case Makefile: {{{
all: a b c
a b c:
  sleep 5 && touch a b c
}}}

If you run this Makefile without "-j", it executes the command once.
If you run it with "-j 4", it executes the touch command three times
in parallel. If you run multiple instances of our external_runtime.py
script in parallel, the three scripts will step on each other's toes,
trying to create and modify the same files at the same time.

Here's a new Makefile which avoids the problem: {{{
all: a b c
b c: a
a:
  sleep 5 && touch a b c
}}}

My patch uses the same trick.

Cheers,

David

--
David James -- http://www.cs.toronto.edu/~james