Re: Fix make -j for SWIG bindings

On 12/17/05, Philip Martin <philip@codematters.co.uk> wrote:
> >> Philip, does the attached patch solve the issue with make -j?
>
> I haven't tried the patch.
>
> >> [[[
> >>
> >> * build/generator/gen_make.py
> >> Ensure that external_runtime.py is only called once. This fixes
> >> "make -j", which crashes when multiple instances of
> >> external_runtime.py try to create the same files.
> >>
> >> Found by: philip
> >>
> >> ]]]
> >>
> > Ouch! That's kind of inelegant. -0.9
> >
> > Exactly what is going on here?
>
> external_runtime.py gets invoked three times with exactly the same
> arguments because build-outputs.mk has three targets that invoke it
> (swig_python_external_runtime.swg, swig_perl_external_runtime.swg,
> swig_ruby_external_runtime.swg) and the swig-headers target depends on
> all three.

Aha! This is exactly the problem. Using regular make, the
"external_runtime.py" only gets invoked once because it generates all
three files (plus some other files) the first time it is invoked.
Using parallel make, it's possible that external_runtime.py could be
run three times in parallel.

Here's how I reproduced the bug. First, I edited build-outputs.mk to
sleep for 5 seconds before running external_runtime.py, so that I
could reliably reproduce the bug without worrying about timing issues.
Then I ran the following commands: {{{
  make autogen-swig-pl
  rm -f subversion/bindings/swig/proxy/*runtime.swg
  make -j 3 autogen-swig-pl
}}}

Here's the output: {{{
james@syntax% make -j 3 autogen-swig /nobackup/clgrp/james/trunks
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
sleep 5 && cd . && /h/46/james/bin/python
build/generator/swig/external_runtime.py build.conf
"/h/46/james/bin/swig"
Traceback (most recent call last):
  File "build/generator/swig/external_runtime.py", line 107, in ?
    gen.write()
  File "build/generator/swig/external_runtime.py", line 98, in write
    self.write_external_runtime()
  File "build/generator/swig/external_runtime.py", line 30, in
write_external_runtime
    self.checkout(".","runtime.swg")
  File "/nobackup/clgrp/james/trunks/build/generator/swig/__init__.py",
line 54, in checkout
    os.remove(out)
OSError: [Errno 2] No such file or directory:
'subversion/bindings/swig/proxy/runtime.swg'
make: *** [subversion/bindings/swig/proxy/swig_ruby_external_runtime.swg]
Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "build/generator/swig/external_runtime.py", line 107, in ?
    gen.write()
  File "build/generator/swig/external_runtime.py", line 98, in write
    self.write_external_runtime()
  File "build/generator/swig/external_runtime.py", line 30, in
write_external_runtime
    self.checkout(".","runtime.swg")
  File "/nobackup/clgrp/james/trunks/build/generator/swig/__init__.py", line 56,
 in checkout
    shutil.copy("%s/%s/%s" % (self.swig_libdir, dir, file), out)
  File "/h/46/james/lib/python2.4/shutil.py", line 82, in copy
    copymode(src, dst)
  File "/h/46/james/lib/python2.4/shutil.py", line 61, in copymode
    os.chmod(dst, mode)
OSError: [Errno 2] No such file or directory:
'subversion/bindings/swig/proxy/runtime.swg'
make: *** [subversion/bindings/swig/proxy/swig_perl_external_runtime.swg]
Error 1
}}}

My patch fixes the problem, because it ensures that
external_runtime.py is only run once, and therefore is not run
multiple times in parallel.

Cheers,

David

P.S. (It's also possible to fix this issue by updating
external_runtime.py to generate each dependency, including the long
long fix and the checked-out SWIG headers, separately. To implement
this, we'd probably split up external_runtime.py into several scripts,
including a 'swig_checkout.py'. Would anyone like to work on this?)

--
David James -- http://www.cs.toronto.edu/~james