Compiling svn + httpd for windows python 2.x

Folks,

We use subversion with Trac behind httpd on Windows Server. As Trac is written in "old" Python (2.x), I have had to resort to building everything from source. This is not simple and so I thought I would publish my notes here in case it helps anyone else and in the hope that if I have made any mistakes, someone will be kind enough to point them out to me!

~ Mark C

These are my notes for building Apache httpd and subversion for use with Trac [1]. All of the components need to be built using the same compiler to avoid run-time issues and, since Trac currently relies on Python 2.x, that means Visual Studio 2008.

[1] http://trac.edgewall.org

Pre-requisites
--------------
All of the following assumes a suitable build environment, which for me is:-
- Windows 7 Enterprise (virtual machine)
- Visual Studio Professional 2008 (plus SP1) == VC9
- Python 2.x (currently 2.7.11) [2]
  - PyWin32 [3]
- StrawberryPerl [4]
- "awk" [5] (can't remember why)
- SCons [6] for building the Serf library
- CMake [7] for building the PCRE library
- SWIG [8] for the Python bindings
- 7-zip [9] for unpacking all the downloads!

[2] https://www.python.org/downloads/
[3] http://sourceforge.net/projects/pywin32/files/pywin32/
[4] http://strawberryperl.com/
[5] https://www.cs.princeton.edu/~bwk/btl.mirror/
[6] http://www.scons.org/
[7] http://www.cmake.org/download/
[8] http://www.swig.org/download.html
[9] http://7-zip.org

Folders
-------
In the following I assume the following folder structure:-
D:\svn
D:\svn\httpd
D:\svn\httpd\srclib\apr
D:\svn\httpd\srclib\apr-iconv
D:\svn\httpd\srclib\apr-util
D:\svn\httpd\srclib\openssl
D:\svn\httpd\srclib\pcre
D:\svn\serf
D:\svn\sqlite
D:\svn\src
D:\svn\zlib

===============================================================================

1) Apache httpd

Download the latest httpd source tarball: https://httpd.apache.org/download.cgi
Unpack the source to:
D:\svn\httpd

-------------------------------------------------------------------------------
2) APR source tarballs

Download from: https://apr.apache.org/download.cgi

Unpack the archives to the * httpd\srclib * folder:
D:\svn\httpd\srclib\apr
D:\svn\httpd\srclib\apr-iconv
D:\svn\httpd\srclib\apr-util

Remember to remove the version numbers from the folder names!

Note: last I tried, apr 1.5.2 causes a number of build errors:
http://mail-archives.apache.org/mod_mbox/apr-dev/201504.mbox/%3c553DB43A.3020408@gknw.net%3e
Reverting to 1.5.1... works

-------------------------------------------------------------------------------
3) OpenSSL

Download the latest OpenSSL source: http://www.openssl.org/source/

Unpack the source to the srclib folder:
D:\svn\httpd\srclib\openssl

See also: http://wiki.openssl.org/index.php/Compilation_and_Installation

-------------------------------------------------------------------------------
4) Download the latest PCRE source: http://www.pcre.org/

Unpack the PCRE (Perl Compatible Regular Expressions) to the srclib folder:
D:\svn\httpd\srclib\pcre

-------------------------------------------------------------------------------
5) Fix line endings

Run `lineends.pl` from the root folder to convert line endings from *nix to
windoze standard (NB: recurses down from current folder):

{{{
d:\> cd svn\httpd
d:\svn\httpd\> perl srclib\apr\build\lineends.pl
}}}

-------------------------------------------------------------------------------
6) Build OpenSSL
-----------------

Run the following in a * Visual Studio * command prompt window...
{{{
cd \svn\httpd\srclib\openssl
perl Configure no-asm no-comp no-rc5 no-idea no-ssl2 enable-mdc2 VC-WIN32
ms\do_ms.bat
nmake -f ms\ntdll.mak
}}}
Note: specifying "no-ssl2 no-ssl3" will result in linker errors!

-------------------------------------------------------------------------------
7) Build PCRE
--------------

[from http://wiki.greenstone.org/doku.php?id=en:developer:compiling_apache]

- Read `srclib/pcre/NON-AUTOTOOLS-BUILD`
  section "BUILDING PCRE ON WINDOWS WITH CMAKE"
- Run the CMake GUI from a VS2008 command prompt ("start cmake-gui")
- Specify the source and output folders as ...\srclib\pcre
- Hit `Configure`
- Select `NMake Makefiles` for the generator, using default native compilers
- Edit the red block of values:
  - BUILD_SHARED LIBS
  - Set CMAKE_BUILD_TYPES to RelWithDebInfo
  - Set CMAKE_INSTALL_PREFIX to the \srclib\pcre folder
  ? Set both unicode support flags (seems likely to be useful?)
  ? not sure about PCRE_NEWLINE for windoze? default: LF (*nix)
- Hit `Configure` again, then `Generate` (then close cmake-gui).

- From the output folder (e.g. srclib\pcre):
{{{
nmake -f makefile
}}}
...should make pcre.dll in the pcre folder, where the apache makefile expects
it to be.

-------------------------------------------------------------------------------
8) Prepare apr, apr-util, apr-iconv, httpd & modules...
------------------------------------------------------

** First ** convert the project files (this avoids lots of RC errors about
embedded quotes around strings in descriptions) using:
{{{
cd \svn\httpd
perl srclib\apr\build\cvtdsp.pl -2005
}}}

** Second ** clear out (rename?) any old \apache24 folder...

The command-line build did not work, so open the `apache.dsw` file:
{{{
start apache.dsw
}}}
...and allow the projects to convert to VS2008 format.

*******
*** Save and close, then add-in the mod_wsgi module...
*******
Extract & rename the mod_wsgi "server" folder into the httpd tree:
- from: mod_wsgi-x.y.z.zip\mod_wsgi-x.y.z\src\server
- to : httpd\modules\wsgi

Copy and edit the "autoindex" project file:
{{{
copy modules\generators\mod_autoindex.vcxproj modules\wsgi\mod_wsgi.vcxproj
}}}

- Edit the vcxproj file:
  - replace all `autoindex` => `wsgi`
  - deleted the <ProjectGUID /> value (will be recreated)

(Re)Start Visual Studio ("start apache.sln")...
- add the project into the httpd solution (like mod_autoindex).
- add all modules\wsgi\*.c files into the project (NB: mod_wsgi.c already included)
- use the properties dialog on the `mod_wsgi` node to:
  - add the Python27\include folder to the compiler options (all builds)
    - C/C++ > General > Additional Include Directories
  - add the Python27\libs folder to the linker options (all builds)
    - Linker > General > Additional Library Directories
  - add the extra linker dependencies (Release builds):
    - Linker > General > Additional Dependencies
    - $(SolutionDir)release\libhttpd.lib
    - $(SolutionDir)srclib\apr\release\libapr-1.lib
    - $(SolutionDir)srclib\apr-util\release\libaprutil-1.lib

-------------------------------------------------------------------------------
9) Build!
    
- Set for Release Win32 build.
- Select and Build the InstallBin project

The InstallBin target should create an Apache24 folder in the root, containing
all of the files required to run httpd (ish).

Copy the mod_wsgi module to the output folder:
{{{
copy modules\wsgi\Release\mod_wsgi.so \apache24\modules
}}}

===============================================================================
================================================================ SUBVERSION ===
===============================================================================

Primary resources:
[1] The INSTALL file in the root of the source tree

Blogs:-
[2] http://blogs.collab.net/subversion/building-subversion-on-windows-a-walk-through
[3] http://svn.haxx.se/dev/archive-2013-04/0332.shtml (Ben Reser)

Download the latest sources:

D:\svn\serf
D:\svn\sqlite
D:\svn\src
D:\svn\zlib

- serf: http://serf.apache.org/download
- sqlite "amalgamation": https://www.sqlite.org/download.html
- subversion: http://subversion.apache.org/download/
- zlib: http://www.zlib.net/ (ZLib source in ZIP file, half-way down)

-------------------------------------------------------------------------------
10) Build zlib:

Extract zlib source to \svn\zlib
{{{
cd \svn\zlib
nmake -f win32\makefile.msc
}}}

Note 1: last I looked there were bugs with the MS ASM build => not using asm.
Note 2: 1.9.x requires zlibstat:

edit D:\svn\zlib\contrib\vstudio\vc9\zlibstat.vcproj:
 - search "ZLIB_WINAPI" replace ""
{{{
vcbuild /rebuild contrib\vstudio\vc9\zlibvc.sln "ReleaseWithoutAsm|Win32"
#copy `zlib\contrib\vstudio\vc9\x86\ZlibDllReleaseWithoutAsm\zlibwapi.dll`
# - to `zlib\zlib.dll`
copy zlib\contrib\vstudio\vc9\x86\ZlibStatReleaseWithoutAsm\zlibstat.lib zlib\zlibstat.lib
}}}

(see: D:\svn\zlib\contrib\vstudio\readme.txt)
(and: http://www.tannerhelland.com/5076/compile-zlib-winapi-wapi-stdcall/)
(but: adding ZLIB_WINAPI is _wrong_ ~ it needs to be _removed_ from the projects!)

-------------------------------------------------------------------------------
11) Build serf:

Extract serf source to \svn\serf

Build from VS cmd window (requires SCons to be installed):
{{{
cd \svn\serf
c:\Python27\Scripts\scons.py APR=\svn\httpd\srclib\apr
                             APU=\svn\httpd\srclib\apr-util
                             OPENSSL=\svn\httpd\srclib\openssl
                             ZLIB=\svn\zlib
}}}

-------------------------------------------------------------------------------
12) Unpack sqlite:

Extract the sqlite "amalgamation" source to \svn\sqlite

-------------------------------------------------------------------------------
13) Subversion:

Unpack the ZIP archive into D:\svn\src

Configure:
{{{
D:\svn\src> gen-make.py -t vcproj --vsnet-version=2008
                        --with-httpd=D:\svn\httpd
                        --with-openssl=d:\svn\httpd\srclib\openssl
                        --with-serf=d:\svn\serf
                        --with-sqlite=d:\svn\sqlite
                        --with-swig=c:\programs\swig
                        --with-zlib=d:\svn\zlib
}}}

Open the subversion_vcnet solution file ("start subversion_vcnet.sln"):
- Select for a Release build
- Check that the `__ALL_TESTS__` project is set as the default
- Build...

This resulted in instances of [3] errors:-
[1] fatal error C1083: Cannot open include file: 'boost/shared_ptr.hpp': No such file or directory
[2] 9x fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
[3] fatal error C1083: Cannot open include file: 'swigutil_pl__pre_perl.h': No such file or directory

...and all come from bindings I do not need (and am not configured to build).

---
Run the tests (copy most of the following to e.g. "run-tests.bat")...
{{{
cd \svn\src
setlocal
set path=%path%;d:\Apache24\bin
set path=%path%;d:\svn\httpd\srclib\apr\Release
set path=%path%;d:\svn\httpd\srclib\apr-util\Release
set path=%path%;d:\svn\httpd\srclib\apr-iconv\Release
win-tests.py --release --cleanup --parallel
win-tests.py -r -c --httpd-dir=d:\Apache24 --httpd-daemon
}}}
-------------------------------------------------------------------------------
14) Python bindings...
http://www.lejordet.com/2009/03/compiling-subversion-python-bindings-on-windows/
...or: D:\svn\src\subversion\bindings\swig\INSTALL
These should have built Ok but need to be collected together for installation.
A) Create a <py-bind>\svn folder somewhere and copy:
- svn\src\subversion\bindings\swig\python\svn\*.py
B) Create a <py-bind>\libsvn folder and copy:
- svn\src\subversion\bindings\swig\python\*.py
- svn\src\Release\subversion\bindings\swig\python\*.pyd
- svn\src\Release\subversion\bindings\swig\python\libsvn_swig_py\*.dll
These files should be installed to the server:-
- C:\Python27\Lib\site-packages\(lib)svn
{{{
cd \svn\src
mkdir \Apache24\python\svn
copy subversion\bindings\swig\python\svn\*.py \Apache24\svn\svn
mkdir \Apache24\python\libsvn
copy subversion\bindings\swig\python\*.py \Apache24\svn\libsvn
copy Release\subversion\bindings\swig\python\*.pyd \Apache24\svn\libsvn
copy Release\subversion\bindings\swig\python\libsvn_swig_py\*.dll \Apache24\svn\libsvn
}}}
===============================================================================
Installation
...is mainly left for the user to match their local systems.
Copy the required files:
- Apache httpd
  - remember to exclude the *.pdb files (if not required)
  - should already include mod_wsgi
- Subversion (main)
  - D:\svn\src\Release\*.dll
  - D:\svn\src\Release\subversion\*\svn*.exe (except `svndiff-test.exe`)
  - D:\svn\src\Release\tools\*\svnauthz*.exe (from server-side)
- Subversion (Python bindings)
  - \Apache24\python (see 13 above) copied to server's python Lib\site-packages folder
- Subversion (httpd modules)
  - D:\svn\src\Release\*.so (to apache \modules folder)
Now you "just" need to install the httpd service and fix the httpd "conf" files
as well as set appropriate folder permissions etc. as required by local policy.
(end)