[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Hook script for sync via FTP

From: Daniel Falk <dan.svnlist_at_mbx.zapto.org>
Date: 2007-08-16 14:41:06 CEST

Hello,

I have written a python script that I have found very useful for some of
my own repositories and would like to contribute it to the community. I
am attaching it. Since it is a script I hope it doesn't get blocked.

My script is called svn2ftp.py and it will ftp changed files,
directories and deletions between revisions, using svn diff --summarize
to know what to sync. It holds onto the last successful sync's revision
number so it will always sync the correct stuff even if it fails for
some reason.

I wrote this to work on my Linux system. I don't know if it will work
on other platforms yet. It requires both pysvn and python subversion
bindings currently. I don't know whether it can be trimmed down to only
having one of these dependencies. Hopefully someone else knows. This
is my first time writing a python script, so I don't expect that I did
everything right. Which brings up a question. How are these
contributed scripts maintained? Are they made part of the main
subversion repository, which developers can submit patches to?

Thanks,
Dan

#!/usr/bin/env python

"""Usage: svn2ftp.py [OPTION...] FTP-HOST REPOS-PATH

Upload to FTP-HOST changes committed to the Subversion repository at
REPOS-PATH. Uses svn diff --summarize to only propagate the changed files

Options:

 -?, --help Show this help message.

 -u, --ftp-user=USER The username for the FTP server. Default: 'anonymous'

 -p, --ftp-password=P The password for the FTP server. Default: '@'
                                                
 -P, --ftp-port=X Port number for the FTP server. Default: 21

 -r, --remote-dir=DIR The remote directory that is expected to resemble the
                        repository project directory

 -a, --access-url=URL This is the URL that should be used when trying to SVN
                        export files so that they can be uploaded to the FTP
                        server

 -s, --status-file=PATH Required. This script needs to store the last
                          successful revision that was transferred to the
                          server. PATH is the location of this file.

 -d, --project-directory=DIR If the project you are interested in sending to
                               the FTP server is not under the root of the
                               repository (/), set this parameter.
                               Example: -d 'project1/trunk/'

"""

import getopt
import sys
import os
import tempfile
from svn import fs, repos, core, client, wc
import pysvn
import ftplib

#defaults
host = ""
user = "anonymous"
password = "@"
port = 21
repo_path = ""
local_repos_path = ""
status_file = ""
project_directory = ""
remote_base_directory = ""

def usage_and_exit(errmsg):
        """Print a usage message, plus an ERRMSG (if provided), then exit.
        If ERRMSG is provided, the usage message is printed to stderr and
        the script exits with a non-zero error code. Otherwise, the usage
        message goes to stdout, and the script exits with a zero
        errorcode."""
        if errmsg is None:
                stream = sys.stdout
        else:
                stream = sys.stderr
        print >> stream, __doc__
        if errmsg:
                print >> stream, "\nError: %s" % (errmsg)
                sys.exit(2)
        sys.exit(0)

                        
def read_args():
        global host
        global user
        global password
        global port
        global repo_path
        global local_repos_path
        global status_file
        global project_directory
        global remote_base_directory
        

        try:
                opts, args = getopt.gnu_getopt(sys.argv[1:], "?u:p:P:r:a:s:d:",
                        ["help",
                        "ftp-user=",
                        "ftp-password=",
                        "ftp-port=",
                        "ftp-remote-dir=",
                        "access-url=",
                        "status-file=",
                        "project-directory=",
                        ])
        except getopt.GetoptError, msg:
                usage_and_exit(msg)

        for opt, arg in opts:
                if opt in ("-?", "--help"):
                        usage_and_exit()
                elif opt in ("-u", "--ftp-user"):
                        user = arg
                elif opt in ("-p", "--ftp-password"):
                        password = arg
                elif opt in ("-P", "--ftp-port"):
                        try:
                           port = int(arg)
                        except ValueError, msg:
                           usage_and_exit("Invalid value '%s' for --ftp-port." % (arg))
                        if port < 1 or port > 65535:
                           usage_and_exit("Value for --ftp-port must be a positive integer less than 65536.")
                elif opt in ("-r", "--ftp-remote-dir"):
                        remote_base_directory = arg
                elif opt in ("-a", "--access-url"):
                        repo_path = arg
                elif opt in ("-s", "--status-file"):
                        status_file = os.path.abspath(arg)
                elif opt in ("-d", "--project-directory"):
                        project_directory = arg

        if len(args) != 2 :
                print str(args)
                usage_and_exit("host and/or local_repos_path not specified (" + len(args) + ")")
        
        host = args[0]
        print "args1: " + args[1]
        print "args0: " + args[0]
        print "abspath: " + os.path.abspath(args[1])
        local_repos_path = os.path.abspath(args[1])
        
        if status_file == "" : usage_and_exit("No status file specified")
        

        
def main():
        global host
        global user
        global password
        global port
        global repo_path
        global local_repos_path
        global status_file
        global project_directory
        global remote_base_directory

        read_args()

        
        #get youngest revision
        print "local_repos_path: " + local_repos_path
        repository = repos.open(local_repos_path)
        fs_ptr = repos.fs(repository)
        youngest_revision = fs.youngest_rev(fs_ptr)
        
        last_sent_revision = get_last_revision()
        
        if youngest_revision == last_sent_revision :
                # no need to continue. we should be up to date.
                return

        rev1 = pysvn.Revision(pysvn.opt_revision_kind.number, last_sent_revision)
        rev2 = pysvn.Revision(pysvn.opt_revision_kind.number, youngest_revision)

        pysvn_client = pysvn.Client()
        summary = pysvn_client.diff_summarize(repo_path, rev1, repo_path, rev2, True, False)

        if len(summary) > 0 :
                ftp = FTPClient(host, user, password)
                ftp.base_path = remote_base_directory
        
                #iterate through all the differences between revisions
                for change in summary :
                        #determine whether the path of the change is relevant to the path that is being sent, and modify the path as appropriate.
                        ftp_relative_path = apply_basedir(change.path)
                        
                        #only try to sync path if the path is in our project_directory
                        if ftp_relative_path != "" :
                                is_file = (change.node_kind == pysvn.node_kind.file)
                                if str(change.summarize_kind) == "delete" :
                                        print "deleting: " + ftp_relative_path
                                        ftp.delete_path("/" + ftp_relative_path, is_file)
                                elif str(change.summarize_kind) == "added" or str(change.summarize_kind) == "modified" :
                                        local_file = ""
                                        if is_file :
                                                local_file = svn_export_temp(pysvn_client, repo_path, rev2, change.path)
                                        print "uploading file: " + ftp_relative_path
                                        ftp.upload_path("/" + ftp_relative_path, is_file, local_file)
                                        if is_file :
                                                os.remove(local_file)
                                elif str(change.summarize_kind) == "normal" :
                                        print "skipping 'normal' element: " + ftp_relative_path
                                else :
                                        raise str("Unknown change summarize kind: " + str(change.summarize_kind) + ", path: " + ftp_relative_path)
                ftp.close()
        
        #write back the last revision that was synced
        print "writing last revision: " + str(youngest_revision)
        set_last_revision(youngest_revision)

#functions for persisting the last successfully synced revision
def get_last_revision():
        if os.path.isfile(status_file) :
                f=open(status_file, 'r')
                line = f.readline()
                f.close()
                try: i = int(line)
                except ValueError:
                        i = 0
        else:
                i = 0
        f = open(status_file, 'w')
        f.write(str(i))
        f.close()
        return i
        
def set_last_revision(rev) :
        f = open(status_file, 'w')
        f.write(str(rev))
        f.close()
        

#augmented ftp client class that can work off a base directory
class FTPClient(ftplib.FTP) :
        def __init__(self, host, username, password) :
                self.base_path = ""
                self.current_path = ""
                ftplib.FTP.__init__(self, host, username, password)
        
        def cwd(self, path) :
                debug_path = path
                if self.current_path == "" :
                        self.current_path = self.pwd()
                        print "pwd: " + self.current_path

                if not os.path.isabs(path) :
                        debug_path = self.base_path + "<" + path
                        path = os.path.join(self.current_path, path)
                elif self.base_path != "" :
                        debug_path = self.base_path + ">" + path.lstrip("/")
                        path = os.path.join(self.base_path, path.lstrip("/"))
                path = os.path.normpath(path)
                
                #by this point the path should be absolute.
                if path != self.current_path :
                        print "change from " + self.current_path + " to " + debug_path
                        ftplib.FTP.cwd(self, path)
                        self.current_path = path
                else :
                        print "staying put : " + self.current_path
        
        def cd_or_create(self, path) :
                assert(os.path.isabs(path), "absolute path expected (" + path + ")")
                try: self.cwd(path)
                except ftplib.error_perm, e:
                        for folder in path.split('/'):
                                if folder == "" :
                                        self.cwd("/")
                                        continue

                                try: self.cwd(folder)
                                except:
                                        print "mkd: (" + path + "):" + folder
                                        self.mkd(folder)
                                        self.cwd(folder)

        def upload_path(self, path, is_file, local_path) :
                if is_file :
                        (path, filename) = os.path.split(path)
                        self.cd_or_create(path)
                        f = open(local_path, 'r')

                        self.storbinary("STOR " + filename, f)
                        
                        f.close()
                else :
                        self.cd_or_create(path)
        
        def delete_path(self, path, is_file) :
                (path, filename) = os.path.split(path)
                print "trying to delete: " + path + ", " + filename
                self.cwd(path)
                if is_file :
                        self.delete(filename)
                else :
                        self.delete_path_recursive(filename)
                
        def delete_path_recursive(self, path):
                if path == "/" :
                        raise "WARNING: trying to delete '/'!"
                #print "enter: " + path
                for node in self.nlst(path) :
                        if node == path :
                                #it's a file. delete and return
                                #print "deleting: " + path
                                self.delete(path)
                                return
                        #print node + ", " + os.path.join(path, node)
                        if node != "." and node != ".." :
                                self.delete_path_recursive(os.path.join(path, node))
                #print "deleting directory: " + path
                try: self.rmd(path)
                except ftplib.error_perm, msg :
                        sys.stderr.write("Error deleting directory " + os.path.join(self.current_path, path) + " : " + str(msg))
                        

#apply the project_directory setting
def apply_basedir(path) :
        #remove any leading stuff (in this case, "trunk/") and decide whether file should be propagated
        if not path.startswith(project_directory) :
                return ""
        return path.replace(project_directory, "", 1)
                
def svn_export_temp(pysvn_client, base_path, rev, path) :
        (fd, dest_path) = tempfile.mkstemp()

        pysvn_client.export( os.path.join(base_path, path),
                        dest_path,
                        force=False,
                        revision=rev,
                        native_eol=None,
                        ignore_externals=False,
                        recurse=True,
                        peg_revision=rev )
        
        return dest_path

if __name__ == "__main__":
        main()

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Thu Aug 16 14:39:09 2007

This is an archived mail posted to the Subversion Dev mailing list.