[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

[PATCH] svnadmin hotcopy, hot-backup.py race conditions fix

From: Vladimir Berezniker <vmpn_at_tigris.org>
Date: 2003-09-08 23:29:36 CEST

Here is the hot copy patch. The patch implements svnadmin hotcopy command with
optional --archive-logs flag. If flag is specified, after the copying is
complete, unused *copied* log files are deleted from the source repository.

There is something that you should be aware of. The /db and /locks directories
are *not* copied file by file. Only the files that should be there are copied.
So if user places their own files in those folders, they will *not* be copied.
However, I believe user should not be putting anything in /db and /lock are
specific to one repository. Any objections?

I have one more question regarding /dav directory. Does it need to copied, or
should it be discarded? As I understood the code, /dav is only used to keep
track of actions while client works with the svn_dav module, and as such does
not need to be copied over.

P.S. The reason there was such a delay between the first draft and this patch is
because I spent the last week in bed, sick. :(

Sincerely,
Vladimir Berezniker

-----------------
Log Message:
-----------------
Implemented hot copy functionality for subversion. This functionality fixes two
race conditions existing in hot-backup.py. Only the logs that have been
successfully copied are archived, therefore any log modified while copy is in
progress is not archived. Also (svn_repos_hotcopy) takes out shared db lock,
eliminating possibility of corruption, if running recovery in parallel to
automated backup.
Updated hot-backup.py to wrap around the new hot copy functionality.

* subversion/include/svn_error.h
     (SVN_ERR_POOL): Implemented macro to handle subversion errors in code that
uses subpools.

* subversion/include/svn_io.h
* subversion/libsvn_subr/io.c
     (svn_io_file_create): Factored out function for file creation.
     (svn_io_file_lock): Factored out function for file locking.
     (svn_io_dir_file_copy): Implemented function for copying a file between two
directories.

* subversion/include/svn_fs.h
     (svn_fs_hotcopy_berkeley): Added prototype for Berkeley hot copy function.

* subversion/include/svn_repos.h
     (svn_repos_hotcopy): Added prototype for subversion repository hot copy
function.

* subversion/libsvn_fs/fs.c:
     (svn_fs__archive_logs): Implemented function that archives only copied
unused Berkeley DB logs.
     (svn_fs_hotcopy_berkeley): Implemented hot copy functionality in accordance
with Berkeley DB documentation.

* subversion/libsvn_repos/repos.h
     (SVN_REPOS__DB_LOGS_LOCKFILE): Added new definition for BDB logs files lock
file.

* subversion/libsvn_repos/repos.c
     (create_db_lock): Factored out function for creation of db lock file.
     (create_locks): Cleanup. Deleted redundant path parameter.
     (create_hooks): Cleanup. Deleted redundant path parameter. Replaced
redundant code with use of svn_io_file_create.
     (hotcopy_structure): Addapted, deleted, (copy_structure) for copying
repository structure with exception of /db and /locks directories.
     (svn_repos_db_logs_lockfile): Implemented function to return path to db
logs lock file.
     (create_db_logs_lock): Implemented function for creation of db logs lock file.
     (lock_db_logs_file): Function for locking db logs lock file.
     (svn_repos_hotcopy): Implemented function to make a hot copy of a repository.

* subversion/svnadmin/main.c
     Added new flag "archive-logs" to specify that logs are to be archived after
the hot copy is complete.
     (parse_local_repos_path): Factored out function for parsing and validating
local repository path.
     (subcommand_hotcopy): Implemented new hotcopy subcommand.

* tools/backup/hot-backup.py.in
     Updated hot backup script to utilize the new hot copy functionality.

-----------------------
The patch.
-----------------------

Index: subversion/svnadmin/main.c
===================================================================
--- subversion/svnadmin/main.c (revision 7005)
+++ subversion/svnadmin/main.c (working copy)
@@ -50,11 +50,43 @@
    return SVN_NO_ERROR;
  }

+static svn_error_t *
+parse_local_repos_path(apr_getopt_t *os, const char ** repos_path,
+ apr_pool_t *pool)
+{
+ *repos_path = NULL;

+ if (os->ind < os->argc)
+ {
+ const char * path = os->argv[os->ind++];
+ SVN_ERR (svn_utf_cstring_to_utf8 (repos_path,
+ path,
+ NULL, pool));
+ *repos_path = svn_path_internal_style (*repos_path, pool);
+ }
+
+ if (*repos_path == NULL)
+ {
+ return svn_error_create (SVN_ERR_CL_ARG_PARSING_ERROR,
+ NULL,
+ "repository argument required\n");
+ }
+ else if (svn_path_is_url (*repos_path))
+ {
+ return svn_error_createf (SVN_ERR_CL_ARG_PARSING_ERROR,
+ NULL,
+ "'%s' is an url when it should be a path\n",
+ *repos_path);
+ }
+
+ return SVN_NO_ERROR;
+}
+
  
  /** Subcommands. **/

  static svn_opt_subcommand_t
+ subcommand_hotcopy,
    subcommand_create,
    subcommand_createtxn,
    subcommand_dump,
@@ -78,7 +110,8 @@
      svnadmin__force_uuid,
      svnadmin__parent_dir,
      svnadmin__bdb_txn_nosync,
- svnadmin__config_dir
+ svnadmin__config_dir,
+ svnadmin__archive_logs
    };

  /* Option codes and descriptions.
@@ -126,6 +159,9 @@
      {"config-dir", svnadmin__config_dir, 1,
       "read user configuration files from directory ARG"},

+ {"archive-logs", svnadmin__archive_logs, 0,
+ "delete copied, unused log files from the source repository."},
+
      {NULL}
    };

@@ -160,6 +196,11 @@
       "Display this usage message.\n",
       {svnadmin__version} },

+ {"hotcopy", subcommand_hotcopy, {0},
+ "usage: svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH\n\n"
+ "Makes a hot copy of a repository.\n\n",
+ {svnadmin__archive_logs} },
+
      {"load", subcommand_load, {0},
       "usage: svnadmin load REPOS_PATH\n\n"
       "Read a 'dumpfile'-formatted stream from stdin, committing\n"
@@ -229,6 +270,7 @@
  struct svnadmin_opt_state
  {
    const char *repository_path;
+ const char *new_repository_path;
    svn_opt_revision_t start_revision, end_revision; /* -r X[:Y] */
    svn_boolean_t help; /* --help or -? */
    svn_boolean_t version; /* --version */
@@ -236,6 +278,7 @@
    svn_boolean_t follow_copies; /* --copies */
    svn_boolean_t quiet; /* --quiet */
    svn_boolean_t bdb_txn_nosync; /* --bdb-txn-nosync */
+ svn_boolean_t archive_logs; /* --archive-logs */
    enum svn_repos_load_uuid uuid_action; /* --ignore-uuid,
                                                         --force-uuid */
    const char *parent_dir;
@@ -644,6 +687,21 @@
  }

+/* This implements `svn_opt_subcommand_t'. */
+svn_error_t *
+subcommand_hotcopy (apr_getopt_t *os, void *baton, apr_pool_t *pool)
+{
+ struct svnadmin_opt_state *opt_state = baton;
+
+ SVN_ERR (svn_repos_hotcopy(opt_state->repository_path,
+ opt_state->new_repository_path,
+ opt_state->archive_logs,
+ pool));
+
+ return SVN_NO_ERROR;
+}
+
+
  
  /** Main. **/

@@ -788,6 +846,9 @@
          opt_state.config_dir = apr_pstrdup (pool, svn_path_canonicalize(opt_arg,
                                                                         pool));
          break;
+ case svnadmin__archive_logs:
+ opt_state.archive_logs = TRUE;
+ break;
        default:
          {
            subcommand_help (NULL, NULL, pool);
@@ -834,36 +895,36 @@
       here and store it in opt_state. */
    if (subcommand->cmd_func != subcommand_help)
      {
- const char *repos_path = NULL;
-
- if (os->ind < os->argc)
+ err = parse_local_repos_path(os,
+ &(opt_state.repository_path),
+ pool);
+ if(err)
          {
- opt_state.repository_path = os->argv[os->ind++];
- SVN_INT_ERR (svn_utf_cstring_to_utf8 (&(opt_state.repository_path),
- opt_state.repository_path,
- NULL, pool));
- repos_path
- = svn_path_internal_style (opt_state.repository_path, pool);
- }
-
- if (repos_path == NULL)
- {
- fprintf (stderr, "repository argument required\n");
- subcommand_help (NULL, NULL, pool);
+ svn_handle_error (err, stderr, 0);
+ svn_opt_subcommand_help (subcommand->name, cmd_table,
+ options_table, pool);
            svn_pool_destroy (pool);
            return EXIT_FAILURE;
          }
- else if (svn_path_is_url (repos_path))
+
+ }
+
+
+ /* If command is hot copy the third argument will be the new
+ repository path. */
+ if (subcommand->cmd_func == subcommand_hotcopy)
+ {
+ err = parse_local_repos_path(os,
+ &(opt_state.new_repository_path),
+ pool);
+ if(err)
          {
- fprintf (stderr,
- "'%s' is a url when it should be a path\n",
- repos_path);
+ svn_handle_error (err, stderr, 0);
+ svn_opt_subcommand_help (subcommand->name, cmd_table,
+ options_table, pool);
            svn_pool_destroy (pool);
            return EXIT_FAILURE;
          }
-
- /* Copy repos path into the OPT_STATE structure. */
- opt_state.repository_path = repos_path;
      }

    /* Check that the subcommand wasn't passed any inappropriate options. */
Index: subversion/include/svn_fs.h
===================================================================
--- subversion/include/svn_fs.h (revision 7005)
+++ subversion/include/svn_fs.h (working copy)
@@ -118,6 +118,15 @@
   */
  svn_error_t *svn_fs_create_berkeley (svn_fs_t *fs, const char *path);

+/** Hot copy Subversion filesystem, stored in a Berkeley DB environment under
+ * @a src_path to @a dest_path. If @a archive_logs is used is @c TRUE,
+ * delete copied unused log files from source repository at @a src_path
+ * Using @a pool for any necessary memory allocations.
+ */
+svn_error_t *svn_fs_hotcopy_berkeley (const char *src_path,
+ const char *dest_path,
+ svn_boolean_t archive_logs,
+ apr_pool_t *pool);

  /** Make @a fs refer to the Berkeley DB-based Subversion filesystem at
   * @a path. @a path is utf8-encoded, and must refer to a file or directory
Index: subversion/include/svn_repos.h
===================================================================
--- subversion/include/svn_repos.h (revision 7005)
+++ subversion/include/svn_repos.h (working copy)
@@ -88,6 +88,16 @@
                                 apr_hash_t *fs_config,
                                 apr_pool_t *pool);

+/** Make a hot copy of the Subversion repository found at @a src_path
+ * to @a dst_path.
+ *
+ * @copydoc svn_fs_hotcopy_berkeley()
+ */
+svn_error_t * svn_repos_hotcopy (const char *src_path,
+ const char *dst_path,
+ svn_boolean_t archive_logs,
+ apr_pool_t *pool);
+
  /** Destroy the Subversion repository found at @a path, using @a pool for any
   * necessary allocations.
   */
Index: subversion/include/svn_io.h
===================================================================
--- subversion/include/svn_io.h (revision 7005)
+++ subversion/include/svn_io.h (working copy)
@@ -292,8 +292,31 @@
                                             const char *file2,
                                             apr_pool_t *pool);

+/** Create file at @a file with contents @a contents.
+ * will be created. Path @a file is utf8-encoded.
+ * Use @a pool for memory allocations.
+ */
+svn_error_t *svn_io_file_create (const char *file,
+ const char *contents,
+ apr_pool_t *pool);

+/** Lock file at @a lock_file. Ff @exclusive is TRUE,
+ * obtain exclusive lock, otherwise obtain shared lock.
+ * Lock will be automaticaly released when @a pool is cleared or destroyed.
+ * Use @a pool for memory allocations.
+ */
+svn_error_t *svn_io_file_lock (const char *lock_file,
+ svn_boolean_t exclusive,
+ apr_pool_t *pool);

+/** Copy file @a file from location @a src_path to location @a dest_path.
+ * Use @a pool for memory allocations.
+ */
+svn_error_t *svn_io_dir_file_copy (const char *src_path,
+ const char *dest_path,
+ const char *file,
+ apr_pool_t *pool);
+
  
  /** Generic byte-streams
   *
Index: subversion/include/svn_error.h
===================================================================
--- subversion/include/svn_error.h (revision 7005)
+++ subversion/include/svn_error.h (working copy)
@@ -198,6 +198,21 @@
        return EXIT_FAILURE; } \
    } while (0)

+/** A statement macro, very similar to @c SVN_ERR.
+ * This macro will check for error and if one exists
+ * will destroy the specified pool. Usefull for functions
+ * that create local subpools that need to be properly disposed
+ * of in case of an error
+ */
+#define SVN_ERR_POOL(expr, subpool) \
+ do { \
+ svn_error_t *svn_err__temp = (expr); \
+ if (svn_err__temp) { \
+ svn_pool_destroy(subpool); \
+ return svn_err__temp; \
+ } \
+ } while (0)
+
  /** @} */

  #ifdef __cplusplus
Index: subversion/libsvn_fs/fs.c
===================================================================
--- subversion/libsvn_fs/fs.c (revision 7005)
+++ subversion/libsvn_fs/fs.c (working copy)
@@ -531,6 +531,121 @@
    return svn_err;
  }

+/**
+ * Archive all unused log files that have been copied to a different location
+ */
+svn_error_t *
+svn_fs__archive_logs(const char *live_path,
+ const char *backup_path,
+ apr_pool_t *pool)
+{
+
+ apr_array_header_t *logfiles;
+
+ SVN_ERR (svn_fs_berkeley_logfiles (&logfiles,
+ live_path,
+ TRUE, /* Only unused logs */
+ pool));
+
+ if (logfiles == NULL) {
+ return SVN_NO_ERROR;
+ }
+
+ { /* Process unused logs from live area */
+ int log;
+ apr_pool_t *sub_pool = svn_pool_create(pool);
+
+ /* Process log files. */
+ for (log = 0; log < logfiles->nelts; svn_pool_clear(sub_pool), log++)
+ {
+ const char * log_file = APR_ARRAY_IDX (logfiles, log, const char *);
+ const char *live_log_path
+ = svn_path_join (live_path, log_file, sub_pool);
+ const char *backup_log_path
+ = svn_path_join (backup_path, log_file, sub_pool);
+
+ { /* Compare files. No point in using MD5 and waisting CPU cycles as we
+ got full copies of both logs */
+ svn_boolean_t files_match = FALSE;
+ svn_error_t * err = svn_io_files_contents_same_p(&files_match,
+ live_log_path,
+ backup_log_path,
+ sub_pool);
+ if (err) {
+ svn_error_clear(err);
+ files_match = FALSE;
+ }
+
+
+ if(files_match == FALSE) {
+ continue;
+ }
+ }
+
+ SVN_ERR_POOL(svn_io_remove_file(live_log_path, sub_pool), sub_pool);
+ }
+
+ svn_pool_destroy(sub_pool);
+ }
+
+ return SVN_NO_ERROR;
+}
+
+svn_error_t *
+svn_fs_hotcopy_berkeley (const char *src_path,
+ const char *dest_path,
+ svn_boolean_t archive_logs,
+ apr_pool_t *pool)
+{
+ /* Check DBD version, just in case */
+ SVN_ERR (check_bdb_version (pool));
+
+ /* Copy the DB_CONFIG file. */
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"DB_CONFIG", pool));
+
+ /* Copy the databases. */
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"nodes", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"revisions", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"transactions", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"copies", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"changes", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"representations", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"strings", pool));
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path, &"uuids", pool));
+
+ {
+ apr_array_header_t *logfiles;
+ int log;
+
+ SVN_ERR (svn_fs_berkeley_logfiles (&logfiles,
+ src_path,
+ FALSE, /* All logs */
+ pool));
+
+ if (logfiles == NULL) {
+ return SVN_NO_ERROR;
+ }
+
+ /* Process log files. */
+ for (log = 0; log < logfiles->nelts; log++)
+ {
+ SVN_ERR(svn_io_dir_file_copy(src_path, dest_path,
+ APR_ARRAY_IDX (logfiles, log, const char *),
+ pool));
+ }
+ }
+
+ /* Since this is a copy we will have exclusive access to the repository. */
+ SVN_ERR(svn_fs_berkeley_recover(dest_path, pool));
+
+ if(archive_logs == TRUE)
+ {
+ SVN_ERR(svn_fs__archive_logs(src_path, dest_path, pool));
+ }
+
+ return SVN_NO_ERROR;
+}
+
  
  /* Gaining access to an existing Berkeley DB-based filesystem. */

Index: subversion/libsvn_subr/io.c
===================================================================
--- subversion/libsvn_subr/io.c (revision 7005)
+++ subversion/libsvn_subr/io.c (working copy)
@@ -182,7 +182,7 @@

  
-/*** Copying and appending files. ***/
+/*** Creating, copying and appending files. ***/

  svn_error_t *
  svn_io_copy_file (const char *src,
@@ -463,7 +463,45 @@
  #endif
  }

+svn_error_t *svn_io_file_create (const char *file,
+ const char *contents,
+ apr_pool_t *pool)
+{
+ apr_status_t apr_err;
+ apr_file_t *f = NULL;
+ apr_size_t written;

+ SVN_ERR (svn_io_file_open (&f, file,
+ (APR_WRITE | APR_CREATE | APR_EXCL),
+ APR_OS_DEFAULT,
+ pool));
+
+ apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
+ if (apr_err)
+ return svn_error_createf
+ (apr_err, NULL, "svn_io_file_create: error writing '%s'", file);
+
+ apr_err = apr_file_close (f);
+ if (apr_err)
+ return svn_error_createf
+ (apr_err, NULL,
+ "svn_io_file_create: error closing '%s'", file);
+ return SVN_NO_ERROR;
+}
+
+svn_error_t *svn_io_dir_file_copy (const char *src_path,
+ const char *dest_path,
+ const char *file,
+ apr_pool_t *pool)
+{
+ const char *file_dest_path = svn_path_join (dest_path, file, pool);
+ const char *file_src_path = svn_path_join (src_path, file, pool);
+
+ SVN_ERR (svn_io_copy_file(file_src_path, file_dest_path, TRUE, pool));
+
+ return SVN_NO_ERROR;
+}
+
  
  /*** Modtime checking. ***/

@@ -730,7 +768,74 @@
    return SVN_NO_ERROR;
  }

+
+/*** File locking. ***/
+/* Clear all outstanding locks on ARG, an open apr_file_t *. */
+static apr_status_t
+svn_io__file_clear_and_close (void *arg)
+{
+ apr_status_t apr_err;
+ apr_file_t *f = arg;

+ /* Remove locks. */
+ apr_err = apr_file_unlock (f);
+ if (apr_err)
+ return apr_err;
+
+ /* Close the file. */
+ apr_err = apr_file_close (f);
+ if (apr_err)
+ return apr_err;
+
+ return 0;
+}
+
+
+svn_error_t *svn_io_file_lock (const char *lock_file,
+ svn_boolean_t exclusive,
+ apr_pool_t *pool)
+{
+ int locktype = APR_FLOCK_SHARED;
+ apr_file_t *lockfile_handle;
+ apr_int32_t flags;
+ apr_status_t apr_err;
+
+ if(exclusive == TRUE) {
+ locktype = APR_FLOCK_EXCLUSIVE;
+ }
+
+ flags = APR_READ;
+ if (locktype == APR_FLOCK_EXCLUSIVE)
+ flags |= APR_WRITE;
+
+ SVN_ERR (svn_io_file_open (&lockfile_handle, lock_file, flags,
+ APR_OS_DEFAULT,
+ pool));
+
+ /* Get lock on the filehandle. */
+ apr_err = apr_file_lock (lockfile_handle, locktype);
+ if (apr_err)
+ {
+ const char *lockname = "unknown";
+ if (locktype == APR_FLOCK_SHARED)
+ lockname = "shared";
+ if (locktype == APR_FLOCK_EXCLUSIVE)
+ lockname = "exclusive";
+
+ return svn_error_createf
+ (apr_err, NULL,
+ "svn_io_file_lock: %s lock on file `%s' failed",
+ lockname, lock_file);
+ }
+
+ apr_pool_cleanup_register (pool, lockfile_handle,
+ svn_io__file_clear_and_close,
+ apr_pool_cleanup_null);
+
+ return SVN_NO_ERROR;
+}
+
+
  
  /* TODO write test for these two functions, then refactor. */

Index: subversion/libsvn_repos/repos.c
===================================================================
--- subversion/libsvn_repos/repos.c (revision 7005)
+++ subversion/libsvn_repos/repos.c (working copy)
@@ -35,7 +35,6 @@
     a builtin template of this name. */
  #define DEFAULT_TEMPLATE_NAME "default"

-
  
  /* Path accessor functions. */

@@ -69,6 +68,12 @@

  const char *
+svn_repos_db_logs_lockfile (svn_repos_t *repos, apr_pool_t *pool)
+{
+ return svn_path_join (repos->lock_path, SVN_REPOS__DB_LOGS_LOCKFILE, pool);
+}
+
+const char *
  svn_repos_hook_dir (svn_repos_t *repos, apr_pool_t *pool)
  {
    return apr_pstrdup (pool, repos->hook_path);
@@ -137,67 +142,73 @@
    return err;
  }

+/* Create the DB logs lockfile. */
+static svn_error_t *
+create_db_logs_lock (svn_repos_t *repos, apr_pool_t *pool) {
+ const char *contents;
+ const char *lockfile_path;

+ lockfile_path = svn_repos_db_logs_lockfile (repos, pool);
+ contents =
+ "DB logs lock file, representing locks on the versioned filesystem logs.\n"
+ "\n"
+ "All log manipulators of the repository's\n"
+ "Berkeley DB environment take out exclusive locks on this file\n"
+ "to ensure that only one accessor manupulates the logs at the time.\n"
+ "\n"
+ "You should never have to edit or remove this file.\n";
+
+ SVN_ERR_W (svn_io_file_create (lockfile_path, contents, pool),
+ "creating db logs lock file");
+
+
+ return SVN_NO_ERROR;
+}
+
+/* Create the DB lockfile. */
  static svn_error_t *
-create_locks (svn_repos_t *repos, const char *path, apr_pool_t *pool)
+create_db_lock (svn_repos_t *repos, apr_pool_t *pool) {
+ const char *contents;
+ const char *lockfile_path;
+
+ lockfile_path = svn_repos_db_lockfile (repos, pool);
+ contents =
+ "DB lock file, representing locks on the versioned filesystem.\n"
+ "\n"
+ "All accessors -- both readers and writers -- of the repository's\n"
+ "Berkeley DB environment take out shared locks on this file, and\n"
+ "each accessor removes its lock when done. If and when the DB\n"
+ "recovery procedure is run, the recovery code takes out an\n"
+ "exclusive lock on this file, so we can be sure no one else is\n"
+ "using the DB during the recovery.\n"
+ "\n"
+ "You should never have to edit or remove this file.\n";
+
+ SVN_ERR_W (svn_io_file_create (lockfile_path, contents, pool),
+ "creating db lock file");
+ return SVN_NO_ERROR;
+}
+
+static svn_error_t *
+create_locks (svn_repos_t *repos, apr_pool_t *pool)
  {
- apr_status_t apr_err;
-
    /* Create the locks directory. */
- SVN_ERR_W (create_repos_dir (path, pool),
+ SVN_ERR_W (create_repos_dir (repos->lock_path, pool),
               "creating lock dir");

- /* Create the DB lockfile under that directory. */
- {
- apr_file_t *f = NULL;
- apr_size_t written;
- const char *contents;
- const char *lockfile_path;
-
- lockfile_path = svn_repos_db_lockfile (repos, pool);
- SVN_ERR_W (svn_io_file_open (&f, lockfile_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating lock file");
-
- contents =
- "DB lock file, representing locks on the versioned filesystem.\n"
- "\n"
- "All accessors -- both readers and writers -- of the repository's\n"
- "Berkeley DB environment take out shared locks on this file, and\n"
- "each accessor removes its lock when done. If and when the DB\n"
- "recovery procedure is run, the recovery code takes out an\n"
- "exclusive lock on this file, so we can be sure no one else is\n"
- "using the DB during the recovery.\n"
- "\n"
- "You should never have to edit or remove this file.\n";
-
- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing lock file `%s'", lockfile_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing lock file `%s'", lockfile_path);
- }
-
+ SVN_ERR(create_db_lock(repos, pool));
+ SVN_ERR(create_db_logs_lock(repos, pool));
    return SVN_NO_ERROR;
  }

  static svn_error_t *
-create_hooks (svn_repos_t *repos, const char *path, apr_pool_t *pool)
+create_hooks (svn_repos_t *repos, apr_pool_t *pool)
  {
    const char *this_path, *contents;
- apr_status_t apr_err;
- apr_file_t *f;
- apr_size_t written;

    /* Create the hook directory. */
- SVN_ERR_W (create_repos_dir (path, pool),
+ SVN_ERR_W (create_repos_dir (repos->hook_path, pool),
               "creating hook directory");

    /*** Write a default template for each standard hook file. */
@@ -208,12 +219,6 @@
                                svn_repos_start_commit_hook (repos, pool),
                                SVN_REPOS__HOOK_DESC_EXT);

- SVN_ERR_W (svn_io_file_open (&f, this_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating hook file");
-
      contents =
        "#!/bin/sh"
        APR_EOL_STR
@@ -298,15 +303,8 @@
        "exit 0"
        APR_EOL_STR;

- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing hook file `%s'", this_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing hook file `%s'", this_path);
+ SVN_ERR_W (svn_io_file_create (this_path, contents, pool),
+ "creating start-commit hook");
    } /* end start-commit hook */

    /* Pre-commit hook. */
@@ -315,12 +313,6 @@
                                svn_repos_pre_commit_hook (repos, pool),
                                SVN_REPOS__HOOK_DESC_EXT);

- SVN_ERR_W (svn_io_file_open (&f, this_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating hook file");
-
      contents =
        "#!/bin/sh"
        APR_EOL_STR
@@ -433,15 +425,8 @@
        "exit 0"
        APR_EOL_STR;

- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing hook file `%s'", this_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing hook file `%s'", this_path);
+ SVN_ERR_W (svn_io_file_create (this_path, contents, pool),
+ "creating pre-commit hook");
    } /* end pre-commit hook */

@@ -451,12 +436,6 @@
                                svn_repos_pre_revprop_change_hook (repos, pool),
                                SVN_REPOS__HOOK_DESC_EXT);

- SVN_ERR_W (svn_io_file_open (&f, this_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating hook file");
-
      contents =
        "#!/bin/sh"
        APR_EOL_STR
@@ -562,15 +541,8 @@
        "exit 1"
        APR_EOL_STR;

- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing hook file `%s'", this_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing hook file `%s'", this_path);
+ SVN_ERR_W (svn_io_file_create (this_path, contents, pool),
+ "creating pre-revprop-change hook");
    } /* end pre-revprop-change hook */

@@ -580,12 +552,6 @@
                                svn_repos_post_commit_hook (repos, pool),
                                SVN_REPOS__HOOK_DESC_EXT);

- SVN_ERR_W (svn_io_file_open (&f, this_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating hook file");
-
      contents =
        "#!/bin/sh"
        APR_EOL_STR
@@ -665,15 +631,8 @@
        "log-commit.py --repository \"$REPOS\" --revision \"$REV\""
        APR_EOL_STR;

- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing hook file `%s'", this_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing hook file `%s'", this_path);
+ SVN_ERR_W (svn_io_file_create (this_path, contents, pool),
+ "creating post-commit hook");
    } /* end post-commit hook */

@@ -683,12 +642,6 @@
                                svn_repos_post_revprop_change_hook (repos, pool),
                                SVN_REPOS__HOOK_DESC_EXT);

- SVN_ERR_W (svn_io_file_open (&f, this_path,
- (APR_WRITE | APR_CREATE | APR_EXCL),
- APR_OS_DEFAULT,
- pool),
- "creating hook file");
-
      contents =
        "#!/bin/sh"
        APR_EOL_STR
@@ -776,15 +729,8 @@
        "propchange-email.pl \"$REPOS\" \"$REV\" \"$USER\" \"$PROPNAME\"
watchers@example.org"
        APR_EOL_STR;

- apr_err = apr_file_write_full (f, contents, strlen (contents), &written);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "writing hook file `%s'", this_path);
-
- apr_err = apr_file_close (f);
- if (apr_err)
- return svn_error_createf (apr_err, NULL,
- "closing hook file `%s'", this_path);
+ SVN_ERR_W (svn_io_file_create (this_path, contents, pool),
+ "creating post-revprop-change hook");
    } /* end post-revprop-change hook */

    return SVN_NO_ERROR;
@@ -856,10 +802,10 @@
               "creating DAV sandbox dir");

    /* Create the lock directory. */
- SVN_ERR (create_locks (repos, repos->lock_path, pool));
+ SVN_ERR (create_locks (repos, pool));

    /* Create the hooks directory. */
- SVN_ERR (create_hooks (repos, repos->hook_path, pool));
+ SVN_ERR (create_hooks (repos, pool));

    /* Write the top-level README file. */
    {
@@ -1245,3 +1191,152 @@

    return SVN_NO_ERROR;
  }
+
+struct hotcopy_ctx_t {
+ const char *dest; /* target location to construct */
+ unsigned int src_len; /* len of the source path*/
+};
+
+
+static svn_error_t *hotcopy_structure (void *baton,
+ const char *path,
+ const apr_finfo_t *finfo,
+ apr_pool_t *pool)
+{
+ const struct hotcopy_ctx_t *ctx = ((struct hotcopy_ctx_t *) baton);
+ const char *sub_path;
+ const char *target;
+ svn_boolean_t fs_dir = FALSE;
+
+ if(strlen(path) == ctx->src_len)
+ {
+ sub_path = &"";
+ }
+ else
+ {
+ sub_path = &path[ctx->src_len+1];
+
+ /* Check if we are inside db directory and if so skip it */
+ if(svn_path_compare_paths(
+ svn_path_get_longest_ancestor(SVN_REPOS__DB_DIR, sub_path, pool),
+ SVN_REPOS__DB_DIR) == 0) {
+ return SVN_NO_ERROR;
+ }
+
+
+ if(svn_path_compare_paths(
+ svn_path_get_longest_ancestor(SVN_REPOS__LOCK_DIR, sub_path, pool),
+ SVN_REPOS__LOCK_DIR) == 0) {
+ return SVN_NO_ERROR;
+ }
+
+ }
+
+ target = svn_path_join(ctx->dest, sub_path, pool);
+
+ if (finfo->filetype == APR_DIR)
+ {
+ SVN_ERR (create_repos_dir (target, pool));
+ }
+ else if(finfo->filetype == APR_REG)
+ {
+
+ SVN_ERR(svn_io_copy_file(path, target, TRUE, pool));
+ }
+
+ return SVN_NO_ERROR;
+}
+
+
+
+static svn_error_t *
+lock_db_logs_file (svn_repos_t *repos,
+ svn_boolean_t exclusive,
+ apr_pool_t *pool)
+{
+ const char * lock_file = svn_repos_db_logs_lockfile(repos, pool);
+ svn_node_kind_t kind;
+ SVN_ERR(svn_io_check_path(lock_file, &kind, pool));
+
+ /* Try to create a lock file if it is missing.
+ Ignore creation errors just in case file got created by another process
+ while we were checking. */
+ if(kind == svn_node_none) {
+ svn_error_t * err = create_db_logs_lock(repos, pool);
+ if(err) {
+ svn_error_clear(err);
+ }
+ }
+
+ SVN_ERR(svn_io_file_lock(lock_file, exclusive, pool));
+
+ return SVN_NO_ERROR;
+}
+
+
+/* Make a copy of a repository with hot backup of fs. */
+svn_error_t *
+svn_repos_hotcopy (const char *src_path,
+ const char *dst_path,
+ svn_boolean_t archive_logs,
+ apr_pool_t *pool)
+{
+ svn_repos_t *src_repos = NULL;
+ svn_repos_t *dst_repos = NULL;
+ struct hotcopy_ctx_t hotcopy_context;
+
+ /* Try to open original repository */
+ SVN_ERR (get_repos (&src_repos, src_path,
+ APR_FLOCK_SHARED,
+ FALSE, /* don't try to open the db yet. */
+ pool));
+
+ /* If we are going to archive logs, then get an exclusive lock on
+ db-logs.lock, to ensure that no one else will work with logs.
+
+ If we are just copying, then get a shared lock to ensure that
+ no one else will archive logs while we copying them */
+
+ SVN_ERR (lock_db_logs_file(src_repos, archive_logs, pool));
+
+ /* Copy the repository to a new path, with exception of
+ specially handled directories */
+
+ hotcopy_context.dest = dst_path;
+ hotcopy_context.src_len = strlen(src_path);
+ SVN_ERR (svn_io_dir_walk (src_path,
+ 0,
+ hotcopy_structure,
+ &hotcopy_context,
+ pool));
+
+ /* Prepare dst_repos object so that we may create locks,
+ so that we may open repository */
+
+ /* Allocate a repository object. */
+ dst_repos = apr_pcalloc (pool, sizeof (*dst_repos));
+
+ /* Initialize the repository paths. */
+ init_repos_dirs (dst_repos, dst_path, pool);
+
+ /* Create the lock directory. */
+ SVN_ERR (create_locks (dst_repos, pool));
+
+ SVN_ERR (create_repos_dir(dst_repos->db_path, pool));
+ /* Reopen repository the right way. Since above is just work around
+ util we can create locks without an open repository */
+
+ /* Exclusevly lock the new repository.
+ No one should be accessing it at the moment */
+ SVN_ERR (get_repos (&dst_repos, dst_path,
+ APR_FLOCK_EXCLUSIVE,
+ FALSE, /* don't try to open the db yet. */
+ pool));
+
+
+ SVN_ERR ( svn_fs_hotcopy_berkeley(src_repos->db_path, dst_repos->db_path,
+ archive_logs, pool));
+
+ return SVN_NO_ERROR;
+}
+
Index: subversion/libsvn_repos/repos.h
===================================================================
--- subversion/libsvn_repos/repos.h (revision 7005)
+++ subversion/libsvn_repos/repos.h (working copy)
@@ -46,6 +46,7 @@

  /* Things for which we keep lockfiles. */
  #define SVN_REPOS__DB_LOCKFILE "db.lock" /* Our Berkeley lockfile. */
+#define SVN_REPOS__DB_LOGS_LOCKFILE "db-logs.lock" /* BDB logs lockfile. */

  /* In the repository hooks directory, look for these files. */
  #define SVN_REPOS__HOOK_START_COMMIT "start-commit"
Index: tools/backup/hot-backup.py.in
===================================================================
--- tools/backup/hot-backup.py.in (revision 7005)
+++ tools/backup/hot-backup.py.in (working copy)
@@ -30,12 +30,9 @@
  # Path to svnlook utility
  svnlook = "@SVN_BINDIR@/svnlook"

-# Path to db_archive program
-db_archive = "/usr/local/BerkeleyDB.4.0/bin/db_archive"
+# Path to svnadmin program with lsbdlogs subcommand
+svnadmin = "@SVN_BINDIR@/svnadmin"

-# Path to db_recover progrem
-db_recover = "/usr/local/BerkeleyDB.4.0/bin/db_recover"
-
  # Number of backups to keep around (0 for "keep them all")
  num_backups = 64

@@ -101,7 +98,7 @@
  print "Youngest revision is", youngest

-### Step 2: copy the whole repository structure.
+### Step 2: Find next available backup path

  backup_subdir = os.path.join(backup_dir, repo + "-" + youngest)

@@ -124,93 +121,20 @@
    else:
      backup_subdir = os.path.join(backup_dir, repo + "-" + youngest + "-1")

-print "Backing up repository to '" + backup_subdir + "'..."
-shutil.copytree(repo_dir, backup_subdir)
-print "Done."
-
-
-### Step 3: re-copy the Berkeley logfiles. They must *always* be
+### Step 3: Ask subversion to make a hot copy of a repository.
  ### copied last.

-infile, outfile, errfile = os.popen3(db_archive + " -l -h "
- + os.path.join(repo_dir, "db"))
-stdout_lines = outfile.readlines()
-stderr_lines = errfile.readlines()
-outfile.close()
-infile.close()
-errfile.close()
+print "Backing up repository to '" + backup_subdir + "'..."
+err_code = os.spawnl(os.P_WAIT, svnadmin, " hotcopy ", repo_dir,
+ backup_subdir, "--archive-logs")
+if(err_code != 0):
+ print "Unable to backup the repository."
+ sys.exit(err_code)
+else:
+ print "Done."

-print "Re-copying logfiles:"

-for item in stdout_lines:
- logfile = string.strip(item)
- src = os.path.join(repo_dir, "db", logfile)
- dst = os.path.join(backup_subdir, "db", logfile)
- print " Re-copying logfile '" + logfile + "'..."
- shutil.copy(src, dst)
-
-print "Backup completed."
-
-
-### Step 4: put the archived database in a consistent state and remove
-### the shared-memory environment files.
-
-infile, outfile, errfile = os.popen3(db_recover + " -h "
- + os.path.join(backup_subdir, "db"))
-stdout_lines = outfile.readlines()
-stderr_lines = errfile.readlines()
-outfile.close()
-infile.close()
-errfile.close()
-
-print "Running db_recover on the archived database:"
-map(sys.stdout.write, stdout_lines)
-map(sys.stdout.write, stderr_lines)
-
-print "Done."
-
-
-### Step 5: look for a write `lock' file in the backup area, else make one.
-
-lockpath = os.path.join(backup_dir, repo + 'lock')
-if os.path.exists(lockpath):
- print "Cannot cleanup logs: lockfile already exists in", backup_dir
- sys.exit(0)
-
-print "Writing lock for logfile cleanup..."
-fp = open(lockpath, 'a') # open in (a)ppend mode
-fp.write("cleaning logfiles for repository " + repo_dir)
-fp.close()
-
-
-### Step 6: ask db_archive which of the live logfiles can be
-### expunged, and remove them.
-
-infile, outfile, errfile = os.popen3(db_archive + " -a -h "
- + os.path.join(repo_dir, "db"))
-stdout_lines = outfile.readlines()
-stderr_lines = errfile.readlines()
-outfile.close()
-infile.close()
-errfile.close()
-
-print "Cleaning obsolete logfiles:"
-
-for item in stdout_lines:
- logfile = string.strip(item)
- print " Deleting '", logfile, "'..."
- os.unlink(logfile)
-
-print "Done."
-
-
-### Step 7: remove the write lock.
-
-os.unlink(lockpath)
-print "Lock removed. Cleanup complete."
-
-
-### Step 8: finally, remove all repository backups other than the last
+### Step 4: finally, remove all repository backups other than the last
  ### NUM_BACKUPS.

  if num_backups > 0:

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Mon Sep 8 23:30:30 2003

This is an archived mail posted to the Subversion Dev mailing list.

This site is subject to the Apache Privacy Policy and the Apache Public Forum Archive Policy.