[PATCH]: Automatic log file removal

Gee, it looked formatted okay in emacs (i used svn-dev.el and all)
I blame something else.

Anyway, this patch adds a thread that auto-removes log files every minute.
This is the right speed for my use, which currently consists of lots of
large cvs repository conversions (which are repeated commits, rather than
imports).

It's more the number of revisions than the size of the repository. In
this case, it's a 16 meg cvs repository, but it's got 2056 seperate
"revisions" in it.
This generates 99 log files (989 meg):

989MB 754KB 745B Log bytes written (772841 bytes).
0 Log bytes written since last checkpoint.
40500 Total log file writes.
899 Total log file write due to overflow.
40021 Total log file flushes.
99 Current log file number.
10388385 Current log file offset.
99 On-disk log file number.
10388385 On-disk log file offset.
1 Max commits in a log flush.
1 Min commits in a log flush.
36435 Number of log flushes containing a transaction commit.

The example i took it from in the berkeley docs had the sleep time at 5
minutes, just for reference.

This patch also ups the log file buffer from 32k to 256k to improve
throughput. This was based on statistics showing it was impeding
performance somewhat. With it at 256k, we don't overflow before a commit
anywhere near as often (the above is from a 256k log file buffer, a 32k
would show that we were consistently filling the log file buffer multiple
times per commit).

The removal thread will exit itself when the fs->env variable it was
passed the address to goes NULL, or when it can't remove a log file, or
the log_archive command returns an error.

The only thing changed from the version in the manual is the timeout, the
formatting, and the check to see when the fs->env we were passed goes
NULL.

I've assumed the berkeley guys know what they are doing when they wrote
the example.
:)

I think this enhancement might have been on the bite-sized task list, i'm
not sure.

 Index: ./fs.c
===================================================================
--- ./fs.c
+++ ./fs.c Mon Feb 4 11:06:08 2002
@@ -38,6 +38,42 @@
 #include "svn_private_config.h"
 
 
+static void *
+logfile_thread (void *arg)
+{
+ DB_ENV **argenv;
+ DB_ENV *dbenv;
+ int ret;
+ char **begin, **list;
+ argenv = arg;
+ dbenv = *argenv;
+
+ /* Check once every minute. */
+ for (;; sleep(60))
+ {
+ if (!*argenv)
+ pthread_exit(NULL);
+ /* Get the list of log files. */
+ if ((ret = dbenv->log_archive(dbenv, &list, DB_ARCH_ABS)) != 0)
+ {
+ dbenv->err(dbenv, ret, "DB_ENV->log_archive");
+ pthread_exit (NULL);
+ }
+
+ /* Remove the log files. */
+ if (list != NULL)
+ {
+ for (begin = list; *list != NULL; ++list)
+ if ((ret = remove(*list)) != 0)
+ {
+ dbenv->err(dbenv, ret, "remove %s", *list);
+ pthread_exit (NULL);
+ }
+ free (begin);
+ }
+ }
+}
+
 /* Checking for return values, and reporting errors. */
 
 
@@ -355,7 +393,11 @@
      from those participating in the deadlock. */
   SVN_ERR (DB_WRAP (fs, "setting deadlock detection policy",
                     fs->env->set_lk_detect (fs->env, DB_LOCK_RANDOM)));
-
+ /* For our purposes, 32k is too small of a log file buffer.
+ Kick it up to 256k to increase throughput. */
+ SVN_ERR (DB_WRAP (fs, "setting log file buffer size",
+ fs->env->set_lg_bsize (fs->env, 256 * 1024)));
+
   return SVN_NO_ERROR;
 }
 
@@ -441,8 +483,9 @@
 svn_error_t *
 svn_fs_open_berkeley (svn_fs_t *fs, const char *path)
 {
+ pthread_t ptid;
   svn_error_t *svn_err;
-
+ int ret;
   SVN_ERR (check_already_open (fs));
 
   /* Initialize paths. */
@@ -459,8 +502,12 @@
                                      | DB_INIT_MPOOL
                                      | DB_INIT_TXN),
                                     0666));
+ if (svn_err) goto error;
+ ret = pthread_create (&ptid, NULL, logfile_thread, (void *)(&fs->env));
+ if (ret != 0)
+ svn_err = svn_error_createf (SVN_ERR_BERKELEY_DB, 0, 0, fs->pool, "spawning log file removal thread");
   if (svn_err) goto error;
-
+
   /* Open the various databases. */
   svn_err = DB_WRAP (fs, "opening `nodes' table",
                      svn_fs__open_nodes_table (&fs->nodes, fs->env, 0));

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Sat Oct 21 14:37:04 2006