[svn.haxx.se] · SVN Dev · SVN Users · SVN Org · TSVN Dev · TSVN Users · Subclipse Dev · Subclipse Users · this month's index

Re: Logging: home-grown mechanism or syslog/event-log?

From: Michael Sinz <michael.sinz_at_gmail.com>
Date: 2005-07-20 14:28:03 CEST

On 7/20/05, John Peacock <jpeacock@rowman.com> wrote:
> Greg Hudson wrote:
> > One possible criticism is that using syslog on Unix unconditionally
> > would not make it easy to interoperate with Dan Bernstein's
> > daemontools as John Szakmeister (I think; both mail archives are down
> > right now) desired.
>
> Yeah, that was me and as long as --foreground redirects all logging to STDOUT,
> I'll be happy.

That is actually usually rather simple with syslog - I don't know how much
Windows application event logging supports such behaviors since it is
not specifically
a human readable format. But with syslog you just openlog with the LOG_PERROR
option flag and you get your log output sent to STDERR in addition to syslogd.

> There was a time when syslogd was not robust under heavy load and log lines
> would just be dropped on the floor. I don't know if that is true anymore, but
> it is something to consider. Given the general rise in performance of CPU's and
> hard drives, I suspect that isn't worth getting too worked up about, but some
> sites view logs as vital information that must be preserved at all costs.

Depending on the syslogd in use, you can get anywhere from the older behavior
(albeit much higher network bandwidth means much less dropping of UDP packets)
up to full TCP socket based, encrypted, reliable logging.

> My personal dislike of syslog stems from a couple of things:
>
> 1) I have to run some other program to rotate logs; sure, many/most Linux
> distros include something to do this, but it is so much easier to use multilog
> and get fully atomic log rolling and purging of old logs. I don't have to keep
> log files around for audit purposes forever, so that is a good thing for me; if
> you needed to keep files around longer, it is trivial to set up a !processor
> directive within multilog to do something other than just rotate.

Log rotation is relatively atomic. You do all of the rotation
operations you want
(rename current log files, create new ones in the old name) and when ready
you signal syslogd. All logs are now flipped to the new log file and those that
you did not rename/create new will not be flipped. So you get atomic rotation
across multiple log files and log sources.

> 2) using syslogd would typically require that the person setting up svnserve
> (for example) add system configuration directives for each instance (in the form
> of a facility line). In other words, using syslog requires external
> configuration changes to be fully functional. OK, I'm fudging a little in that
> if you don't make those changes, syslogd will just mix your log lines in with
> the unwashed masses of other log lines, but I view that as completely unworkable.

Depending on the needs, having at least one log file that is "the
unwashed masses"
is actually very good as it lets you do better tracking of what really happened.
You will see all of the "issues" in one place such that a
multi-subsystem failure
or attack can be spotted much more easily.

Now, for auditing of actions for a specific service you may wish to have things
split out. Old syslogd did not do that well but many modern syslogd can do some
amazing things here.
 
> 3) syslog uses a "threat" model for log levels, as opposed to a "functional"
> model. By this I mean that typically you normally have to set the log level
> high enough to capture the information you want and exclude the mass of detail
> which isn't important in normal operation. Unfortunately, since they set the
> LOG_XXX levels assuming that you were interested in only how _bad_ things are,
> not considering that sometimes you might want to log good things.

Yes, the threat model is the traditional first line of "what gets
saved" type of thing.
But that does not mean that you have to use that consistantly across
all services
or, in many modern syslogd setups, need to use that at all.

I use threat level to fork log entries into multiple locations - the
highest levels are
logged not just to the standard file but to a secondary server and sometimes to
a console for instant operator notification.

Many modern syslogd setups provide full regexp log message support, such that
actual data within the message can be used to select actions.

BTW - openlog() is a very low overhead operation - it just sets some values such
that the "ident", "facility", and options are set up. Traditionally,
closelog() was never
really called. openlog() is not needed but you may wish to specify
the ident, options,
and facility rather than just getting the application defaults (yes,
facility is old-school
in that it has a list of system actions that are, in some cases, just outdated.)

> The reason I dislike this mode of operations is that successful operations would
> typically be logged at LOG_NOTICE, but that also means that you have to see all
> of the LOG_WARN entries, which are often harmless warnings that the programmer
> didn't know what else to do with. There is a very coarse granularity caused by
> the way the log levels have been ordered, yet (as Brane pointed out) syslog
> levels are too numerous to easily keep track of. I suspect this is one reason
> why Apache went with a split between error and access logs. Again, I'm fudging
> a bit since you can add lines to /etc/syslogd.conf to split logs into multiple
> files, but the same objections as #2 apply here.

That is not true - well, depending on syslogd I guess - but the ones I have used
let you turn on individual levels, albeit it would take two rules, one
to strip the higher
level LOG_WARN and then another to include the lower level LOG_NOTICE.
But again, if you need fancier logging, look at some of the fancier
syslogd implementations.

You may wish to look at things like the FreeBSD syslogd - which has some nice
additional features but is still rather old, or the syslog-ng, which
adds some really
fancy stuff. Then there are the syslogs that are built to log into
databases for more
complex query operations. syslog+mysql is rather simple - others are much more
complex.

Which all goes to show another reason using syslog is nice - it is a
well known structure
into which you can plug in external solutions to add the features you want. A
home-grown logging system means that you are tied to the files/system that the
logging system provides. (I want my single, redundant, secure log
server for the
multiple machines I have - thus only one place to look and only one
place to deal with
the log file sizes, rotation, etc. And it can be secured such that
erasing log entries
would require breaking into that machine or machines)

-- 
Michael Sinz               Technology and Engineering Director/Consultant
"Starting Startups"                          mailto:Michael.Sinz@sinz.org
My place on the web                      http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Received on Wed Jul 20 23:06:37 2005

This is an archived mail posted to the Subversion Dev mailing list.