Skip to content

P
Postgres FD Implementation
Commit fd801f4f authored by Andrew Dunstan's avatar Andrew Dunstan
Browse files
Options

Provide for logfiles in machine readable CSV format. In consequence, rename 

redirect_stderr to logging_collector.
Original patch from Arul Shaji, subsequently modified by Greg Smith, and then
heavily modified by me.
parent e53a5487
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 663 additions and 127 deletions
doc/src/sgml/config.sgml
View file @ fd801f4f
<!-- $PostgreSQL: pgsql/doc/src/sgml/config.sgml,v 1.137 2007/08/13 19:27:11 tgl Exp $ -->
<!-- $PostgreSQL: pgsql/doc/src/sgml/config.sgml,v 1.138 2007/08/19 01:41:23 adunstan Exp $ -->
<chapter Id="runtime-config">
<title>Server Configuration</title>
......@@ -2261,7 +2261,7 @@ SELECT * FROM parent WHERE key = 2400;
<para>
<productname>PostgreSQL</productname> supports several methods
for logging server messages, including
<systemitem>stderr</systemitem> and
<systemitem>stderr</systemitem>, <systemitem>csvlog</systemitem> and
<systemitem>syslog</systemitem>. On Windows,
<systemitem>eventlog</systemitem> is also supported. Set this
parameter to a list of desired log destinations separated by
......@@ -2270,17 +2270,24 @@ SELECT * FROM parent WHERE key = 2400;
This parameter can only be set in the <filename>postgresql.conf</>
file or on the server command line.
</para>
<para> If <varname>log_destination</> is set to <systemitem>csvlog</systemitem>,
the log is output as comma seperated values. The format is:
timestamp with milliseconds, username, database name, session id, host:port number,
process id, per process line number, command tag, session start time, transaction id,
error severity, SQL state code, statement/error message.
</para>
</listitem>
</varlistentry>
<varlistentry id="guc-redirect-stderr" xreflabel="redirect_stderr">
<term><varname>redirect_stderr</varname> (<type>boolean</type>)</term>
<varlistentry id="guc-logging-collector" xreflabel="logging_collector">
<term><varname>logging_collector</varname> (<type>boolean</type>)</term>
<indexterm>
<primary><varname>redirect_stderr</> configuration parameter</primary>
<primary><varname>logging_collector</> configuration parameter</primary>
</indexterm>
<listitem>
<para>
This parameter allows messages sent to <application>stderr</> to be
This parameter allows messages sent to <application>stderr</>,
and CSV logs, to be
captured and redirected into log files.
This method, in combination with logging to <application>stderr</>,
is often more useful than
......@@ -2288,6 +2295,8 @@ SELECT * FROM parent WHERE key = 2400;
might not appear in <application>syslog</> output (a common example
is dynamic-linker failure messages).
This parameter can only be set at server start.
<varname>start_log_collector</varname> must be enabled to generate
CSV logs.
</para>
</listitem>
</varlistentry>
......@@ -2299,8 +2308,8 @@ SELECT * FROM parent WHERE key = 2400;
</indexterm>
<listitem>
<para>
When <varname>redirect_stderr</> is enabled, this parameter
determines the directory in which log files will be created.
When <varname>logging_collector</> is enabled,
this parameter determines the directory in which log files will be created.
It can be specified as an absolute path, or relative to the
cluster data directory.
This parameter can only be set in the <filename>postgresql.conf</>
......@@ -2316,8 +2325,8 @@ SELECT * FROM parent WHERE key = 2400;
</indexterm>
<listitem>
<para>
When <varname>redirect_stderr</varname> is enabled, this parameter
sets the file names of the created log files. The value
When <varname>logging_collector</varname> is enabled,
this parameter sets the file names of the created log files. The value
is treated as a <systemitem>strftime</systemitem> pattern,
so <literal>%</literal>-escapes can be used to specify time-varying
file names. (Note that if there are
......@@ -2332,6 +2341,14 @@ SELECT * FROM parent WHERE key = 2400;
This parameter can only be set in the <filename>postgresql.conf</>
file or on the server command line.
</para>
<para>
If <varname>log_destination</> is set to <systemitem>csvlog</>,
<literal>.csv</> will be appended to the timestamped
<varname>log_filename</> to create the final log file name.
(If log_filename ends in <literal>.log</>, the suffix is overwritten.)
In the case of the example above, the
file name will be <literal>server_log.1093827753.csv</literal>
</para>
</listitem>
</varlistentry>
......@@ -2342,8 +2359,8 @@ SELECT * FROM parent WHERE key = 2400;
</indexterm>
<listitem>
<para>
When <varname>redirect_stderr</varname> is enabled, this parameter
determines the maximum lifetime of an individual log file.
When <varname>logging_collector</varname> is enabled,
this parameter determines the maximum lifetime of an individual log file.
After this many minutes have elapsed, a new log file will
be created. Set to zero to disable time-based creation of
new log files.
......@@ -2360,8 +2377,8 @@ SELECT * FROM parent WHERE key = 2400;
</indexterm>
<listitem>
<para>
When <varname>redirect_stderr</varname> is enabled, this parameter
determines the maximum size of an individual log file.
When <varname>logging_collector</varname> is enabled,
this parameter determines the maximum size of an individual log file.
After this many kilobytes have been emitted into a log file,
a new log file will be created. Set to zero to disable size-based
creation of new log files.
......@@ -2378,8 +2395,8 @@ SELECT * FROM parent WHERE key = 2400;
</indexterm>
<listitem>
<para>
When <varname>redirect_stderr</varname> is enabled, this parameter will cause
<productname>PostgreSQL</productname> to truncate (overwrite),
When <varname>logging_collector</varname> is enabled,
this parameter will cause <productname>PostgreSQL</productname> to truncate (overwrite),
rather than append to, any existing log file of the same name.
However, truncation will occur only when a new file is being opened
due to time-based rotation, not during server startup or size-based
......@@ -2599,7 +2616,7 @@ SELECT * FROM parent WHERE key = 2400;
The server's standard output and standard error are redirected
to <literal>/dev/null</>, so any messages sent to them will be lost.
Unless <application>syslog</> logging is selected or
<varname>redirect_stderr</> is enabled, using this parameter
<varname>start_log_collector</> is enabled, using this parameter
is discouraged because it makes it impossible to see error messages.
This parameter can only be set at server start.
</para>
......@@ -3066,6 +3083,94 @@ SELECT * FROM parent WHERE key = 2400;
</varlistentry>
</variablelist>
</sect2>
<sect2 id="runtime-config-logging-csvlog">
<title>Using the csvlog</title>
<para>
Including <literal>csvlog</> in the <varname>log_destination</> list
provides a convenient way to import log files into a database table.
Here is a sample table definition for storing csvlog output:
</para>
<programlisting>
CREATE TABLE postgres_log
(
log_time timestamp,
username text,
database_name text,
sessionid text not null,
connection_from text,
process_id text,
process_line_num int not null,
command_tag text,
session_start_time timestamp,
transaction_id int,
error_severity text,
sql_state_code text,
statement text,
PRIMARY KEY (sessionid, process_line_num)
);
</programlisting>
<para>
In order to import into this table, use the COPY FROM command:
</para>
<programlisting>
COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</programlisting>
<para>
There are a few things you need to import csvlog files easily and
automatically:
<orderedlist>
<listitem>
<para>
Use a consistant, predictable naming scheme for your log files
with <varname>log_filename</varname>. This lets you predict what
the file name will be when it is ready to be imported.
guess what
the file name will be and know when an individual log file is
complete and therefore ready to be imported.
</para>
</listitem>
<listitem>
<para>
Set <varname>log_rotation_size</varname> to 0 to disable
size-based log rotation, as it makes the log filename difficult
to predict.
</para>
</listitem>
<listitem>
<para>
Set <varname>log_truncate_on_rotate</varname> = on so that old
log data isn't mixed with the new in the same file.
</para>
</listitem>
<listitem>
<para>
The example above includes a useful primary key on the log
file data, which will protect against accidentally importing
the same information twice. The COPY command commits all of
the data it imports at one time, and any single error will
cause the entire import to fail.
If you import a partial log file and later import the file again
when it is complete, the primary key violation will cause the
import to fail. Wait until the log is complete and closed before
import. This will also protect against accidently importing a
partial line that hasn't been completely written, which would
also cause the COPY to fail.
</para>
</listitem>
</orderedlist>
</para>
</sect2>
</sect1>
......
doc/src/sgml/func.sgml
View file @ fd801f4f
<!-- $PostgreSQL: pgsql/doc/src/sgml/func.sgml,v 1.385 2007/08/13 01:18:47 tgl Exp $ -->
<!-- $PostgreSQL: pgsql/doc/src/sgml/func.sgml,v 1.386 2007/08/19 01:41:23 adunstan Exp $ -->
<chapter id="functions">
<title>Functions and Operators</title>
......@@ -11254,9 +11254,9 @@ SELECT set_config('log_statement_stats', 'off', false);
<para>
<function>pg_rotate_logfile</> signals the log-file manager to switch
to a new output file immediately. This works only when
<varname>redirect_stderr</> is used for logging, since otherwise there
is no log-file manager subprocess.
to a new output file immediately. This works only when the built-in
log collector if running, since otherwise there is no log-file manager
subprocess.
</para>
<indexterm zone="functions-admin">
......
doc/src/sgml/maintenance.sgml
View file @ fd801f4f
<!-- $PostgreSQL: pgsql/doc/src/sgml/maintenance.sgml,v 1.77 2007/07/23 17:22:00 alvherre Exp $ -->
<!-- $PostgreSQL: pgsql/doc/src/sgml/maintenance.sgml,v 1.78 2007/08/19 01:41:24 adunstan Exp $ -->
<chapter id="maintenance">
<title>Routine Database Maintenance Tasks</title>
......@@ -691,10 +691,11 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
A better approach is to send the server's
<systemitem>stderr</> output to some type of log rotation program.
There is a built-in log rotation program, which you can use by
setting the configuration parameter <literal>redirect_stderr</> to
setting the configuration parameter <literal>logging_collector</> to
<literal>true</> in <filename>postgresql.conf</>. The control
parameters for this program are described in <xref
linkend="runtime-config-logging-where">.
linkend="runtime-config-logging-where">. You can also use this approach
to capture the log data in machine readable CSV format.
</para>
<para>
......
src/backend/postmaster/postmaster.c
View file @ fd801f4f
......@@ -37,7 +37,7 @@
*
*
* IDENTIFICATION
* $PostgreSQL: pgsql/src/backend/postmaster/postmaster.c,v 1.540 2007/08/09 01:18:43 tgl Exp $
* $PostgreSQL: pgsql/src/backend/postmaster/postmaster.c,v 1.541 2007/08/19 01:41:24 adunstan Exp $
*
* NOTES
*
......@@ -1282,8 +1282,8 @@ ServerLoop(void)
}
}
/* If we have lost the system logger, try to start a new one */
if (SysLoggerPID == 0 && Redirect_stderr)
/* If we have lost the log collector, try to start a new one */
if (SysLoggerPID == 0 && Logging_collector)
SysLoggerPID = SysLogger_Start();
/*
......
src/backend/postmaster/syslogger.c
View file @ fd801f4f
This diff is collapsed.
src/backend/utils/adt/misc.c
View file @ fd801f4f
......@@ -8,7 +8,7 @@
*
*
* IDENTIFICATION
* $PostgreSQL: pgsql/src/backend/utils/adt/misc.c,v 1.56 2007/01/05 22:19:41 momjian Exp $
* $PostgreSQL: pgsql/src/backend/utils/adt/misc.c,v 1.57 2007/08/19 01:41:25 adunstan Exp $
*
*-------------------------------------------------------------------------
*/
......@@ -144,10 +144,10 @@ pg_rotate_logfile(PG_FUNCTION_ARGS)
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
(errmsg("must be superuser to rotate log files"))));
if (!Redirect_stderr)
if (!Logging_collector)
{
ereport(WARNING,
(errmsg("rotation not possible because log redirection not active")));
(errmsg("rotation not possible because log collection not active")));
PG_RETURN_BOOL(false);
}
......
src/backend/utils/error/elog.c
View file @ fd801f4f
This diff is collapsed.
src/backend/utils/misc/guc.c
View file @ fd801f4f
......@@ -10,7 +10,7 @@
* Written by Peter Eisentraut <peter_e@gmx.net>.
*
* IDENTIFICATION
* $PostgreSQL: pgsql/src/backend/utils/misc/guc.c,v 1.412 2007/08/13 19:27:11 tgl Exp $
* $PostgreSQL: pgsql/src/backend/utils/misc/guc.c,v 1.413 2007/08/19 01:41:25 adunstan Exp $
*
*--------------------------------------------------------------------
*/
......@@ -968,11 +968,11 @@ static struct config_bool ConfigureNamesBool[] =
false, NULL, NULL
},
{
{"redirect_stderr", PGC_POSTMASTER, LOGGING_WHERE,
gettext_noop("Start a subprocess to capture stderr output into log files."),
{"logging_collector", PGC_POSTMASTER, LOGGING_WHERE,
gettext_noop("Start a subprocess to capture stderr output and/or csvlogs into log files."),
NULL
},
&Redirect_stderr,
&Logging_collector,
false, NULL, NULL
},
{
......@@ -2241,7 +2241,7 @@ static struct config_string ConfigureNamesString[] =
{"log_destination", PGC_SIGHUP, LOGGING_WHERE,
gettext_noop("Sets the destination for server log output."),
gettext_noop("Valid values are combinations of \"stderr\", \"syslog\", "
"and \"eventlog\", depending on the platform."),
" \"csvlog\" and \"eventlog\", depending on the platform."),
GUC_LIST_INPUT
},
&log_destination_string,
......@@ -6313,6 +6313,8 @@ assign_log_destination(const char *value, bool doit, GucSource source)
if (pg_strcasecmp(tok, "stderr") == 0)
newlogdest |= LOG_DESTINATION_STDERR;
else if (pg_strcasecmp(tok, "csvlog") == 0)
newlogdest |= LOG_DESTINATION_CSVLOG;
#ifdef HAVE_SYSLOG
else if (pg_strcasecmp(tok, "syslog") == 0)
newlogdest |= LOG_DESTINATION_SYSLOG;
......
src/backend/utils/misc/postgresql.conf.sample
View file @ fd801f4f
......@@ -229,15 +229,16 @@
# - Where to Log -
#log_destination = 'stderr' # Valid values are combinations of
# stderr, syslog and eventlog,
# stderr, csvlog, syslog and eventlog,
# depending on platform.
# csvlog requires logging_collector to be on
# This is used when logging to stderr:
#redirect_stderr = off # Enable capturing of stderr into log
# files
#logging_collector = off # Enable capturing of stderr and csvlog
# into log files. Required to be on for csvlogs.
# (change requires restart)
# These are only used if redirect_stderr is on:
# These are only used if logging_collector is on:
#log_directory = 'pg_log' # Directory where log files are written
# Can be absolute or relative to PGDATA
#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern.
......
src/include/postmaster/syslogger.h
View file @ fd801f4f
......@@ -5,7 +5,7 @@
*
* Copyright (c) 2004-2007, PostgreSQL Global Development Group
*
* $PostgreSQL: pgsql/src/include/postmaster/syslogger.h,v 1.10 2007/07/25 12:22:53 mha Exp $
* $PostgreSQL: pgsql/src/include/postmaster/syslogger.h,v 1.11 2007/08/19 01:41:25 adunstan Exp $
*
*-------------------------------------------------------------------------
*/
......@@ -24,9 +24,9 @@
* also cope with non-protocol data coming down the pipe, though we cannot
* guarantee long strings won't get split apart.
*
* We use 't' or 'f' instead of a bool for is_last to make the protocol a tiny
* bit more robust against finding a false double nul byte prologue. But we
* still might find it in the len and/or pid bytes unless we're careful.
* We use non-nul bytes in is_last to make the protocol a tiny bit
* more robust against finding a false double nul byte prologue. But
* we still might find it in the len and/or pid bytes unless we're careful.
*/
#ifdef PIPE_BUF
......@@ -46,7 +46,9 @@ typedef struct
char nuls[2]; /* always \0\0 */
uint16 len; /* size of this chunk (counts data only) */
int32 pid; /* writer's pid */
char is_last; /* last chunk of message? 't' or 'f' */
char is_last; /* last chunk of message? 't' or 'f'
* ('T' or 'F' for CSV case)
*/
char data[1]; /* data payload starts here */
} PipeProtoHeader;
......@@ -61,7 +63,7 @@ typedef union
/* GUC options */
extern bool Redirect_stderr;
extern bool Logging_collector;
extern int Log_RotationAge;
extern int Log_RotationSize;
extern PGDLLIMPORT char *Log_directory;
......@@ -79,7 +81,7 @@ extern HANDLE syslogPipe[2];
extern int SysLogger_Start(void);
extern void write_syslogger_file(const char *buffer, int count);
extern void write_syslogger_file(const char *buffer, int count, int dest);
#ifdef EXEC_BACKEND
extern void SysLoggerMain(int argc, char *argv[]);
......
src/include/utils/elog.h
View file @ fd801f4f
......@@ -7,7 +7,7 @@
* Portions Copyright (c) 1996-2007, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* $PostgreSQL: pgsql/src/include/utils/elog.h,v 1.87 2007/07/25 12:22:54 mha Exp $
* $PostgreSQL: pgsql/src/include/utils/elog.h,v 1.88 2007/08/19 01:41:25 adunstan Exp $
*
*-------------------------------------------------------------------------
*/
......@@ -291,6 +291,7 @@ extern int Log_destination;
#define LOG_DESTINATION_STDERR 1
#define LOG_DESTINATION_SYSLOG 2
#define LOG_DESTINATION_EVENTLOG 4
#define LOG_DESTINATION_CSVLOG 8
/* Other exported functions */
extern void DebugFileOpen(void);
......
src/test/bench/runwisc.sh
View file @ fd801f4f
#!/bin/sh
# $PostgreSQL: pgsql/src/test/bench/runwisc.sh,v 1.9 2007/08/01 22:24:32 momjian Exp $
# $PostgreSQL: pgsql/src/test/bench/runwisc.sh,v 1.10 2007/08/19 01:41:25 adunstan Exp $
if [ ! -d $1 ]; then
echo " you must specify a valid data directory " >&2
......@@ -14,4 +14,4 @@ echo =============== vacuuming benchmark database... ================= >&2
echo "vacuum" | postgres -D"$1" bench > /dev/null
echo =============== running benchmark... ================= >&2
time postgres -D"$1" -texecutor -tplanner -c log_min_messages=log -c log_destination=stderr -c redirect_stderr=off bench < bench.sql 2>&1
time postgres -D"$1" -texecutor -tplanner -c log_min_messages=log -c log_destination=stderr -c start_log_collector=off bench < bench.sql 2>&1
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment