Skip to content

P
Postgres FD Implementation
Commit 048d148f authored by Magnus Hagander's avatar Magnus Hagander
Browse files
Options

Add pg_basebackup tool for streaming base backups 

This tool makes it possible to do the pg_start_backup/
copy files/pg_stop_backup step in a single command.

There are still some steps to be done before this is a
complete backup solution, such as the ability to stream
the required WAL logs, but it's still usable, and
could do with some buildfarm coverage.

In passing, make the checkpoint request optionally
fast instead of hardcoding it.

Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
parent 6f59777c
Hide whitespace changes
Inline Side-by-side
Showing with 1528 additions and 13 deletions
doc/src/sgml/backup.sgml
View file @ 048d148f
......@@ -812,6 +812,16 @@ SELECT pg_stop_backup();
</orderedlist>
</para>
<para>
You can also use the <xref linkend="app-pgbasebackup"> tool to take
the backup, instead of manually copying the files. This tool will take
care of the <function>pg_start_backup()</>, copy and
<function>pg_stop_backup()</> steps automatically, and transfers the
backup over a regular <productname>PostgreSQL</productname> connection
using the replication protocol, instead of requiring filesystem level
access.
</para>
<para>
Some file system backup tools emit warnings or errors
if the files they are trying to copy change while the copy proceeds.
......
doc/src/sgml/config.sgml
View file @ 048d148f
......@@ -1921,7 +1921,8 @@ SET ENABLE_SEQSCAN TO OFF;
<listitem>
<para>
Specifies the maximum number of concurrent connections from standby
servers (i.e., the maximum number of simultaneously running WAL sender
servers or streaming base backup clients (i.e., the maximum number of
simultaneously running WAL sender
processes). The default is zero. This parameter can only be set at
server start. <varname>wal_level</> must be set to <literal>archive</>
or <literal>hot_standby</> to allow connections from standby servers.
......
doc/src/sgml/protocol.sgml
View file @ 048d148f
......@@ -1460,7 +1460,7 @@ The commands accepted in walsender mode are:
</varlistentry>
<varlistentry>
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>]</term>
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>]</term>
<listitem>
<para>
Instructs the server to start streaming a base backup.
......@@ -1496,6 +1496,15 @@ The commands accepted in walsender mode are:
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>FAST</></term>
<listitem>
<para>
Request a fast checkpoint.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
......
doc/src/sgml/ref/allfiles.sgml
View file @ 048d148f
......@@ -160,6 +160,7 @@ Complete list of usable sgml source files in this directory.
<!entity dropuser system "dropuser.sgml">
<!entity ecpgRef system "ecpg-ref.sgml">
<!entity initdb system "initdb.sgml">
<!entity pgBasebackup system "pg_basebackup.sgml">
<!entity pgConfig system "pg_config-ref.sgml">
<!entity pgControldata system "pg_controldata.sgml">
<!entity pgCtl system "pg_ctl-ref.sgml">
......
doc/src/sgml/ref/pg_basebackup.sgml 0 → 100644
View file @ 048d148f
<!--
doc/src/sgml/ref/pg_basebackup.sgml
PostgreSQL documentation
-->
<refentry id="app-pgbasebackup">
<refmeta>
<refentrytitle>pg_basebackup</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo>Application</refmiscinfo>
</refmeta>
<refnamediv>
<refname>pg_basebackup</refname>
<refpurpose>take a base backup of a <productname>PostgreSQL</productname> cluster</refpurpose>
</refnamediv>
<indexterm zone="app-pgbasebackup">
<primary>pg_basebackup</primary>
</indexterm>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_basebackup</command>
<arg rep="repeat"><replaceable>option</></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsect1>
<title>
Description
</title>
<para>
<application>pg_basebackup</application> is used to take base backups of
a running <productname>PostgreSQL</productname> database cluster. These
are taken without affecting other clients to the database, and can be used
both for point-in-time recovery (see <xref linkend="continuous-archiving">)
and as the starting point for a log shipping or streaming replication standby
servers (see <xref linkend="warm-standby">).
</para>
<para>
<application>pg_basebackup</application> makes a binary copy of the database
cluster files, while making sure the system is automatically put in and
out of backup mode automatically. Backups are always taken of the entire
database cluster, it is not possible to back up individual databases or
database objects. For individual database backups, a tool such as
<xref linkend="APP-PGDUMP"> must be used.
</para>
<para>
The backup is made over a regular <productname>PostgreSQL</productname>
connection, and uses the replication protocol. The connection must be
made with a user having <literal>REPLICATION</literal> permissions (see
<xref linkend="role-attributes">), and the user must be granted explicit
permissions in <filename>pg_hba.conf</filename>. The server must also
be configured with <xref linkend="guc-max-wal-senders"> set high enough
to leave at least one session available for the backup.
</para>
<para>
Only one backup can be concurrently active in
<productname>PostgreSQL</productname>, meaning that only one instance of
<application>pg_basebackup</application> can run at the same time
against a single database cluster.
</para>
</refsect1>
<refsect1>
<title>Options</title>
<para>
The following command-line options control the location and format of the
output.
<variablelist>
<varlistentry>
<term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
<term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
<listitem>
<para>
Directory to write the output to.
</para>
<para>
When the backup is in tar mode, and the directory is specified as
<literal>-</literal> (dash), the tar file will be written to
<literal>stdout</literal>.
</para>
<para>
This parameter is required.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
<listitem>
<para>
Selects the format for the output. <replaceable>format</replaceable>
can be one of the following:
<variablelist>
<varlistentry>
<term><literal>p</literal></term>
<term><literal>plain</literal></term>
<listitem>
<para>
Write the output as plain files, with the same layout as the
current data directory and tablespaces. When the cluster has
no additional tablespaces, the whole database will be placed in
the target directory. If the cluster contains additional
tablespaces, the main data directory will be placed in the
target directory, but all other tablespaces will be placed
in the same absolute path as they have on the server.
</para>
<para>
This is the default format.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>t</literal></term>
<term><literal>tar</literal></term>
<listitem>
<para>
Write the output as tar files in the target directory. The main
data directory will be written to a file named
<filename>base.tar</filename>, and all other tablespaces will
be named after the tablespace oid.
</para>
<para>
If the value <literal>-</literal> (dash) is specified as
target directory, the tar contents will be written to
standard output, suitable for piping to for example
<productname>gzip</productname>. This is only possible if
the cluster has no additional tablespaces.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
Enables gzip compression of tar file output. Compression is only
available when generating tar files, and is not available when sending
output to standard output.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
The following command-line options control the generation of the
backup and the running of the program.
<variablelist>
<varlistentry>
<term><option>-c <replaceable class="parameter">fast|spread</replaceable></option></term>
<term><option>--checkpoint <replaceable class="parameter">fast|spread</replaceable></option></term>
<listitem>
<para>
Sets checkpoint mode to fast or spread (default).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-l <replaceable class="parameter">label</replaceable></option></term>
<term><option>--label=<replaceable class="parameter">label</replaceable></option></term>
<listitem>
<para>
Sets the label for the backup. If none is specified, a default value of
<literal>pg_basebackup base backup</literal> will be used.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-P</option></term>
<term><option>--progress</option></term>
<listitem>
<para>
Enables progress reporting. Turning this on will deliver an approximate
progress report during the backup. Since the database may change during
the backup, this is only an approximation and may not end at exactly
<literal>100%</literal>.
</para>
<para>
When this is enabled, the backup will start by enumerating the size of
the entire database, and then go back and send the actual contents.
This may make the backup take slightly longer, and in particular it
will take longer before the first data is sent.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-v</option></term>
<term><option>--verbose</option></term>
<listitem>
<para>
Enables verbose mode. Will output some extra steps during startup and
shutdown, as well as show the exact filename that is currently being
processed if progress reporting is also enabled.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
The following command-line options control the database connection parameters.
<variablelist>
<varlistentry>
<term><option>-h <replaceable class="parameter">host</replaceable></option></term>
<term><option>--host=<replaceable class="parameter">host</replaceable></option></term>
<listitem>
<para>
Specifies the host name of the machine on which the server is
running. If the value begins with a slash, it is used as the
directory for the Unix domain socket. The default is taken
from the <envar>PGHOST</envar> environment variable, if set,
else a Unix domain socket connection is attempted.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-p <replaceable class="parameter">port</replaceable></option></term>
<term><option>--port=<replaceable class="parameter">port</replaceable></option></term>
<listitem>
<para>
Specifies the TCP port or local Unix domain socket file
extension on which the server is listening for connections.
Defaults to the <envar>PGPORT</envar> environment variable, if
set, or a compiled-in default.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-U <replaceable>username</replaceable></option></term>
<term><option>--username=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
<para>
User name to connect as.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-w</></term>
<term><option>--no-password</></term>
<listitem>
<para>
Never issue a password prompt. If the server requires
password authentication and a password is not available by
other means such as a <filename>.pgpass</filename> file, the
connection attempt will fail. This option can be useful in
batch jobs and scripts where no user is present to enter a
password.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-W</option></term>
<term><option>--password</option></term>
<listitem>
<para>
Force <application>pg_basebackup</application> to prompt for a
password before connecting to a database.
</para>
<para>
This option is never essential, since
<application>pg_bsaebackup</application> will automatically prompt
for a password if the server demands password authentication.
However, <application>pg_basebackup</application> will waste a
connection attempt finding out that the server wants a password.
In some cases it is worth typing <option>-W</> to avoid the extra
connection attempt.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
Other, less commonly used, parameters are also available:
<variablelist>
<varlistentry>
<term><option>-V</></term>
<term><option>--version</></term>
<listitem>
<para>
Print the <application>pg_basebackup</application> version and exit.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-?</></term>
<term><option>--help</></term>
<listitem>
<para>
Show help about <application>pg_basebackup</application> command line
arguments, and exit.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</refsect1>
<refsect1>
<title>Environment</title>
<para>
This utility, like most other <productname>PostgreSQL</> utilities,
uses the environment variables supported by <application>libpq</>
(see <xref linkend="libpq-envars">).
</para>
</refsect1>
<refsect1>
<title>Notes</title>
<para>
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties. Only regular files and directories are allowed
in the data directory, no symbolic links or special device files.
</para>
<para>
The way <productname>PostgreSQL</productname> manages tablespaces, the path
for all additional tablespaces must be identical whenever a backup is
restored. The main data directory, however, is relocatable to any location.
</para>
</refsect1>
<refsect1>
<title>Examples</title>
<para>
To create a base backup of the server at <literal>mydbserver</literal>
and store it in the local directory
<filename>/usr/local/pgsql/data</filename>:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -h mydbserver -D /usr/local/pgsql/data</userinput>
</screen>
</para>
<para>
To create a backup of the local server with one maximum compressed
tar file for each tablespace, and store it in the directory
<filename>backup</filename>, showing a progress report while running:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -D backup -Ft -Z9 -P</userinput>
</screen>
</para>
<para>
To create a backup of a single-tablespace local database and compress
this with <productname>bzip2</productname>:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -D - -Ft | bzip2 > backup.tar.bz2</userinput>
</screen>
(this command will fail if there are multiple tablespaces in the
database)
</para>
</refsect1>
<refsect1>
<title>See Also</title>
<simplelist type="inline">
<member><xref linkend="APP-PGDUMP"></member>
</simplelist>
</refsect1>
</refentry>
doc/src/sgml/reference.sgml
View file @ 048d148f
......@@ -202,6 +202,7 @@
&droplang;
&dropuser;
&ecpgRef;
&pgBasebackup;
&pgConfig;
&pgDump;
&pgDumpall;
......
src/backend/replication/basebackup.c
View file @ 048d148f
......@@ -40,7 +40,7 @@ static void send_int8_string(StringInfoData *buf, int64 intval);
static void SendBackupHeader(List *tablespaces);
static void SendBackupDirectory(char *location, char *spcoid);
static void base_backup_cleanup(int code, Datum arg);
static void perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir);
static void perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir, bool fastcheckpoint);
typedef struct
{
......@@ -67,9 +67,9 @@ base_backup_cleanup(int code, Datum arg)
* clobbered by longjmp" from stupider versions of gcc.
*/
static void
perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir)
perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir, bool fastcheckpoint)
{
do_pg_start_backup(backup_label, true);
do_pg_start_backup(backup_label, fastcheckpoint);
PG_ENSURE_ERROR_CLEANUP(base_backup_cleanup, (Datum) 0);
{
......@@ -135,7 +135,7 @@ perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir)
* pg_stop_backup() for the user.
*/
void
SendBaseBackup(const char *backup_label, bool progress)
SendBaseBackup(const char *backup_label, bool progress, bool fastcheckpoint)
{
DIR *dir;
MemoryContext backup_context;
......@@ -168,7 +168,7 @@ SendBaseBackup(const char *backup_label, bool progress)
ereport(ERROR,
(errmsg("unable to open directory pg_tblspc: %m")));
perform_base_backup(backup_label, progress, dir);
perform_base_backup(backup_label, progress, dir, fastcheckpoint);
FreeDir(dir);
......@@ -333,7 +333,16 @@ sendDir(char *path, int basepathlen, bool sizeonly)
if (strcmp(pathbuf, "./pg_xlog") == 0)
{
if (!sizeonly)
{
/* If pg_xlog is a symlink, write it as a directory anyway */
#ifndef WIN32
if (S_ISLNK(statbuf.st_mode))
#else
if (pgwin32_is_junction(pathbuf))
#endif
statbuf.st_mode = S_IFDIR | S_IRWXU;
_tarWriteHeader(pathbuf + basepathlen + 1, NULL, &statbuf);
}
size += 512; /* Size of the header just added */
continue; /* don't recurse into pg_xlog */
}
......
src/backend/replication/repl_gram.y
View file @ 048d148f
......@@ -66,11 +66,12 @@ Node *replication_parse_result;
%token K_IDENTIFY_SYSTEM
%token K_LABEL
%token K_PROGRESS
%token K_FAST
%token K_START_REPLICATION
%type <node> command
%type <node> base_backup start_replication identify_system
%type <boolval> opt_progress
%type <boolval> opt_progress opt_fast
%type <str> opt_label
%%
......@@ -102,15 +103,16 @@ identify_system:
;
/*
* BASE_BACKUP [LABEL <label>] [PROGRESS]
* BASE_BACKUP [LABEL <label>] [PROGRESS] [FAST]
*/
base_backup:
K_BASE_BACKUP opt_label opt_progress
K_BASE_BACKUP opt_label opt_progress opt_fast
{
BaseBackupCmd *cmd = (BaseBackupCmd *) makeNode(BaseBackupCmd);
cmd->label = $2;
cmd->progress = $3;
cmd->fastcheckpoint = $4;
$$ = (Node *) cmd;
}
......@@ -123,6 +125,9 @@ opt_label: K_LABEL SCONST { $$ = $2; }
opt_progress: K_PROGRESS { $$ = true; }
| /* EMPTY */ { $$ = false; }
;
opt_fast: K_FAST { $$ = true; }
| /* EMPTY */ { $$ = false; }
;
/*
* START_REPLICATION %X/%X
......
src/backend/replication/repl_scanner.l
View file @ 048d148f
......@@ -57,6 +57,7 @@ quotestop {quote}
%%
BASE_BACKUP { return K_BASE_BACKUP; }
FAST { return K_FAST; }
IDENTIFY_SYSTEM { return K_IDENTIFY_SYSTEM; }
LABEL { return K_LABEL; }
PROGRESS { return K_PROGRESS; }
......
src/backend/replication/walsender.c
View file @ 048d148f
......@@ -402,7 +402,7 @@ HandleReplicationCommand(const char *cmd_string)
{
BaseBackupCmd *cmd = (BaseBackupCmd *) cmd_node;
SendBaseBackup(cmd->label, cmd->progress);
SendBaseBackup(cmd->label, cmd->progress, cmd->fastcheckpoint);
/* Send CommandComplete and ReadyForQuery messages */
EndCommand("SELECT", DestRemote);
......
src/bin/Makefile
View file @ 048d148f
......@@ -14,7 +14,7 @@ top_builddir = ../..
include $(top_builddir)/src/Makefile.global
SUBDIRS = initdb pg_ctl pg_dump \
psql scripts pg_config pg_controldata pg_resetxlog
psql scripts pg_config pg_controldata pg_resetxlog pg_basebackup
ifeq ($(PORTNAME), win32)
SUBDIRS+=pgevent
endif
......
src/bin/pg_basebackup/Makefile 0 → 100644
View file @ 048d148f
#-------------------------------------------------------------------------
#
# Makefile for src/bin/pg_basebackup
#
# Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
# Portions Copyright (c) 1994, Regents of the University of California
#
# src/bin/pg_basebackup/Makefile
#
#-------------------------------------------------------------------------
PGFILEDESC = "pg_basebackup - takes a streaming base backup of a PostgreSQL instance"
PGAPPICON=win32
subdir = src/bin/pg_basebackup
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
OBJS= pg_basebackup.o $(WIN32RES)
all: pg_basebackup
pg_basebackup: $(OBJS) | submake-libpq submake-libpgport
$(CC) $(CFLAGS) $(OBJS) $(libpq_pgport) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
clean distclean maintainer-clean:
rm -f pg_basebackup$(X) $(OBJS)
src/bin/pg_basebackup/nls.mk 0 → 100644
View file @ 048d148f
# src/bin/pg_basebackup/nls.mk
CATALOG_NAME := pg_basebackup
AVAIL_LANGUAGES :=
GETTEXT_FILES := pg_basebackup.c
GETTEXT_TRIGGERS:= _
src/bin/pg_basebackup/pg_basebackup.c 0 → 100644
View file @ 048d148f
/*-------------------------------------------------------------------------
*
* pg_basebackup.c - receive a base backup using streaming replication protocol
*
* Author: Magnus Hagander <magnus@hagander.net>
*
* Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/bin/pg_basebackup/pg_basebackup.c
*-------------------------------------------------------------------------
*/
#include "postgres_fe.h"
#include "libpq-fe.h"
#include <unistd.h>
#include <dirent.h>
#include <sys/stat.h>
#ifdef HAVE_LIBZ
#include <zlib.h>
#endif
#include "getopt_long.h"
/* Global options */
static const char *progname;
char *basedir = NULL;
char format = 'p'; /* p(lain)/t(ar) */
char *label = "pg_basebackup base backup";
bool showprogress = false;
int verbose = 0;
int compresslevel = 0;
bool fastcheckpoint = false;
char *dbhost = NULL;
char *dbuser = NULL;
char *dbport = NULL;
int dbgetpassword = 0; /* 0=auto, -1=never, 1=always */
/* Progress counters */
static uint64 totalsize;
static uint64 totaldone;
static int tablespacecount;
/* Connection kept global so we can disconnect easily */
static PGconn *conn = NULL;
#define disconnect_and_exit(code) \
{ \
if (conn != NULL) PQfinish(conn); \
exit(code); \
}
/* Function headers */
static char *xstrdup(const char *s);
static void *xmalloc0(int size);
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname);
static void progress_report(int tablespacenum, char *fn);
static PGconn *GetConnection(void);
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
static void BaseBackup();
#ifdef HAVE_LIBZ
static const char *
get_gz_error(gzFile *gzf)
{
int errnum;
const char *errmsg;
errmsg = gzerror(gzf, &errnum);
if (errnum == Z_ERRNO)
return strerror(errno);
else
return errmsg;
}
#endif
/*
* strdup() and malloc() replacements that prints an error and exits
* if something goes wrong. Can never return NULL.
*/
static char *
xstrdup(const char *s)
{
char *result;
result = strdup(s);
if (!result)
{
fprintf(stderr, _("%s: out of memory\n"), progname);
exit(1);
}
return result;
}
static void *
xmalloc0(int size)
{
void *result;
result = malloc(size);
if (!result)
{
fprintf(stderr, _("%s: out of memory\n"), progname);
exit(1);
}
MemSet(result, 0, size);
return result;
}
static void
usage(void)
{
printf(_("%s takes base backups of running PostgreSQL servers\n\n"),
progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions controlling the output:\n"));
printf(_(" -D, --pgdata=directory receive base backup into directory\n"));
printf(_(" -F, --format=p|t output format (plain, tar)\n"));
printf(_(" -Z, --compress=0-9 compress tar output\n"));
printf(_("\nGeneral options:\n"));
printf(_(" -c, --checkpoint=fast|spread\n"
" set fast or spread checkpointing\n"));
printf(_(" -l, --label=label set backup label\n"));
printf(_(" -P, --progress show progress information\n"));
printf(_(" -v, --verbose output verbose messages\n"));
printf(_(" -?, --help show this help, then exit\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_("\nConnection options:\n"));
printf(_(" -h, --host=HOSTNAME database server host or socket directory\n"));
printf(_(" -p, --port=PORT database server port number\n"));
printf(_(" -U, --username=NAME connect as specified database user\n"));
printf(_(" -w, --no-password never prompt for password\n"));
printf(_(" -W, --password force password prompt (should happen automatically)\n"));
printf(_("\nReport bugs to <pgsql-bugs@postgresql.org>.\n"));
}
/*
* Verify that the given directory exists and is empty. If it does not
* exist, it is created. If it exists but is not empty, an error will
* be give and the process ended.
*/
static void
verify_dir_is_empty_or_create(char *dirname)
{
switch (pg_check_dir(dirname))
{
case 0:
/*
* Does not exist, so create
*/
if (pg_mkdir_p(dirname, S_IRWXU) == -1)
{
fprintf(stderr,
_("%s: could not create directory \"%s\": %s\n"),
progname, dirname, strerror(errno));
disconnect_and_exit(1);
}
return;
case 1:
/*
* Exists, empty
*/
return;
case 2:
/*
* Exists, not empty
*/
fprintf(stderr,
_("%s: directory \"%s\" exists but is not empty\n"),
progname, dirname);
disconnect_and_exit(1);
case -1:
/*
* Access problem
*/
fprintf(stderr, _("%s: could not access directory \"%s\": %s\n"),
progname, dirname, strerror(errno));
disconnect_and_exit(1);
}
}
/*
* Print a progress report based on the global variables. If verbose output
* is enabled, also print the current file name.
*/
static void
progress_report(int tablespacenum, char *fn)
{
if (verbose)
fprintf(stderr,
INT64_FORMAT "/" INT64_FORMAT " kB (%i%%) %i/%i tablespaces (%-30s)\r",
totaldone / 1024, totalsize,
(int) ((totaldone / 1024) * 100 / totalsize),
tablespacenum, tablespacecount, fn);
else
fprintf(stderr, INT64_FORMAT "/" INT64_FORMAT " kB (%i%%) %i/%i tablespaces\r",
totaldone / 1024, totalsize,
(int) ((totaldone / 1024) * 100 / totalsize),
tablespacenum, tablespacecount);
}
/*
* Receive a tar format file from the connection to the server, and write
* the data from this file directly into a tar file. If compression is
* enabled, the data will be compressed while written to the file.
*
* The file will be named base.tar[.gz] if it's for the main data directory
* or <tablespaceoid>.tar[.gz] if it's for another tablespace.
*
* No attempt to inspect or validate the contents of the file is done.
*/
static void
ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
{
char fn[MAXPGPATH];
char *copybuf = NULL;
FILE *tarfile = NULL;
#ifdef HAVE_LIBZ
gzFile *ztarfile = NULL;
#endif
if (PQgetisnull(res, rownum, 0))
/*
* Base tablespaces
*/
if (strcmp(basedir, "-") == 0)
tarfile = stdout;
else
{
#ifdef HAVE_LIBZ
if (compresslevel > 0)
{
snprintf(fn, sizeof(fn), "%s/base.tar.gz", basedir);
ztarfile = gzopen(fn, "wb");
if (gzsetparams(ztarfile, compresslevel, Z_DEFAULT_STRATEGY) != Z_OK)
{
fprintf(stderr, _("%s: could not set compression level %i: %s\n"),
progname, compresslevel, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
snprintf(fn, sizeof(fn), "%s/base.tar", basedir);
tarfile = fopen(fn, "wb");
}
}
else
{
/*
* Specific tablespace
*/
#ifdef HAVE_LIBZ
if (compresslevel > 0)
{
snprintf(fn, sizeof(fn), "%s/%s.tar.gz", basedir, PQgetvalue(res, rownum, 0));
ztarfile = gzopen(fn, "wb");
if (gzsetparams(ztarfile, compresslevel, Z_DEFAULT_STRATEGY) != Z_OK)
{
fprintf(stderr, _("%s: could not set compression level %i: %s\n"),
progname, compresslevel, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
snprintf(fn, sizeof(fn), "%s/%s.tar", basedir, PQgetvalue(res, rownum, 0));
tarfile = fopen(fn, "wb");
}
}
#ifdef HAVE_LIBZ
if (compresslevel > 0)
{
if (!ztarfile)
{
/* Compression is in use */
fprintf(stderr, _("%s: could not create compressed file \"%s\": %s\n"),
progname, fn, get_gz_error(ztarfile));
disconnect_and_exit(1);
}
}
else
#endif
{
/* Either no zlib support, or zlib support but compresslevel = 0 */
if (!tarfile)
{
fprintf(stderr, _("%s: could not create file \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
}
/*
* Get the COPY data stream
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COPY_OUT)
{
fprintf(stderr, _("%s: could not get COPY data stream: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
while (1)
{
int r;
if (copybuf != NULL)
{
PQfreemem(copybuf);
copybuf = NULL;
}
r = PQgetCopyData(conn, &copybuf, 0);
if (r == -1)
{
/*
* End of chunk. Close file (but not stdout).
*
* Also, write two completely empty blocks at the end of the tar
* file, as required by some tar programs.
*/
char zerobuf[1024];
MemSet(zerobuf, 0, sizeof(zerobuf));
#ifdef HAVE_LIBZ
if (ztarfile != NULL)
{
if (gzwrite(ztarfile, zerobuf, sizeof(zerobuf)) != sizeof(zerobuf))
{
fprintf(stderr, _("%s: could not write to compressed file \"%s\": %s\n"),
progname, fn, get_gz_error(ztarfile));
}
}
else
#endif
{
if (fwrite(zerobuf, sizeof(zerobuf), 1, tarfile) != 1)
{
fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
}
if (strcmp(basedir, "-") != 0)
{
#ifdef HAVE_LIBZ
if (ztarfile != NULL)
gzclose(ztarfile);
#endif
if (tarfile != NULL)
fclose(tarfile);
}
break;
}
else if (r == -2)
{
fprintf(stderr, _("%s: could not read COPY data: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
#ifdef HAVE_LIBZ
if (ztarfile != NULL)
{
if (gzwrite(ztarfile, copybuf, r) != r)
{
fprintf(stderr, _("%s: could not write to compressed file \"%s\": %s\n"),
progname, fn, get_gz_error(ztarfile));
}
}
else
#endif
{
if (fwrite(copybuf, r, 1, tarfile) != 1)
{
fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
}
totaldone += r;
if (showprogress)
progress_report(rownum, fn);
} /* while (1) */
if (copybuf != NULL)
PQfreemem(copybuf);
}
/*
* Receive a tar format stream from the connection to the server, and unpack
* the contents of it into a directory. Only files, directories and
* symlinks are supported, no other kinds of special files.
*
* If the data is for the main data directory, it will be restored in the
* specified directory. If it's for another tablespace, it will be restored
* in the original directory, since relocation of tablespaces is not
* supported.
*/
static void
ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
{
char current_path[MAXPGPATH];
char fn[MAXPGPATH];
int current_len_left;
int current_padding;
char *copybuf = NULL;
FILE *file = NULL;
if (PQgetisnull(res, rownum, 0))
strcpy(current_path, basedir);
else
strcpy(current_path, PQgetvalue(res, rownum, 1));
/*
* Make sure we're unpacking into an empty directory
*/
verify_dir_is_empty_or_create(current_path);
/*
* Get the COPY data
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COPY_OUT)
{
fprintf(stderr, _("%s: could not get COPY data stream: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
while (1)
{
int r;
if (copybuf != NULL)
{
PQfreemem(copybuf);
copybuf = NULL;
}
r = PQgetCopyData(conn, &copybuf, 0);
if (r == -1)
{
/*
* End of chunk
*/
if (file)
fclose(file);
break;
}
else if (r == -2)
{
fprintf(stderr, _("%s: could not read COPY data: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
if (file == NULL)
{
#ifndef WIN32
mode_t filemode;
#endif
/*
* No current file, so this must be the header for a new file
*/
if (r != 512)
{
fprintf(stderr, _("%s: Invalid tar block header size: %i\n"),
progname, r);
disconnect_and_exit(1);
}
totaldone += 512;
if (sscanf(copybuf + 124, "%11o", &current_len_left) != 1)
{
fprintf(stderr, _("%s: could not parse file size!\n"),
progname);
disconnect_and_exit(1);
}
/* Set permissions on the file */
if (sscanf(&copybuf[100], "%07o ", &filemode) != 1)
{
fprintf(stderr, _("%s: could not parse file mode!\n"),
progname);
disconnect_and_exit(1);
}
/*
* All files are padded up to 512 bytes
*/
current_padding =
((current_len_left + 511) & ~511) - current_len_left;
/*
* First part of header is zero terminated filename
*/
snprintf(fn, sizeof(fn), "%s/%s", current_path, copybuf);
if (fn[strlen(fn) - 1] == '/')
{
/*
* Ends in a slash means directory or symlink to directory
*/
if (copybuf[156] == '5')
{
/*
* Directory
*/
fn[strlen(fn) - 1] = '\0'; /* Remove trailing slash */
if (mkdir(fn, S_IRWXU) != 0)
{
fprintf(stderr,
_("%s: could not create directory \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
#ifndef WIN32
if (chmod(fn, filemode))
fprintf(stderr, _("%s: could not set permissions on directory \"%s\": %s\n"),
progname, fn, strerror(errno));
#endif
}
else if (copybuf[156] == '2')
{
/*
* Symbolic link
*/
fn[strlen(fn) - 1] = '\0'; /* Remove trailing slash */
if (symlink(&copybuf[157], fn) != 0)
{
fprintf(stderr,
_("%s: could not create symbolic link from %s to %s: %s\n"),
progname, fn, &copybuf[157], strerror(errno));
disconnect_and_exit(1);
}
}
else
{
fprintf(stderr, _("%s: unknown link indicator \"%c\"\n"),
progname, copybuf[156]);
disconnect_and_exit(1);
}
continue; /* directory or link handled */
}
/*
* regular file
*/
file = fopen(fn, "wb");
if (!file)
{
fprintf(stderr, _("%s: could not create file \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
#ifndef WIN32
if (chmod(fn, filemode))
fprintf(stderr, _("%s: could not set permissions on file \"%s\": %s\n"),
progname, fn, strerror(errno));
#endif
if (current_len_left == 0)
{
/*
* Done with this file, next one will be a new tar header
*/
fclose(file);
file = NULL;
continue;
}
} /* new file */
else
{
/*
* Continuing blocks in existing file
*/
if (current_len_left == 0 && r == current_padding)
{
/*
* Received the padding block for this file, ignore it and
* close the file, then move on to the next tar header.
*/
fclose(file);
file = NULL;
totaldone += r;
continue;
}
if (fwrite(copybuf, r, 1, file) != 1)
{
fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
progname, fn, strerror(errno));
disconnect_and_exit(1);
}
totaldone += r;
if (showprogress)
progress_report(rownum, fn);
current_len_left -= r;
if (current_len_left == 0 && current_padding == 0)
{
/*
* Received the last block, and there is no padding to be
* expected. Close the file and move on to the next tar
* header.
*/
fclose(file);
file = NULL;
continue;
}
} /* continuing data in existing file */
} /* loop over all data blocks */
if (file != NULL)
{
fprintf(stderr, _("%s: last file was never finsihed!\n"), progname);
disconnect_and_exit(1);
}
if (copybuf != NULL)
PQfreemem(copybuf);
}
static PGconn *
GetConnection(void)
{
PGconn *tmpconn;
int argcount = 4; /* dbname, replication, fallback_app_name,
* password */
int i;
const char **keywords;
const char **values;
char *password = NULL;
if (dbhost)
argcount++;
if (dbuser)
argcount++;
if (dbport)
argcount++;
keywords = xmalloc0((argcount + 1) * sizeof(*keywords));
values = xmalloc0((argcount + 1) * sizeof(*values));
keywords[0] = "dbname";
values[0] = "replication";
keywords[1] = "replication";
values[1] = "true";
keywords[2] = "fallback_application_name";
values[2] = progname;
i = 3;
if (dbhost)
{
keywords[i] = "host";
values[i] = dbhost;
i++;
}
if (dbuser)
{
keywords[i] = "user";
values[i] = dbuser;
i++;
}
if (dbport)
{
keywords[i] = "port";
values[i] = dbport;
i++;
}
while (true)
{
if (dbgetpassword == 1)
{
/* Prompt for a password */
password = simple_prompt(_("Password: "), 100, false);
keywords[argcount - 1] = "password";
values[argcount - 1] = password;
}
tmpconn = PQconnectdbParams(keywords, values, true);
if (password)
free(password);
if (PQstatus(tmpconn) == CONNECTION_BAD &&
PQconnectionNeedsPassword(tmpconn) &&
dbgetpassword != -1)
{
dbgetpassword = 1; /* ask for password next time */
PQfinish(tmpconn);
continue;
}
if (PQstatus(tmpconn) != CONNECTION_OK)
{
fprintf(stderr, _("%s: could not connect to server: %s\n"),
progname, PQerrorMessage(tmpconn));
exit(1);
}
/* Connection ok! */
free(values);
free(keywords);
return tmpconn;
}
}
static void
BaseBackup()
{
PGresult *res;
char current_path[MAXPGPATH];
char escaped_label[MAXPGPATH];
int i;
/*
* Connect in replication mode to the server
*/
conn = GetConnection();
PQescapeStringConn(conn, escaped_label, label, sizeof(escaped_label), &i);
snprintf(current_path, sizeof(current_path), "BASE_BACKUP LABEL '%s' %s %s",
escaped_label,
showprogress ? "PROGRESS" : "",
fastcheckpoint ? "FAST" : "");
if (PQsendQuery(conn, current_path) == 0)
{
fprintf(stderr, _("%s: could not start base backup: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
/*
* Get the header
*/
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
fprintf(stderr, _("%s: could not initiate base backup: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
if (PQntuples(res) < 1)
{
fprintf(stderr, _("%s: no data returned from server.\n"), progname);
disconnect_and_exit(1);
}
/*
* Sum up the total size, for progress reporting
*/
totalsize = totaldone = 0;
tablespacecount = PQntuples(res);
for (i = 0; i < PQntuples(res); i++)
{
if (showprogress)
totalsize += atol(PQgetvalue(res, i, 2));
/*
* Verify tablespace directories are empty. Don't bother with the
* first once since it can be relocated, and it will be checked before
* we do anything anyway.
*/
if (format == 'p' && i > 0)
verify_dir_is_empty_or_create(PQgetvalue(res, i, 1));
}
/*
* When writing to stdout, require a single tablespace
*/
if (format == 't' && strcmp(basedir, "-") == 0 && PQntuples(res) > 1)
{
fprintf(stderr, _("%s: can only write single tablespace to stdout, database has %i.\n"),
progname, PQntuples(res));
disconnect_and_exit(1);
}
/*
* Start receiving chunks
*/
for (i = 0; i < PQntuples(res); i++)
{
if (format == 't')
ReceiveTarFile(conn, res, i);
else
ReceiveAndUnpackTarFile(conn, res, i);
} /* Loop over all tablespaces */
if (showprogress)
{
progress_report(PQntuples(res), "");
fprintf(stderr, "\n"); /* Need to move to next line */
}
PQclear(res);
res = PQgetResult(conn);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
fprintf(stderr, _("%s: final receive failed: %s\n"),
progname, PQerrorMessage(conn));
disconnect_and_exit(1);
}
/*
* End of copy data. Final result is already checked inside the loop.
*/
PQfinish(conn);
if (verbose)
fprintf(stderr, "%s: base backup completed.\n", progname);
}
int
main(int argc, char **argv)
{
static struct option long_options[] = {
{"help", no_argument, NULL, '?'},
{"version", no_argument, NULL, 'V'},
{"pgdata", required_argument, NULL, 'D'},
{"format", required_argument, NULL, 'F'},
{"checkpoint", required_argument, NULL, 'c'},
{"compress", required_argument, NULL, 'Z'},
{"label", required_argument, NULL, 'l'},
{"host", required_argument, NULL, 'h'},
{"port", required_argument, NULL, 'p'},
{"username", required_argument, NULL, 'U'},
{"no-password", no_argument, NULL, 'w'},
{"password", no_argument, NULL, 'W'},
{"verbose", no_argument, NULL, 'v'},
{"progress", no_argument, NULL, 'P'},
{NULL, 0, NULL, 0}
};
int c;
int option_index;
progname = get_progname(argv[0]);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_basebackup"));
if (argc > 1)
{
if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
{
usage();
exit(0);
}
else if (strcmp(argv[1], "-V") == 0
|| strcmp(argv[1], "--version") == 0)
{
puts("pg_basebackup (PostgreSQL) " PG_VERSION);
exit(0);
}
}
while ((c = getopt_long(argc, argv, "D:F:l:Z:c:h:p:U:wWvP",
long_options, &option_index)) != -1)
{
switch (c)
{
case 'D':
basedir = xstrdup(optarg);
break;
case 'F':
if (strcmp(optarg, "p") == 0 || strcmp(optarg, "plain") == 0)
format = 'p';
else if (strcmp(optarg, "t") == 0 || strcmp(optarg, "tar") == 0)
format = 't';
else
{
fprintf(stderr, _("%s: invalid output format \"%s\", must be \"plain\" or \"tar\"\n"),
progname, optarg);
exit(1);
}
break;
case 'l':
label = xstrdup(optarg);
break;
case 'Z':
compresslevel = atoi(optarg);
if (compresslevel <= 0 || compresslevel > 9)
{
fprintf(stderr, _("%s: invalid compression level \"%s\"\n"),
progname, optarg);
exit(1);
}
break;
case 'c':
if (strcasecmp(optarg, "fast") == 0)
fastcheckpoint = true;
else if (strcasecmp(optarg, "spread") == 0)
fastcheckpoint = false;
else
{
fprintf(stderr, _("%s: invalid checkpoint argument \"%s\", must be \"fast\" or \"spread\"\n"),
progname, optarg);
exit(1);
}
break;
case 'h':
dbhost = xstrdup(optarg);
break;
case 'p':
if (atoi(optarg) <= 0)
{
fprintf(stderr, _("%s: invalid port number \"%s\"\n"),
progname, optarg);
exit(1);
}
dbport = xstrdup(optarg);
break;
case 'U':
dbuser = xstrdup(optarg);
break;
case 'w':
dbgetpassword = -1;
break;
case 'W':
dbgetpassword = 1;
break;
case 'v':
verbose++;
break;
case 'P':
showprogress = true;
break;
default:
/*
* getopt_long already emitted a complaint
*/
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
}
/*
* Any non-option arguments?
*/
if (optind < argc)
{
fprintf(stderr,
_("%s: too many command-line arguments (first is \"%s\")\n"),
progname, argv[optind]);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Required arguments
*/
if (basedir == NULL)
{
fprintf(stderr, _("%s: no target directory specified\n"), progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
/*
* Mutually exclusive arguments
*/
if (format == 'p' && compresslevel > 0)
{
fprintf(stderr,
_("%s: only tar mode backups can be compressed\n"),
progname);
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
progname);
exit(1);
}
#ifndef HAVE_LIBZ
if (compresslevel > 0)
{
fprintf(stderr,
_("%s: this build does not support compression\n"),
progname);
exit(1);
}
#else
if (compresslevel > 0 && strcmp(basedir, "-") == 0)
{
fprintf(stderr,
_("%s: compression is not supported on standard output\n"),
progname);
exit(1);
}
#endif
/*
* Verify that the target directory exists, or create it. For plaintext
* backups, always require the directory. For tar backups, require it
* unless we are writing to stdout.
*/
if (format == 'p' || strcmp(basedir, "-") != 0)
verify_dir_is_empty_or_create(basedir);
BaseBackup();
return 0;
}
src/include/replication/basebackup.h
View file @ 048d148f
......@@ -12,6 +12,6 @@
#ifndef _BASEBACKUP_H
#define _BASEBACKUP_H
extern void SendBaseBackup(const char *backup_label, bool progress);
extern void SendBaseBackup(const char *backup_label, bool progress, bool fastcheckpoint);
#endif /* _BASEBACKUP_H */
src/include/replication/replnodes.h
View file @ 048d148f
......@@ -47,6 +47,7 @@ typedef struct BaseBackupCmd
NodeTag type;
char *label;
bool progress;
bool fastcheckpoint;
} BaseBackupCmd;
......
src/tools/msvc/Mkvcbuild.pm
View file @ 048d148f
......@@ -275,6 +275,8 @@ sub mkvcbuild
$initdb->AddLibrary('wsock32.lib');
$initdb->AddLibrary('ws2_32.lib');
my $pgbasebackup = AddSimpleFrontend('pg_basebackup', 1);
my $pgconfig = AddSimpleFrontend('pg_config');
my $pgcontrol = AddSimpleFrontend('pg_controldata');
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment