Skip to content

P
Postgres FD Implementation
Commit 9dac1b46 authored by Tom Lane's avatar Tom Lane
Browse files
Options

Rewrite docs section about routine vacuuming --- it's gotten rather mangled of 

late, with lots of redundancy, bad grammar, and just plain poor exposition.
Make it clear that autovacuum is now considered the normal solution.
parent 2ac64dba
Hide whitespace changes
Inline Side-by-side
Showing with 164 additions and 127 deletions
doc/src/sgml/maintenance.sgml
View file @ 9dac1b46
<!-- $PostgreSQL: pgsql/doc/src/sgml/maintenance.sgml,v 1.85 2007/11/28 15:42:31 petere Exp $ --> <!-- $PostgreSQL: pgsql/doc/src/sgml/maintenance.sgml,v 1.86 2008/06/16 03:13:14 tgl Exp $ -->
<chapter id="maintenance"> <chapter id="maintenance">
<title>Routine Database Maintenance Tasks</title> <title>Routine Database Maintenance Tasks</title>
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
be performed regularly to achieve optimum performance. The tasks be performed regularly to achieve optimum performance. The tasks
discussed here are <emphasis>required</emphasis>, but they discussed here are <emphasis>required</emphasis>, but they
are repetitive in nature and can easily be automated using standard are repetitive in nature and can easily be automated using standard
Unix tools such as <application>cron</application> scripts or tools such as <application>cron</application> scripts or
Windows' <application>Task Scheduler</>. But it is the database Windows' <application>Task Scheduler</>. But it is the database
administrator's responsibility to set up appropriate scripts, and to administrator's responsibility to set up appropriate scripts, and to
check that they execute successfully. check that they execute successfully.
...@@ -59,9 +59,28 @@ ...@@ -59,9 +59,28 @@
</indexterm> </indexterm>
<para> <para>
<productname>PostgreSQL</productname>'s <productname>PostgreSQL</productname> databases require periodic
<xref linkend="sql-vacuum" endterm="sql-vacuum-title"> command has to run on maintenance known as <firstterm>vacuuming</>. For many installations, it
a regular basis for several reasons: is sufficient to let vacuuming be performed by the <firstterm>autovacuum
daemon</>, which is described in <xref linkend="autovacuum">. You might
need to adjust the autovacuuming parameters described there to obtain best
results for your situation. Some database administrators will want to
supplement or replace the daemon's activities with manually-managed
<command>VACUUM</> commands, which typically are executed according to a
schedule by <application>cron</application> or <application>Task
Scheduler</> scripts. To set up manually-managed vacuuming properly,
it is essential to understand the issues discussed in the next few
subsections. Administrators who rely on autovacuuming may still wish
to skim this material to help them understand and adjust autovacuuming.
</para>
<sect2 id="vacuum-basics">
<title>Vacuuming Basics</title>
<para>
<productname>PostgreSQL</productname>'s
<xref linkend="sql-vacuum" endterm="sql-vacuum-title"> command has to
process each table on a regular basis for several reasons:
<orderedlist> <orderedlist>
<listitem> <listitem>
...@@ -79,29 +98,38 @@ ...@@ -79,29 +98,38 @@
<firstterm>transaction ID wraparound</>.</simpara> <firstterm>transaction ID wraparound</>.</simpara>
</listitem> </listitem>
</orderedlist> </orderedlist>
</para>
<para> Each of these reasons dictates performing <command>VACUUM</> operations
The standard form of <command>VACUUM</> can run in parallel with production of varying frequency and scope, as explained in the following subsections.
database operations. Commands such as <command>SELECT</command>, </para>
<command>INSERT</command>, <command>UPDATE</command>, and <command>DELETE</command>
will continue to function as normal, though you will not be able to modify the
definition of a table with commands such as <command>ALTER TABLE ADD COLUMN</command>
while it is being vacuumed.
Also, <command>VACUUM</command> requires a substantial amount of I/O
traffic, which can cause poor performance for other active sessions.
There are configuration parameters that can be adjusted to reduce the
performance impact of background vacuuming &mdash; see
<xref linkend="runtime-config-resource-vacuum-cost">.
</para>
<para> <para>
Fortunately, <xref linkend="autovacuum" endterm="autovacuum-title"> There are two variants of <command>VACUUM</>: standard <command>VACUUM</>
monitors table and <command>VACUUM FULL</>. <command>VACUUM FULL</> can reclaim more
activity and performs <command>VACUUM</command>s when necessary. disk space but runs much more slowly. Also,
Autovacuum works dynamically so it is often better the standard form of <command>VACUUM</> can run in parallel with production
administration-scheduled vacuuming. database operations. (Commands such as <command>SELECT</command>,
</para> <command>INSERT</command>, <command>UPDATE</command>, and
<command>DELETE</command> will continue to function as normal, though you
will not be able to modify the definition of a table with commands such as
<command>ALTER TABLE</command> while it is being vacuumed.)
<command>VACUUM FULL</> requires exclusive lock on the table it is
working on, and therefore cannot be done in parallel with other use
of the table. Another disadvantage of <command>VACUUM FULL</> is that
while it reduces table size, it does not reduce index size proportionally;
in fact it can make indexes <emphasis>larger</>. Generally, therefore,
administrators should strive to use standard <command>VACUUM</> and
avoid <command>VACUUM FULL</>.
</para>
<para>
<command>VACUUM</command> creates a substantial amount of I/O
traffic, which can cause poor performance for other active sessions.
There are configuration parameters that can be adjusted to reduce the
performance impact of background vacuuming &mdash; see
<xref linkend="runtime-config-resource-vacuum-cost">.
</para>
</sect2>
<sect2 id="vacuum-for-space-recovery"> <sect2 id="vacuum-for-space-recovery">
<title>Recovering Disk Space</title> <title>Recovering Disk Space</title>
...@@ -111,72 +139,70 @@ ...@@ -111,72 +139,70 @@
</indexterm> </indexterm>
<para> <para>
In normal <productname>PostgreSQL</productname> operation, an In <productname>PostgreSQL</productname>, an
<command>UPDATE</> or <command>DELETE</> of a row does not <command>UPDATE</> or <command>DELETE</> of a row does not
immediately remove the old version of the row. immediately remove the old version of the row.
This approach is necessary to gain the benefits of multiversion This approach is necessary to gain the benefits of multiversion
concurrency control (see <xref linkend="mvcc">): the row versions concurrency control (see <xref linkend="mvcc">): the row version
must not be deleted while it is still potentially visible to other must not be deleted while it is still potentially visible to other
transactions. But eventually, an outdated or deleted row version is no transactions. But eventually, an outdated or deleted row version is no
longer of interest to any transaction. The space it occupies must be longer of interest to any transaction. The space it occupies must then be
reclaimed for reuse by new rows, to avoid infinite growth of disk reclaimed for reuse by new rows, to avoid infinite growth of disk
space requirements. This is done by running <command>VACUUM</>. space requirements. This is done by running <command>VACUUM</>.
</para> </para>
<para> <para>
There are two variants of the <command>VACUUM</command> The standard form of <command>VACUUM</command> removes dead row
command. The first form, known as <quote>lazy vacuum</quote> or versions in tables and indexes and marks the space available for
just <command>VACUUM</command>, marks dead data in tables and future reuse. However, it will not return the space to the operating
indexes for future reuse; it does <emphasis>not</emphasis> attempt system, except in the special case where one or more pages at the
to reclaim the space used by this dead data unless the space is end of a table become entirely free and an exclusive table lock can be
at the end of the table and an exclusive table lock can be easily easily obtained. In contrast, <command>VACUUM FULL</> actively compacts
obtained. Unused space at the start or middle of the file does tables by moving row versions to earlier pages. It is thus able to
not result in the file being shortened and space returned to the force pages at the end of the table to become entirely free, whereupon
operating system. This variant of <command>VACUUM</command> can be it will return them to the operating system. However, if many rows
run concurrently with normal database operations. must be moved, this can take a long time. Also, moving a row requires
</para> transiently making duplicate index entries for it (the entry pointing
to its new location must be made before the old entry can be removed);
<para> so moving a lot of rows this way causes severe index bloat.
The second form is the <command>VACUUM FULL</command>
command. This uses a more aggressive algorithm for reclaiming the
space consumed by dead row versions. Any space that is freed by
<command>VACUUM FULL</command> is immediately returned to the
operating system, and the table data is physically compacted on
the disk. Unfortunately, this variant of the
<command>VACUUM</command> command acquires an exclusive lock on
each table while <command>VACUUM FULL</command> is processing
it. Therefore, frequently using <command>VACUUM FULL</command> can
have an extremely negative effect on the performance of concurrent
database queries.
</para> </para>
<para> <para>
Fortunately, <xref linkend="autovacuum" endterm="autovacuum-title"> The usual goal of routine vacuuming is to do standard <command>VACUUM</>s
monitors table often enough to avoid needing <command>VACUUM FULL</>. The
activity and performs <command>VACUUM</command>s when necessary. This autovacuum daemon attempts to work this way, and in fact will
eliminates the need for administrators to worry about disk space never issue <command>VACUUM FULL</>. In this approach, the idea
recovery in all but the most unusual cases. is not to keep tables at their minimum size, but to maintain steady-state
usage of disk space: each table occupies space equivalent to its
minimum size plus however much space gets used up between vacuumings.
Although <command>VACUUM FULL</> can be used to shrink a table back
to its minimum size and return the disk space to the operating system,
there is not much point in this if the table will just grow again in the
future. Thus, moderately-frequent standard <command>VACUUM</> runs are a
better approach than infrequent <command>VACUUM FULL</> runs for
maintaining heavily-updated tables.
</para> </para>
<para> <para>
For administrators who want to control <command>VACUUM</command> Some administrators prefer to schedule vacuuming themselves, for example
themselves, the standard form of <command>VACUUM</> is best used to doing all the work at night when load is low.
maintain a steady-state usage of disk space. If you need to return The difficulty with doing vacuuming according to a fixed schedule
disk space to the operating system, you can use <command>VACUUM is that if a table has an unexpected spike in update activity, it may
FULL</>, but this is unwise if the table will just grow again in the get bloated to the point that <command>VACUUM FULL</> is really necessary
future. Moderately-frequent standard <command>VACUUM</> runs are a to reclaim space. Using the autovacuum daemon alleviates this problem,
better approach than infrequent <command>VACUUM FULL</> runs for since the daemon schedules vacuuming dynamically in response to update
maintaining heavily-updated tables. However, if some heavily-updated activity. It is unwise to disable the daemon completely unless you
tables have gone too long with infrequent <command>VACUUM</>, you can have an extremely predictable workload. One possible compromise is
use <command>VACUUM FULL</> or <command>CLUSTER</> to get performance to set the daemon's parameters so that it will only react to unusually
back (it is much slower to scan a table containing almost only dead heavy update activity, thus keeping things from getting out of hand,
rows). while scheduled <command>VACUUM</>s are expected to do the bulk of the
work when the load is typical.
</para> </para>
<para> <para>
For those not using autovacuum, one approach is to schedule a For those not using autovacuum, a typical approach is to schedule a
database-wide <command>VACUUM</> once a day during low-usage period, database-wide <command>VACUUM</> once a day during a low-usage period,
supplemented by more frequent vacuuming of heavily-updated tables if supplemented by more frequent vacuuming of heavily-updated tables as
necessary. (Some installations with extremely high update rates vacuum necessary. (Some installations with extremely high update rates vacuum
their busiest tables as often as once every few minutes.) If you have their busiest tables as often as once every few minutes.) If you have
multiple databases in a cluster, don't forget to multiple databases in a cluster, don't forget to
...@@ -184,24 +210,38 @@ ...@@ -184,24 +210,38 @@
linkend="app-vacuumdb" endterm="app-vacuumdb-title"> might be helpful. linkend="app-vacuumdb" endterm="app-vacuumdb-title"> might be helpful.
</para> </para>
<tip>
<para> <para>
<command>VACUUM FULL</> is recommended for cases where you know Neither form of <command>VACUUM</> is entirely satisfactory when
you have deleted the majority of rows in a table, so that the a table contains large numbers of dead row versions as a result of
steady-state size of the table can be shrunk substantially with massive update or delete activity. If you have such a table and
<command>VACUUM FULL</>'s more aggressive approach. Use plain you need to reclaim the excess disk space it occupies, the best
<command>VACUUM</>, not <command>VACUUM FULL</>, for routine way is to use <xref linkend="sql-cluster" endterm="sql-cluster-title">
vacuuming for space recovery. or one of the table-rewriting variants of
<xref linkend="sql-altertable" endterm="sql-altertable-title">.
These commands rewrite an entire new copy of the table and build
new indexes for it. Like <command>VACUUM FULL</>, they require
exclusive lock. Note that they also temporarily use extra disk
space, since the old copies of the table and indexes can't be
released until the new ones are complete. In the worst case where
your disk is nearly full, <command>VACUUM FULL</> may be the only
workable alternative.
</para> </para>
</tip>
<tip>
<para> <para>
If you have a table whose entire contents are deleted on a periodic If you have a table whose entire contents are deleted on a periodic
basis, consider doing it with <command>TRUNCATE</command> rather basis, consider doing it with
<xref linkend="sql-truncate" endterm="sql-truncate-title"> rather
than using <command>DELETE</command> followed by than using <command>DELETE</command> followed by
<command>VACUUM</command>. <command>TRUNCATE</command> removes the <command>VACUUM</command>. <command>TRUNCATE</command> removes the
entire content of the table immediately, without requiring a entire content of the table immediately, without requiring a
subsequent <command>VACUUM</command> or <command>VACUUM subsequent <command>VACUUM</command> or <command>VACUUM
FULL</command> to reclaim the now-unused disk space. FULL</command> to reclaim the now-unused disk space.
The disadvantage is that strict MVCC semantics are violated.
</para> </para>
</tip>
</sect2> </sect2>
<sect2 id="vacuum-for-statistics"> <sect2 id="vacuum-for-statistics">
...@@ -227,6 +267,18 @@ ...@@ -227,6 +267,18 @@
degrade database performance. degrade database performance.
</para> </para>
<para>
The autovacuum daemon, if enabled, will automatically issue
<command>ANALYZE</> commands whenever the content of a table has
changed sufficiently. However, administrators might prefer to rely
on manually-scheduled <command>ANALYZE</> operations, particularly
if it is known that update activity on a table will not affect the
statistics of <quote>interesting</> columns. The daemon schedules
<command>ANALYZE</> strictly as a function of the number of rows
inserted or updated; it has no knowledge of whether that will lead
to meaningful statistical changes.
</para>
<para> <para>
As with vacuuming for space recovery, frequent updates of statistics As with vacuuming for space recovery, frequent updates of statistics
are more useful for heavily-updated tables than for seldom-updated are more useful for heavily-updated tables than for seldom-updated
...@@ -247,9 +299,10 @@ ...@@ -247,9 +299,10 @@
It is possible to run <command>ANALYZE</> on specific tables and even It is possible to run <command>ANALYZE</> on specific tables and even
just specific columns of a table, so the flexibility exists to update some just specific columns of a table, so the flexibility exists to update some
statistics more frequently than others if your application requires it. statistics more frequently than others if your application requires it.
In practice, however, it is usually best to just analyze the entire database In practice, however, it is usually best to just analyze the entire
because it is a fast operation. It uses a statistical random sampling of database, because it is a fast operation. <command>ANALYZE</> uses a
the rows of a table rather than reading every single row. statistical random sampling of the rows of a table rather than reading
every single row.
</para> </para>
<tip> <tip>
...@@ -257,29 +310,13 @@ ...@@ -257,29 +310,13 @@
Although per-column tweaking of <command>ANALYZE</> frequency might not be Although per-column tweaking of <command>ANALYZE</> frequency might not be
very productive, you might well find it worthwhile to do per-column very productive, you might well find it worthwhile to do per-column
adjustment of the level of detail of the statistics collected by adjustment of the level of detail of the statistics collected by
<command>ANALYZE</>. Columns that are heavily used in <literal>WHERE</> clauses <command>ANALYZE</>. Columns that are heavily used in <literal>WHERE</>
and have highly irregular data distributions might require a finer-grain clauses and have highly irregular data distributions might require a
data histogram than other columns. See <command>ALTER TABLE SET finer-grain data histogram than other columns. See <command>ALTER TABLE
STATISTICS</>. SET STATISTICS</>, or change the database-wide default using the <xref
linkend="guc-default-statistics-target"> configuration parameter.
</para> </para>
</tip> </tip>
<para>
Fortunately, <xref linkend="autovacuum" endterm="autovacuum-title">
monitors table
activity and performs <command>ANALYZE</command>s when necessary. This
eliminates the need for administrators to manually schedule
<command>ANALYZE</command>.
</para>
<para>
For those not using autovacuum, one approach is to schedule a
database-wide <command>ANALYZE</> once a day at a low-usage time of
day; this can usefully be combined with a nightly <command>VACUUM</>.
However, sites with relatively slowly changing table statistics might
find that this is overkill, and that less-frequent <command>ANALYZE</>
runs are sufficient.
</para>
</sect2> </sect2>
<sect2 id="vacuum-for-wraparound"> <sect2 id="vacuum-for-wraparound">
...@@ -342,15 +379,12 @@ ...@@ -342,15 +379,12 @@
<para> <para>
The maximum time that a table can go unvacuumed is two billion The maximum time that a table can go unvacuumed is two billion
transactions minus the <varname>vacuum_freeze_min_age</> that was used transactions minus the <varname>vacuum_freeze_min_age</> that was used
when it was last vacuumed. when it was last vacuumed. If it were to go unvacuumed for longer than
If it were to go unvacuumed for longer than that, that, data loss could result. To ensure that this does not happen,
data loss could result. To ensure that this does not autovacuum is invoked on any table that might contain XIDs older than the
happen, <xref linkend="autovacuum" endterm="autovacuum-title"> age specified by the configuration parameter <xref
is invoked on any table linkend="guc-autovacuum-freeze-max-age">. (This will happen even if
that might contain XIDs older than the age specified by the autovacuum is otherwise disabled.)
configuration parameter
<xref linkend="guc-autovacuum-freeze-max-age">. (This will happen
even if autovacuum is otherwise disabled.)
</para> </para>
<para> <para>
...@@ -464,15 +498,15 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb". ...@@ -464,15 +498,15 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb".
</sect2> </sect2>
<sect2 id="autovacuum"> <sect2 id="autovacuum">
<title id="autovacuum-title">The Auto-Vacuum Daemon</title> <title id="autovacuum-title">The Autovacuum Daemon</title>
<indexterm> <indexterm>
<primary>autovacuum</primary> <primary>autovacuum</primary>
<secondary>general information</secondary> <secondary>general information</secondary>
</indexterm> </indexterm>
<para> <para>
Beginning in <productname>PostgreSQL</productname> 8.1, there is an <productname>PostgreSQL</productname> has an optional but highly
optional feature called <firstterm>autovacuum</firstterm>, recommended feature called <firstterm>autovacuum</firstterm>,
whose purpose is to automate the execution of whose purpose is to automate the execution of
<command>VACUUM</command> and <command>ANALYZE </command> commands. <command>VACUUM</command> and <command>ANALYZE </command> commands.
When enabled, autovacuum checks for When enabled, autovacuum checks for
...@@ -485,20 +519,21 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb". ...@@ -485,20 +519,21 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb".
</para> </para>
<para> <para>
Beginning in <productname>PostgreSQL</productname> 8.3, autovacuum has a The <quote>autovacuum daemon</> actually consists of multiple processes.
multiprocess architecture: There is a daemon process, called the There is a persistent daemon process, called the
<firstterm>autovacuum launcher</firstterm>, which is in charge of starting <firstterm>autovacuum launcher</firstterm>, which is in charge of starting
an <firstterm>autovacuum worker</firstterm> process on each database every an <firstterm>autovacuum worker</firstterm> process on each database every
<xref linkend="guc-autovacuum-naptime"> seconds. On each run, the worker <xref linkend="guc-autovacuum-naptime"> seconds. On each run, the worker
process checks each table within that database, and <command>VACUUM</> or process checks each table within that database, and executes
<command>ANALYZE</> commands are issued as needed. <command>VACUUM</> and/or <command>ANALYZE</> commands as needed.
</para> </para>
<para> <para>
The <xref linkend="guc-autovacuum-max-workers"> setting limits how many The <xref linkend="guc-autovacuum-max-workers"> setting limits how many
workers may be running at any time. If several large tables all become workers may be running at any time. If several large tables all become
eligible for vacuuming in a short amount of time, all autovacuum workers eligible for vacuuming in a short amount of time, all autovacuum workers
may end up vacuuming those tables for a very long time. This would result may become occupied with vacuuming those tables for a long period.
This would result
in other tables and databases not being vacuumed until a worker became in other tables and databases not being vacuumed until a worker became
available. There is not a limit on how many workers might be in a available. There is not a limit on how many workers might be in a
single database, but workers do try to avoid repeating work that has single database, but workers do try to avoid repeating work that has
...@@ -510,9 +545,7 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb". ...@@ -510,9 +545,7 @@ HINT: Stop the postmaster and use a standalone backend to VACUUM in "mydb".
<para> <para>
Tables whose <structfield>relfrozenxid</> value is more than Tables whose <structfield>relfrozenxid</> value is more than
<varname>autovacuum_freeze_max_age</> transactions old are always <varname>autovacuum_freeze_max_age</> transactions old are always
vacuumed. Otherwise, vacuumed. Otherwise, if the number of tuples obsoleted since the last
two conditions are used to determine which operation(s)
to apply. If the number of obsolete tuples since the last
<command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the <command>VACUUM</command> exceeds the <quote>vacuum threshold</quote>, the
table is vacuumed. The vacuum threshold is defined as: table is vacuumed. The vacuum threshold is defined as:
<programlisting> <programlisting>
...@@ -528,7 +561,11 @@ vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuple ...@@ -528,7 +561,11 @@ vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuple
collector; it is a semi-accurate count updated by each collector; it is a semi-accurate count updated by each
<command>UPDATE</command> and <command>DELETE</command> operation. (It <command>UPDATE</command> and <command>DELETE</command> operation. (It
is only semi-accurate because some information might be lost under heavy is only semi-accurate because some information might be lost under heavy
load.) For analyze, a similar condition is used: the threshold, defined as: load.)
</para>
<para>
For analyze, a similar condition is used: the threshold, defined as:
<programlisting> <programlisting>
analyze threshold = analyze base threshold + analyze scale factor * number of tuples analyze threshold = analyze base threshold + analyze scale factor * number of tuples
</programlisting> </programlisting>
...@@ -560,14 +597,14 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu ...@@ -560,14 +597,14 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu
The next two parameters, the vacuum cost delay The next two parameters, the vacuum cost delay
(<structname>pg_autovacuum</structname>.<structfield>vac_cost_delay</structfield>) (<structname>pg_autovacuum</structname>.<structfield>vac_cost_delay</structfield>)
and the vacuum cost limit and the vacuum cost limit
(<structname>pg_autovacuum</structname>.<structfield>vac_cost_limit</structfield>), (<structname>pg_autovacuum</structname>.<structfield>vac_cost_limit</structfield>),
are used to set table-specific values for the are used to set table-specific values for the
<xref linkend="runtime-config-resource-vacuum-cost" endterm="runtime-config-resource-vacuum-cost-title"> <xref linkend="runtime-config-resource-vacuum-cost" endterm="runtime-config-resource-vacuum-cost-title">
feature. feature.
The last two parameters, The last two parameters,
(<structname>pg_autovacuum</structname>.<structfield>freeze_min_age</structfield>) (<structname>pg_autovacuum</structname>.<structfield>freeze_min_age</structfield>)
and and
(<structname>pg_autovacuum</structname>.<structfield>freeze_max_age</structfield>), (<structname>pg_autovacuum</structname>.<structfield>freeze_max_age</structfield>),
are used to set table-specific values for are used to set table-specific values for
<xref linkend="guc-vacuum-freeze-min-age"> and <xref linkend="guc-vacuum-freeze-min-age"> and
<xref linkend="guc-autovacuum-freeze-max-age"> respectively. <xref linkend="guc-autovacuum-freeze-max-age"> respectively.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment