From d675226e20f7a2b66bf42e2ffb14bb9b48f65302 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Sun, 24 Jul 2005 17:18:52 +0000
Subject: [PATCH] Regenerate badly-obsolete README file.
---
src/test/regress/README | 556 ++++++++++++++++++++--------------------
1 file changed, 277 insertions(+), 279 deletions(-)
diff --git a/src/test/regress/README b/src/test/regress/README
index 2bebafa3d19..dc3000cff38 100644
--- a/src/test/regress/README
+++ b/src/test/regress/README
@@ -1,280 +1,278 @@
- Regression Tests
-The regression tests are a comprehensive set of tests for the SQL
-implementation in PostgreSQL. They test standard SQL operations as well as the
-extended capabilities of PostgreSQL. From PostgreSQL 6.1 onward, the regression
-tests are current for every official release.
-
--------------------------------------------------------------------------------
-
-Running the Tests
-
-The regression test can be run against an already installed and running server,
-or using a temporary installation within the build tree. Furthermore, there is
-a "parallel" and a "sequential" mode for running the tests. The sequential
-method runs each test script in turn, whereas the parallel method starts up
-multiple server processes to run groups of tests in parallel. Parallel testing
-gives confidence that interprocess communication and locking are working
-correctly. For historical reasons, the sequential test is usually run against
-an existing installation and the parallel method against a temporary
-installation, but there are no technical reasons for this.
-
-To run the regression tests after building but before installation, type
-
- gmake check
-
-in the top-level directory. (Or you can change to "src/test/regress" and run
-the command there.) This will first build several auxiliary files, such as some
-sample user-defined trigger functions, and then run the test driver script. At
-the end you should see something like
-
- ======================
- All 93 tests passed.
- ======================
-
-or otherwise a note about which tests failed. See the Section called Test
-Evaluation below for more.
-
-Because this test method runs a temporary server, it will not work when you are
-the root user (since the server will not start as root). If you already did the
-build as root, you do not have to start all over. Instead, make the regression
-test directory writable by some other user, log in as that user, and restart
-the tests. For example
-
- root# chmod -R a+w src/test/regress
- root# chmod -R a+w contrib/spi
- root# su - joeuser
- joeuser$ cd top-level build directory
- joeuser$ gmake check
-
-(The only possible "security risk" here is that other users might be able to
-alter the regression test results behind your back. Use common sense when
-managing user permissions.)
-
-Alternatively, run the tests after installation.
-
-The parallel regression test starts quite a few processes under your user ID.
-Presently, the maximum concurrency is twenty parallel test scripts, which means
-sixty processes: there's a server process, a psql, and usually a shell parent
-process for the psql for each test script. So if your system enforces a per-
-user limit on the number of processes, make sure this limit is at least
-seventy-five or so, else you may get random-seeming failures in the parallel
-test. If you are not in a position to raise the limit, you can cut down the
-degree of parallelism by setting the MAX_CONNECTIONS parameter. For example,
-
- gmake MAX_CONNECTIONS=10 check
-
-runs no more than ten tests concurrently.
-
-On some systems, the default Bourne-compatible shell ("/bin/sh") gets confused
-when it has to manage too many child processes in parallel. This may cause the
-parallel test run to lock up or fail. In such cases, specify a different
-Bourne-compatible shell on the command line, for example:
-
- gmake SHELL=/bin/ksh check
-
-If no non-broken shell is available, you may be able to work around the problem
-by limiting the number of connections, as shown above.
-
-To run the tests after installation, initialize a data area and start the
-server, then type
-
- gmake installcheck
-
-The tests will expect to contact the server at the local host and the default
-port number, unless directed otherwise by PGHOST and PGPORT environment
-variables.
-
--------------------------------------------------------------------------------
-
-Test Evaluation
-
-Some properly installed and fully functional PostgreSQL installations can
-"fail" some of these regression tests due to platform-specific artifacts such
-as varying floating-point representation and time zone support. The tests are
-currently evaluated using a simple "diff" comparison against the outputs
-generated on a reference system, so the results are sensitive to small system
-differences. When a test is reported as "failed", always examine the
-differences between expected and actual results; you may well find that the
-differences are not significant. Nonetheless, we still strive to maintain
-accurate reference files across all supported platforms, so it can be expected
-that all tests pass.
-
-The actual outputs of the regression tests are in files in the "src/test/
-regress/results" directory. The test script uses "diff" to compare each output
-file against the reference outputs stored in the "src/test/regress/expected"
-directory. Any differences are saved for your inspection in "src/test/regress/
-regression.diffs". (Or you can run "diff" yourself, if you prefer.)
-
--------------------------------------------------------------------------------
-
-Error message differences
-
-Some of the regression tests involve intentional invalid input values. Error
-messages can come from either the PostgreSQL code or from the host platform
-system routines. In the latter case, the messages may vary between platforms,
-but should reflect similar information. These differences in messages will
-result in a "failed" regression test that can be validated by inspection.
-
--------------------------------------------------------------------------------
-
-Locale differences
-
-If you run the tests against an already-installed server that was initialized
-with a collation-order locale other than C, then there may be differences due
-to sort order and follow-up failures. The regression test suite is set up to
-handle this problem by providing alternative result files that together are
-known to handle a large number of locales. For example, for the char test, the
-expected file "char.out" handles the C and POSIX locales, and the file
-"char_1.out" handles many other locales. The regression test driver will
-automatically pick the best file to match against when checking for success and
-for computing failure differences. (This means that the regression tests cannot
-detect whether the results are appropriate for the configured locale. The tests
-will simply pick the one result file that works best.)
-
-If for some reason the existing expected files do not cover some locale, you
-can add a new file. The naming scheme is testname_digit.out. The actual digit
-is not significant. Remember that the regression test driver will consider all
-such files to be equally valid test results. If the test results are platform-
-specific, the technique described in the Section called Platform-specific
-comparison files should be used instead.
-
--------------------------------------------------------------------------------
-
-Date and time differences
-
-A few of the queries in the "horology" test will fail if you run the test on
-the day of a daylight-saving time changeover, or the day after one. These
-queries expect that the intervals between midnight yesterday, midnight today
-and midnight tomorrow are exactly twenty-four hours --- which is wrong if
-daylight-saving time went into or out of effect meanwhile.
-
- Note: Because USA daylight-saving time rules are used, this problem
- always occurs on the first Sunday of April, the last Sunday of
- October, and their following Mondays, regardless of when daylight-
- saving time is in effect where you live. Also note that the problem
- appears or disappears at midnight Pacific time (UTC-7 or UTC-8), not
- midnight your local time. Thus the failure may appear late on
- Saturday or persist through much of Tuesday, depending on where you
- live.
-
-Most of the date and time results are dependent on the time zone environment.
-The reference files are generated for time zone PST8PDT (Berkeley, California),
-and there will be apparent failures if the tests are not run with that time
-zone setting. The regression test driver sets environment variable PGTZ to
-PST8PDT, which normally ensures proper results. However, your operating system
-must provide support for the PST8PDT time zone, or the time zone-dependent
-tests will fail. To verify that your machine does have this support, type the
-following:
-
- env TZ=PST8PDT date
-
-The command above should have returned the current system time in the PST8PDT
-time zone. If the PST8PDT time zone is not available, then your system may have
-returned the time in UTC. If the PST8PDT time zone is missing, you can set the
-time zone rules explicitly:
-
- PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
-
-There appear to be some systems that do not accept the recommended syntax for
-explicitly setting the local time zone rules; you may need to use a different
-PGTZ setting on such machines.
-
-Some systems using older time-zone libraries fail to apply daylight-saving
-corrections to dates before 1970, causing pre-1970 PDT times to be displayed in
-PST instead. This will result in localized differences in the test results.
-
--------------------------------------------------------------------------------
-
-Floating-point differences
-
-Some of the tests involve computing 64-bit floating-point numbers (double
-precision) from table columns. Differences in results involving mathematical
-functions of double precision columns have been observed. The float8 and
-geometry tests are particularly prone to small differences across platforms, or
-even with different compiler optimization options. Human eyeball comparison is
-needed to determine the real significance of these differences which are
-usually 10 places to the right of the decimal point.
-
-Some systems display minus zero as -0, while others just show 0.
-
-Some systems signal errors from pow() and exp() differently from the mechanism
-expected by the current PostgreSQL code.
-
--------------------------------------------------------------------------------
-
-Row ordering differences
-
-You might see differences in which the same rows are output in a different
-order than what appears in the expected file. In most cases this is not,
-strictly speaking, a bug. Most of the regression test scripts are not so
-pedantic as to use an ORDER BY for every single SELECT, and so their result row
-orderings are not well-defined according to the letter of the SQL
-specification. In practice, since we are looking at the same queries being
-executed on the same data by the same software, we usually get the same result
-ordering on all platforms, and so the lack of ORDER BY isn't a problem. Some
-queries do exhibit cross-platform ordering differences, however. (Ordering
-differences can also be triggered by non-C locale settings.)
-
-Therefore, if you see an ordering difference, it's not something to worry
-about, unless the query does have an ORDER BY that your result is violating.
-But please report it anyway, so that we can add an ORDER BY to that particular
-query and thereby eliminate the bogus "failure" in future releases.
-
-You might wonder why we don't order all the regression test queries explicitly
-to get rid of this issue once and for all. The reason is that that would make
-the regression tests less useful, not more, since they'd tend to exercise query
-plan types that produce ordered results to the exclusion of those that don't.
-
--------------------------------------------------------------------------------
-
-The "random" test
-
-There is at least one case in the random test script that is intended to
-produce random results. This causes random to fail the regression test once in
-a while (perhaps once in every five to ten trials). Typing
-
- diff results/random.out expected/random.out
-
-should produce only one or a few lines of differences. You need not worry
-unless the random test always fails in repeated attempts. (On the other hand,
-if the random test is *never* reported to fail even in many trials of the
-regression tests, you probably *should* worry.)
-
--------------------------------------------------------------------------------
-
-Platform-specific comparison files
-
-Since some of the tests inherently produce platform-specific results, we have
-provided a way to supply platform-specific result comparison files. Frequently,
-the same variation applies to multiple platforms; rather than supplying a
-separate comparison file for every platform, there is a mapping file that
-defines which comparison file to use. So, to eliminate bogus test "failures"
-for a particular platform, you must choose or make a variant result file, and
-then add a line to the mapping file, which is "src/test/regress/resultmap".
-
-Each line in the mapping file is of the form
-
- testname/platformpattern=comparisonfilename
-
-The test name is just the name of the particular regression test module. The
-platform pattern is a pattern in the style of the Unix tool "expr" (that is, a
-regular expression with an implicit ^ anchor at the start). It is matched
-against the platform name as printed by "config.guess" followed by :gcc or :cc,
-depending on whether you use the GNU compiler or the system's native compiler
-(on systems where there is a difference). The comparison file name is the name
-of the substitute result comparison file.
-
-For example: some systems using older time zone libraries fail to apply
-daylight-saving corrections to dates before 1970, causing pre-1970 PDT times to
-be displayed in PST instead. This causes a few differences in the "horology"
-regression test. Therefore, we provide a variant comparison file, "horology-no-
-DST-before-1970.out", which includes the results to be expected on these
-systems. To silence the bogus "failure" message on HPUX platforms, "resultmap"
-includes
-
- horology/.*-hpux=horology-no-DST-before-1970
-
-which will trigger on any machine for which the output of "config.guess"
-includes -hpux. Other lines in "resultmap" select the variant comparison file
-for other platforms where it's appropriate.
+ Regression Tests
+
+ The regression tests are a comprehensive set of tests for the SQL
+ implementation in PostgreSQL. They test standard SQL operations as
+ well as the extended capabilities of PostgreSQL.
+ _________________________________________________________________
+
+ Running the Tests
+
+ The regression tests can be run against an already installed and
+ running server, or using a temporary installation within the build
+ tree. Furthermore, there is a "parallel" and a "sequential" mode for
+ running the tests. The sequential method runs each test script in
+ turn, whereas the parallel method starts up multiple server processes
+ to run groups of tests in parallel. Parallel testing gives confidence
+ that interprocess communication and locking are working correctly. For
+ historical reasons, the sequential test is usually run against an
+ existing installation and the parallel method against a temporary
+ installation, but there are no technical reasons for this.
+
+ To run the regression tests after building but before installation,
+ type
+gmake check
+
+ in the top-level directory. (Or you can change to "src/test/regress"
+ and run the command there.) This will first build several auxiliary
+ files, such as some sample user-defined trigger functions, and then
+ run the test driver script. At the end you should see something like
+======================
+ All 98 tests passed.
+======================
+
+ or otherwise a note about which tests failed. See the section called
+ Test Evaluation below before assuming that a "failure" represents a
+ serious problem.
+
+ Because this test method runs a temporary server, it will not work
+ when you are the root user (since the server will not start as root).
+ If you already did the build as root, you do not have to start all
+ over. Instead, make the regression test directory writable by some
+ other user, log in as that user, and restart the tests. For example
+root# chmod -R a+w src/test/regress
+root# chmod -R a+w contrib/spi
+root# su - joeuser
+joeuser$ cd top-level build directory
+joeuser$ gmake check
+
+ (The only possible "security risk" here is that other users might be
+ able to alter the regression test results behind your back. Use common
+ sense when managing user permissions.)
+
+ Alternatively, run the tests after installation.
+
+ If you have configured PostgreSQL to install into a location where an
+ older PostgreSQL installation already exists, and you perform gmake
+ check before installing the new version, you may find that the tests
+ fail because the new programs try to use the already-installed shared
+ libraries. (Typical symptoms are complaints about undefined symbols.)
+ If you wish to run the tests before overwriting the old installation,
+ you'll need to build with configure --disable-rpath. It is not
+ recommended that you use this option for the final installation,
+ however.
+
+ The parallel regression test starts quite a few processes under your
+ user ID. Presently, the maximum concurrency is twenty parallel test
+ scripts, which means sixty processes: there's a server process, a
+ psql, and usually a shell parent process for the psql for each test
+ script. So if your system enforces a per-user limit on the number of
+ processes, make sure this limit is at least seventy-five or so, else
+ you may get random-seeming failures in the parallel test. If you are
+ not in a position to raise the limit, you can cut down the degree of
+ parallelism by setting the MAX_CONNECTIONS parameter. For example,
+gmake MAX_CONNECTIONS=10 check
+
+ runs no more than ten tests concurrently.
+
+ On some systems, the default Bourne-compatible shell ("/bin/sh") gets
+ confused when it has to manage too many child processes in parallel.
+ This may cause the parallel test run to lock up or fail. In such
+ cases, specify a different Bourne-compatible shell on the command
+ line, for example:
+gmake SHELL=/bin/ksh check
+
+ If no non-broken shell is available, you may be able to work around
+ the problem by limiting the number of connections, as shown above.
+
+ To run the tests after installation, initialize a data area and start
+ the server, then type
+gmake installcheck
+
+ or for a parallel test
+gmake installcheck-parallel
+
+ The tests will expect to contact the server at the local host and the
+ default port number, unless directed otherwise by PGHOST and PGPORT
+ environment variables.
+ _________________________________________________________________
+
+ Test Evaluation
+
+ Some properly installed and fully functional PostgreSQL installations
+ can "fail" some of these regression tests due to platform-specific
+ artifacts such as varying floating-point representation and message
+ wording. The tests are currently evaluated using a simple "diff"
+ comparison against the outputs generated on a reference system, so the
+ results are sensitive to small system differences. When a test is
+ reported as "failed", always examine the differences between expected
+ and actual results; you may well find that the differences are not
+ significant. Nonetheless, we still strive to maintain accurate
+ reference files across all supported platforms, so it can be expected
+ that all tests pass.
+
+ The actual outputs of the regression tests are in files in the
+ "src/test/regress/results" directory. The test script uses "diff" to
+ compare each output file against the reference outputs stored in the
+ "src/test/regress/expected" directory. Any differences are saved for
+ your inspection in "src/test/regress/regression.diffs". (Or you can
+ run "diff" yourself, if you prefer.)
+
+ If for some reason a particular platform generates a "failure" for a
+ given test, but inspection of the output convinces you that the result
+ is valid, you can add a new comparison file to silence the failure
+ report in future test runs. See the section called Variant Comparison
+ Files for details.
+ _________________________________________________________________
+
+ Error message differences
+
+ Some of the regression tests involve intentional invalid input values.
+ Error messages can come from either the PostgreSQL code or from the
+ host platform system routines. In the latter case, the messages may
+ vary between platforms, but should reflect similar information. These
+ differences in messages will result in a "failed" regression test that
+ can be validated by inspection.
+ _________________________________________________________________
+
+ Locale differences
+
+ If you run the tests against an already-installed server that was
+ initialized with a collation-order locale other than C, then there may
+ be differences due to sort order and follow-up failures. The
+ regression test suite is set up to handle this problem by providing
+ alternative result files that together are known to handle a large
+ number of locales.
+ _________________________________________________________________
+
+ Date and time differences
+
+ Most of the date and time results are dependent on the time zone
+ environment. The reference files are generated for time zone PST8PDT
+ (Berkeley, California), and there will be apparent failures if the
+ tests are not run with that time zone setting. The regression test
+ driver sets environment variable PGTZ to PST8PDT, which normally
+ ensures proper results.
+ _________________________________________________________________
+
+ Floating-point differences
+
+ Some of the tests involve computing 64-bit floating-point numbers
+ (double precision) from table columns. Differences in results
+ involving mathematical functions of double precision columns have been
+ observed. The float8 and geometry tests are particularly prone to
+ small differences across platforms, or even with different compiler
+ optimization options. Human eyeball comparison is needed to determine
+ the real significance of these differences which are usually 10 places
+ to the right of the decimal point.
+
+ Some systems display minus zero as -0, while others just show 0.
+
+ Some systems signal errors from pow() and exp() differently from the
+ mechanism expected by the current PostgreSQL code.
+ _________________________________________________________________
+
+ Row ordering differences
+
+ You might see differences in which the same rows are output in a
+ different order than what appears in the expected file. In most cases
+ this is not, strictly speaking, a bug. Most of the regression test
+ scripts are not so pedantic as to use an ORDER BY for every single
+ SELECT, and so their result row orderings are not well-defined
+ according to the letter of the SQL specification. In practice, since
+ we are looking at the same queries being executed on the same data by
+ the same software, we usually get the same result ordering on all
+ platforms, and so the lack of ORDER BY isn't a problem. Some queries
+ do exhibit cross-platform ordering differences, however. When testing
+ against an already-installed server, ordering differences can also be
+ caused by non-C locale settings or non-default parameter settings,
+ such as custom values of work_mem or the planner cost parameters.
+
+ Therefore, if you see an ordering difference, it's not something to
+ worry about, unless the query does have an ORDER BY that your result
+ is violating. But please report it anyway, so that we can add an ORDER
+ BY to that particular query and thereby eliminate the bogus "failure"
+ in future releases.
+
+ You might wonder why we don't order all the regression test queries
+ explicitly to get rid of this issue once and for all. The reason is
+ that that would make the regression tests less useful, not more, since
+ they'd tend to exercise query plan types that produce ordered results
+ to the exclusion of those that don't.
+ _________________________________________________________________
+
+ The "random" test
+
+ The random test script is intended to produce random results. In rare
+ cases, this causes the random regression test to fail. Typing
+diff results/random.out expected/random.out
+
+ should produce only one or a few lines of differences. You need not
+ worry unless the random test fails repeatedly.
+ _________________________________________________________________
+
+ Variant Comparison Files
+
+ Since some of the tests inherently produce environment-dependent
+ results, we have provided ways to specify alternative "expected"
+ result files. Each regression test can have several comparison files
+ showing possible results on different platforms. There are two
+ independent mechanisms for determining which comparison file is used
+ for each test.
+
+ The first mechanism allows comparison files to be selected for
+ specific platforms. There is a mapping file,
+ "src/test/regress/resultmap", that defines which comparison file to
+ use for each platform. To eliminate bogus test "failures" for a
+ particular platform, you first choose or make a variant result file,
+ and then add a line to the "resultmap" file.
+
+ Each line in the mapping file is of the form
+testname/platformpattern=comparisonfilename
+
+ The test name is just the name of the particular regression test
+ module. The platform pattern is a pattern in the style of the Unix
+ tool "expr" (that is, a regular expression with an implicit ^ anchor
+ at the start). It is matched against the platform name as printed by
+ "config.guess" followed by :gcc or :cc, depending on whether you use
+ the GNU compiler or the system's native compiler (on systems where
+ there is a difference). The comparison file name is the base name of
+ the substitute result comparison file.
+
+ For example: some systems interpret very small floating-point values
+ as zero, rather than reporting an underflow error. This causes a few
+ differences in the "float8" regression test. Therefore, we provide a
+ variant comparison file, "float8-small-is-zero.out", which includes
+ the results to be expected on these systems. To silence the bogus
+ "failure" message on OpenBSD platforms, "resultmap" includes
+float8/i.86-.*-openbsd=float8-small-is-zero
+
+ which will trigger on any machine for which the output of
+ "config.guess" matches i.86-.*-openbsd. Other lines in "resultmap"
+ select the variant comparison file for other platforms where it's
+ appropriate.
+
+ The second selection mechanism for variant comparison files is much
+ more automatic: it simply uses the "best match" among several supplied
+ comparison files. The regression test driver script considers both the
+ standard comparison file for a test, testname.out, and variant files
+ named testname_digit.out (where the "digit" is any single digit 0-9).
+ If any such file is an exact match, the test is considered to pass;
+ otherwise, the one that generates the shortest diff is used to create
+ the failure report. (If "resultmap" includes an entry for the
+ particular test, then the base "testname" is the substitute name given
+ in "resultmap".)
+
+ For example, for the char test, the comparison file "char.out"
+ contains results that are expected in the C and POSIX locales, while
+ the file "char_1.out" contains results sorted as they appear in many
+ other locales.
+
+ The best-match mechanism was devised to cope with locale-dependent
+ results, but it can be used in any situation where the test results
+ cannot be predicted easily from the platform name alone. A limitation
+ of this mechanism is that the test driver cannot tell which variant is
+ actually "correct" for the current environment; it will just pick the
+ variant that seems to work best. Therefore it is safest to use this
+ mechanism only for variant results that you are willing to consider
+ equally valid in all contexts.
--
GitLab