Skip to content
Snippets Groups Projects
Commit 8daeb5dd authored by Tom Lane's avatar Tom Lane
Browse files

Add SP-GiST (space-partitioned GiST) index access method.

SP-GiST is comparable to GiST in flexibility, but supports non-balanced
partitioned search structures rather than balanced trees.  As described at
PGCon 2011, this new indexing structure can beat GiST in both index build
time and query speed for search problems that it is well matched to.

There are a number of areas that could still use improvement, but at this
point the code seems committable.

Teodor Sigaev and Oleg Bartunov, with considerable revisions by Tom Lane
parent 19fc0fe3
Branches
Tags
No related merge requests found
Showing
with 4722 additions and 53 deletions
......@@ -569,6 +569,15 @@
</listitem>
</varlistentry>
<varlistentry>
<term><acronym>SP-GiST</acronym></term>
<listitem>
<para>
<link linkend="SPGiST">Space-Partitioned Generalized Search Tree</link>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><acronym>SQL</acronym></term>
<listitem>
......
......@@ -82,6 +82,7 @@
<!ENTITY catalogs SYSTEM "catalogs.sgml">
<!ENTITY geqo SYSTEM "geqo.sgml">
<!ENTITY gist SYSTEM "gist.sgml">
<!ENTITY spgist SYSTEM "spgist.sgml">
<!ENTITY gin SYSTEM "gin.sgml">
<!ENTITY planstats SYSTEM "planstats.sgml">
<!ENTITY indexam SYSTEM "indexam.sgml">
......
......@@ -116,7 +116,7 @@ CREATE INDEX test1_id_index ON test1 (id);
<para>
<productname>PostgreSQL</productname> provides several index types:
B-tree, Hash, GiST and GIN. Each index type uses a different
B-tree, Hash, GiST, SP-GiST and GIN. Each index type uses a different
algorithm that is best suited to different types of queries.
By default, the <command>CREATE INDEX</command> command creates
B-tree indexes, which fit the most common situations.
......@@ -253,6 +253,37 @@ SELECT * FROM places ORDER BY location <-> point '(101,456)' LIMIT 10;
to do this is again dependent on the particular operator class being used.
</para>
<para>
<indexterm>
<primary>index</primary>
<secondary>SP-GiST</secondary>
</indexterm>
<indexterm>
<primary>SP-GiST</primary>
<see>index</see>
</indexterm>
SP-GiST indexes, like GiST indexes, offer an infrastructure that supports
various kinds of searches. SP-GiST permits implementation of a wide range
of different non-balanced disk-based data structures, such as quadtrees,
k-d trees, and suffix trees (tries). As an example, the standard distribution of
<productname>PostgreSQL</productname> includes SP-GiST operator classes
for two-dimensional points, which support indexed
queries using these operators:
<simplelist>
<member><literal>&lt;&lt;</literal></member>
<member><literal>&gt;&gt;</literal></member>
<member><literal>~=</literal></member>
<member><literal>&lt;@</literal></member>
<member><literal>&lt;^</literal></member>
<member><literal>&gt;^</literal></member>
</simplelist>
(See <xref linkend="functions-geometry"> for the meaning of
these operators.)
For more information see <xref linkend="SPGiST">.
</para>
<para>
<indexterm>
<primary>index</primary>
......@@ -263,7 +294,7 @@ SELECT * FROM places ORDER BY location <-> point '(101,456)' LIMIT 10;
<see>index</see>
</indexterm>
GIN indexes are inverted indexes which can handle values that contain more
than one key, arrays for example. Like GiST, GIN can support
than one key, arrays for example. Like GiST and SP-GiST, GIN can support
many different user-defined indexing strategies and the particular
operators with which a GIN index can be used vary depending on the
indexing strategy.
......
......@@ -1460,7 +1460,7 @@ SELECT pg_advisory_lock(q.id) FROM
<variablelist>
<varlistentry>
<term>
B-tree and <acronym>GiST</acronym> indexes
B-tree, <acronym>GiST</acronym> and <acronym>SP-GiST</acronym> indexes
</term>
<listitem>
<para>
......@@ -1510,8 +1510,8 @@ SELECT pg_advisory_lock(q.id) FROM
applications; since they also have more features than hash
indexes, they are the recommended index type for concurrent
applications that need to index scalar data. When dealing with
non-scalar data, B-trees are not useful, and GiST or GIN indexes should
be used instead.
non-scalar data, B-trees are not useful, and GiST, SP-GiST or GIN
indexes should be used instead.
</para>
</sect1>
</chapter>
......@@ -242,6 +242,7 @@
&geqo;
&indexam;
&gist;
&spgist;
&gin;
&storage;
&bki;
......
......@@ -144,8 +144,8 @@ ALTER OPERATOR FAMILY <replaceable>name</replaceable> USING <replaceable class="
and hash functions it is not necessary to specify <replaceable
class="parameter">op_type</replaceable> since the function's input
data type(s) are always the correct ones to use. For B-tree sort
support functions and all functions in GiST and GIN operator classes,
it is necessary to specify the operand data type(s) the function
support functions and all functions in GiST, SP-GiST and GIN operator
classes, it is necessary to specify the operand data type(s) the function
is to be used with.
</para>
......@@ -245,8 +245,8 @@ ALTER OPERATOR FAMILY <replaceable>name</replaceable> USING <replaceable class="
type(s). The name of the operator or function occupying the slot is not
mentioned. Also, for <literal>DROP FUNCTION</> the type(s) to specify
are the input data type(s) the function is intended to support; for
GIN and GiST indexes this might have nothing to do with the actual input
argument types of the function.
GiST, SP-GiST and GIN indexes this might have nothing to do with the actual
input argument types of the function.
</para>
<para>
......
......@@ -57,7 +57,7 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
<para>
<productname>PostgreSQL</productname> provides the index methods
B-tree, hash, GiST, and GIN. Users can also define their own index
B-tree, hash, GiST, SP-GiST, and GIN. Users can also define their own index
methods, but that is fairly complicated.
</para>
......@@ -154,8 +154,8 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
<para>
The name of the index method to be used. Choices are
<literal>btree</literal>, <literal>hash</literal>,
<literal>gist</literal>, and <literal>gin</>. The
default method is <literal>btree</literal>.
<literal>gist</literal>, <literal>spgist</> and <literal>gin</>.
The default method is <literal>btree</literal>.
</para>
</listitem>
</varlistentry>
......@@ -281,12 +281,11 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
<para>
The optional <literal>WITH</> clause specifies <firstterm>storage
parameters</> for the index. Each index method has its own set of allowed
storage parameters. The B-tree, hash and GiST index methods all accept a
single parameter:
storage parameters. The B-tree, hash, GiST and SP-GiST index methods all
accept this parameter:
</para>
<variablelist>
<varlistentry>
<term><literal>FILLFACTOR</></term>
<listitem>
......@@ -307,7 +306,25 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
GiST indexes additionally accept this parameter:
</para>
<variablelist>
<varlistentry>
<term><literal>BUFFERING</></term>
<listitem>
<para>
Determines whether the buffering build technique described in
<xref linkend="gist-buffering-build"> is used to build the index. With
<literal>OFF</> it is disabled, with <literal>ON</> it is enabled, and
with <literal>AUTO</> it is initially disabled, but turned on
on-the-fly once the index size reaches <xref linkend="guc-effective-cache-size">. The default is <literal>AUTO</>.
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
......@@ -315,7 +332,6 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
</para>
<variablelist>
<varlistentry>
<term><literal>FASTUPDATE</></term>
<listitem>
......@@ -339,27 +355,6 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ <replaceable class="parameter">name</
</note>
</listitem>
</varlistentry>
</variablelist>
<para>
GiST indexes additionally accept parameter:
</para>
<variablelist>
<varlistentry>
<term><literal>BUFFERING</></term>
<listitem>
<para>
Determines whether the buffering build technique described in
<xref linkend="gist-buffering-build"> is used to build the index. With
<literal>OFF</> it is disabled, with <literal>ON</> it is enabled, and
with <literal>AUTO</> it is initially disabled, but turned on
on-the-fly once the index size reaches <xref linkend="guc-effective-cache-size">. The default is <literal>AUTO</>.
</para>
</listitem>
</varlistentry>
</variablelist>
</refsect2>
......
......@@ -172,7 +172,7 @@ CREATE OPERATOR CLASS <replaceable class="parameter">name</replaceable> [ DEFAUL
the input data type(s) of the function (for B-tree comparison functions
and hash functions)
or the class's data type (for B-tree sort support functions and all
functions in GiST and GIN operator classes). These defaults
functions in GiST, SP-GiST and GIN operator classes). These defaults
are correct, and so <replaceable
class="parameter">op_type</replaceable> need not be specified in
<literal>FUNCTION</> clauses, except for the case of a B-tree sort
......@@ -232,7 +232,7 @@ CREATE OPERATOR CLASS <replaceable class="parameter">name</replaceable> [ DEFAUL
<para>
The data type actually stored in the index. Normally this is
the same as the column data type, but some index methods
(currently GIN and GiST) allow it to be different. The
(currently GiST and GIN) allow it to be different. The
<literal>STORAGE</> clause must be omitted unless the index
method allows a different type to be used.
</para>
......
......@@ -540,7 +540,8 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI
cannot be used. Although it's allowed, there is little point in using
B-tree or hash indexes with an exclusion constraint, because this
does nothing that an ordinary unique constraint doesn't do better.
So in practice the access method will always be <acronym>GiST</>.
So in practice the access method will always be <acronym>GiST</> or
<acronym>SP-GiST</>.
</para>
<para>
......
<!-- doc/src/sgml/spgist.sgml -->
<chapter id="SPGiST">
<title>SP-GiST Indexes</title>
<indexterm>
<primary>index</primary>
<secondary>SP-GiST</secondary>
</indexterm>
<sect1 id="spgist-intro">
<title>Introduction</title>
<para>
<acronym>SP-GiST</acronym> is an abbreviation for space-partitioned
<acronym>GiST</acronym>. <acronym>SP-GiST</acronym> supports partitioned
search trees, which facilitate development of a wide range of different
non-balanced data structures, such as quad-trees, k-d trees, and suffix
trees (tries). The common feature of these structures is that they
repeatedly divide the search space into partitions that need not be
of equal size. Searches that are well matched to the partitioning rule
can be very fast.
</para>
<para>
These popular data structures were originally developed for in-memory
usage. In main memory, they are usually designed as a set of dynamically
allocated nodes linked by pointers. This is not suitable for direct
storing on disk, since these chains of pointers can be rather long which
would require too many disk accesses. In contrast, disk-based data
structures should have a high fanout to minimize I/O. The challenge
addressed by <acronym>SP-GiST</acronym> is to map search tree nodes to
disk pages in such a way that a search need access only a few disk pages,
even if it traverses many nodes.
</para>
<para>
Like <acronym>GiST</acronym>, <acronym>SP-GiST</acronym> is meant to allow
the development of custom data types with the appropriate access methods,
by an expert in the domain of the data type, rather than a database expert.
</para>
<para>
Some of the information here is derived from Purdue University's
SP-GiST Indexing Project
<ulink url="http://www.cs.purdue.edu/spgist/">web site</ulink>.
The <acronym>SP-GiST</acronym> implementation in
<productname>PostgreSQL</productname> is primarily maintained by Teodor
Sigaev and Oleg Bartunov, and there is more information on their
<!-- URL will be changed -->
<ulink url="http://www.sai.msu.su/~megera/wiki/spgist_dev">web site</ulink>.
</para>
</sect1>
<sect1 id="spgist-extensibility">
<title>Extensibility</title>
<para>
<acronym>SP-GiST</acronym> offers an interface with a high level of
abstraction, requiring the access method developer to implement only
methods specific to a given data type. The <acronym>SP-GiST</acronym> core
is responsible for efficient disk mapping and searching the tree structure.
It also takes care of concurrency and logging considerations.
</para>
<para>
Leaf tuples of an <acronym>SP-GiST</acronym> tree contain values of the
same data type as the indexed column. Leaf tuples at the root level will
always contain the original indexed data value, but leaf tuples at lower
levels might contain only a compressed representation, such as a suffix.
In that case the operator class support functions must be able to
reconstruct the original value using information accumulated from the
inner tuples that are passed through to reach the leaf level.
</para>
<para>
Inner tuples are more complex, since they are branching points in the
search tree. Each inner tuple contains a set of one or more
<firstterm>nodes</>, which represent groups of similar leaf values.
A node contains a downlink that leads to either another, lower-level inner
tuple, or a short list of leaf tuples that all lie on the same index page.
Each node has a <firstterm>label</> that describes it; for example,
in a suffix tree the node label could be the next character of the string
value. Optionally, an inner tuple can have a <firstterm>prefix</> value
that describes all its members. In a suffix tree this could be the common
prefix of the represented strings. The prefix value is not necessarily
really a prefix, but can be any data needed by the operator class;
for example, in a quad-tree it can store the central point that the four
quadrants are measured with respect to. A quad-tree inner tuple would
then also contain four nodes corresponding to the quadrants around this
central point.
</para>
<para>
Some tree algorithms require knowledge of level (or depth) of the current
tuple, so the <acronym>SP-GiST</acronym> core provides the possibility for
operator classes to manage level counting while descending the tree.
There is also support for incrementally reconstructing the represented
value when that is needed.
</para>
<para>
There are five user-defined methods that an index operator class for
<acronym>SP-GiST</acronym> must provide. All five follow the convention
of accepting two <type>internal</> arguments, the first of which is a
pointer to a C struct containing input values for the support method,
while the second argument is a pointer to a C struct where output values
must be placed. Four of the methods just return <type>void</>, since
all their results appear in the output struct; but
<function>leaf_consistent</> additionally returns a <type>boolean</> result.
The methods must not modify any fields of their input structs. In all
cases, the output struct is initialized to zeroes before calling the
user-defined method.
</para>
<para>
The five user-defined methods are:
</para>
<variablelist>
<varlistentry>
<term><function>config</></term>
<listitem>
<para>
Returns static information about the index implementation, including
the datatype OIDs of the prefix and node label data types.
</para>
<para>
The <acronym>SQL</> declaration of the function must look like this:
<programlisting>
CREATE FUNCTION my_config(internal, internal) RETURNS void ...
</programlisting>
The first argument is a pointer to a <structname>spgConfigIn</>
C struct, containing input data for the function.
The second argument is a pointer to a <structname>spgConfigOut</>
C struct, which the function must fill with result data.
<programlisting>
typedef struct spgConfigIn
{
Oid attType; /* Data type to be indexed */
} spgConfigIn;
typedef struct spgConfigOut
{
Oid prefixType; /* Data type of inner-tuple prefixes */
Oid labelType; /* Data type of inner-tuple node labels */
bool longValuesOK; /* Opclass can cope with values &gt; 1 page */
} spgConfigOut;
</programlisting>
<structfield>attType</> is passed in order to support polymorphic
index operator classes; for ordinary fixed-data-type opclasses, it
will always have the same value and so can be ignored.
</para>
<para>
For operator classes that do not use prefixes,
<structfield>prefixType</> can be set to <literal>VOIDOID</>.
Likewise, for operator classes that do not use node labels,
<structfield>labelType</> can be set to <literal>VOIDOID</>.
<structfield>longValuesOK</> should be set true only when the
<structfield>attType</> is of variable length and the operator
class is capable of segmenting long values by repeated suffixing
(see <xref linkend="spgist-limits">).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>choose</></term>
<listitem>
<para>
Chooses a method for inserting a new value into an inner tuple.
</para>
<para>
The <acronym>SQL</> declaration of the function must look like this:
<programlisting>
CREATE FUNCTION my_choose(internal, internal) RETURNS void ...
</programlisting>
The first argument is a pointer to a <structname>spgChooseIn</>
C struct, containing input data for the function.
The second argument is a pointer to a <structname>spgChooseOut</>
C struct, which the function must fill with result data.
<programlisting>
typedef struct spgChooseIn
{
Datum datum; /* original datum to be indexed */
Datum leafDatum; /* current datum to be stored at leaf */
int level; /* current level (counting from zero) */
/* Data from current inner tuple */
bool allTheSame; /* tuple is marked all-the-same? */
bool hasPrefix; /* tuple has a prefix? */
Datum prefixDatum; /* if so, the prefix value */
int nNodes; /* number of nodes in the inner tuple */
Datum *nodeLabels; /* node label values (NULL if none) */
} spgChooseIn;
typedef enum spgChooseResultType
{
spgMatchNode = 1, /* descend into existing node */
spgAddNode, /* add a node to the inner tuple */
spgSplitTuple /* split inner tuple (change its prefix) */
} spgChooseResultType;
typedef struct spgChooseOut
{
spgChooseResultType resultType; /* action code, see above */
union
{
struct /* results for spgMatchNode */
{
int nodeN; /* descend to this node (index from 0) */
int levelAdd; /* increment level by this much */
Datum restDatum; /* new leaf datum */
} matchNode;
struct /* results for spgAddNode */
{
Datum nodeLabel; /* new node's label */
int nodeN; /* where to insert it (index from 0) */
} addNode;
struct /* results for spgSplitTuple */
{
/* Info to form new inner tuple with one node */
bool prefixHasPrefix; /* tuple should have a prefix? */
Datum prefixPrefixDatum; /* if so, its value */
Datum nodeLabel; /* node's label */
/* Info to form new lower-level inner tuple with all old nodes */
bool postfixHasPrefix; /* tuple should have a prefix? */
Datum postfixPrefixDatum; /* if so, its value */
} splitTuple;
} result;
} spgChooseOut;
</programlisting>
<structfield>datum</> is the original datum that was to be inserted
into the index.
<structfield>leafDatum</> is initially the same as
<structfield>datum</>, but can change at lower levels of the tree
if the <function>choose</function> or <function>picksplit</function>
methods change it. When the insertion search reaches a leaf page,
the current value of <structfield>leafDatum</> is what will be stored
in the newly created leaf tuple.
<structfield>level</> is the current inner tuple's level, starting at
zero for the root level.
<structfield>allTheSame</> is true if the current inner tuple is
marked as containing multiple equivalent nodes
(see <xref linkend="spgist-all-the-same">).
<structfield>hasPrefix</> is true if the current inner tuple contains
a prefix; if so,
<structfield>prefixDatum</> is its value.
<structfield>nNodes</> is the number of child nodes contained in the
inner tuple, and
<structfield>nodeLabels</> is an array of their label values, or
NULL if there are no labels.
</para>
<para>
The <function>choose</function> function can determine either that
the new value matches one of the existing child nodes, or that a new
child node must be added, or that the new value is inconsistent with
the tuple prefix and so the inner tuple must be split to create a
less restrictive prefix.
</para>
<para>
If the new value matches one of the existing child nodes,
set <structfield>resultType</> to <literal>spgMatchNode</>.
Set <structfield>nodeN</> to the index (from zero) of that node in
the node array.
Set <structfield>levelAdd</> to the increment in
<structfield>level</> caused by descending through that node,
or leave it as zero if the operator class does not use levels.
Set <structfield>restDatum</> to equal <structfield>datum</>
if the operator class does not modify datums from one level to the
next, or otherwise set it to the modified value to be used as
<structfield>leafDatum</> at the next level.
</para>
<para>
If a new child node must be added,
set <structfield>resultType</> to <literal>spgAddNode</>.
Set <structfield>nodeLabel</> to the label to be used for the new
node, and set <structfield>nodeN</> to the index (from zero) at which
to insert the node in the node array.
After the node has been added, the <function>choose</function>
function will be called again with the modified inner tuple;
that call should result in an <literal>spgMatchNode</> result.
</para>
<para>
If the new value is inconsistent with the tuple prefix,
set <structfield>resultType</> to <literal>spgSplitTuple</>.
This action moves all the existing nodes into a new lower-level
inner tuple, and replaces the existing inner tuple with a tuple
having a single node that links to the new lower-level inner tuple.
Set <structfield>prefixHasPrefix</> to indicate whether the new
upper tuple should have a prefix, and if so set
<structfield>prefixPrefixDatum</> to the prefix value. This new
prefix value must be sufficiently less restrictive than the original
to accept the new value to be indexed, and it should be no longer
than the original prefix.
Set <structfield>nodeLabel</> to the label to be used for the
node that will point to the new lower-level inner tuple.
Set <structfield>postfixHasPrefix</> to indicate whether the new
lower-level inner tuple should have a prefix, and if so set
<structfield>postfixPrefixDatum</> to the prefix value. The
combination of these two prefixes and the additional label must
have the same meaning as the original prefix, because there is
no opportunity to alter the node labels that are moved to the new
lower-level tuple, nor to change any child index entries.
After the node has been split, the <function>choose</function>
function will be called again with the replacement inner tuple.
That call will usually result in an <literal>spgAddNode</> result,
since presumably the node label added in the split step will not
match the new value; so after that, there will be a third call
that finally returns <literal>spgMatchNode</> and allows the
insertion to descend to the leaf level.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>picksplit</></term>
<listitem>
<para>
Decides how to create a new inner tuple over a set of leaf tuples.
</para>
<para>
The <acronym>SQL</> declaration of the function must look like this:
<programlisting>
CREATE FUNCTION my_picksplit(internal, internal) RETURNS void ...
</programlisting>
The first argument is a pointer to a <structname>spgPickSplitIn</>
C struct, containing input data for the function.
The second argument is a pointer to a <structname>spgPickSplitOut</>
C struct, which the function must fill with result data.
<programlisting>
typedef struct spgPickSplitIn
{
int nTuples; /* number of leaf tuples */
Datum *datums; /* their datums (array of length nTuples) */
int level; /* current level (counting from zero) */
} spgPickSplitIn;
typedef struct spgPickSplitOut
{
bool hasPrefix; /* new inner tuple should have a prefix? */
Datum prefixDatum; /* if so, its value */
int nNodes; /* number of nodes for new inner tuple */
Datum *nodeLabels; /* their labels (or NULL for no labels) */
int *mapTuplesToNodes; /* node index for each leaf tuple */
Datum *leafTupleDatums; /* datum to store in each new leaf tuple */
} spgPickSplitOut;
</programlisting>
<structfield>nTuples</> is the number of leaf tuples provided.
<structfield>datums</> is an array of their datum values.
<structfield>level</> is the current level that all the leaf tuples
share, which will become the level of the new inner tuple.
</para>
<para>
Set <structfield>hasPrefix</> to indicate whether the new inner
tuple should have a prefix, and if so set
<structfield>prefixDatum</> to the prefix value.
Set <structfield>nNodes</> to indicate the number of nodes that
the new inner tuple will contain, and
set <structfield>nodeLabels</> to an array of their label values.
(If the nodes do not require labels, set <structfield>nodeLabels</>
to NULL; see <xref linkend="spgist-null-labels"> for details.)
Set <structfield>mapTuplesToNodes</> to an array that gives the index
(from zero) of the node that each leaf tuple should be assigned to.
Set <structfield>leafTupleDatums</> to an array of the values to
be stored in the new leaf tuples (these will be the same as the
input <structfield>datums</> if the operator class does not modify
datums from one level to the next).
Note that the <function>picksplit</> function is
responsible for palloc'ing the
<structfield>nodeLabels</>, <structfield>mapTuplesToNodes</> and
<structfield>leafTupleDatums</> arrays.
</para>
<para>
If more than one leaf tuple is supplied, it is expected that the
<function>picksplit</> function will classify them into more than
one node; otherwise it is not possible to split the leaf tuples
across multiple pages, which is the ultimate purpose of this
operation. Therefore, if the <function>picksplit</> function
ends up placing all the leaf tuples in the same node, the core
SP-GiST code will override that decision and generate an inner
tuple in which the leaf tuples are assigned at random to several
identically-labeled nodes. Such a tuple is marked
<literal>allTheSame</> to signify that this has happened. The
<function>choose</> and <function>inner_consistent</> functions
must take suitable care with such inner tuples.
See <xref linkend="spgist-all-the-same"> for more information.
</para>
<para>
<function>picksplit</> can be applied to a single leaf tuple only
in the case that the <function>config</> function set
<structfield>longValuesOK</> to true and a larger-than-a-page input
value has been supplied. In this case the point of the operation is
to strip off a prefix and produce a new, shorter leaf datum value.
The call will be repeated until a leaf datum short enough to fit on
a page has been produced. See <xref linkend="spgist-limits"> for
more information.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>inner_consistent</></term>
<listitem>
<para>
Returns set of nodes (branches) to follow during tree search.
</para>
<para>
The <acronym>SQL</> declaration of the function must look like this:
<programlisting>
CREATE FUNCTION my_inner_consistent(internal, internal) RETURNS void ...
</programlisting>
The first argument is a pointer to a <structname>spgInnerConsistentIn</>
C struct, containing input data for the function.
The second argument is a pointer to a <structname>spgInnerConsistentOut</>
C struct, which the function must fill with result data.
<programlisting>
typedef struct spgInnerConsistentIn
{
StrategyNumber strategy; /* operator strategy number */
Datum query; /* operator's RHS value */
Datum reconstructedValue; /* value reconstructed at parent */
int level; /* current level (counting from zero) */
/* Data from current inner tuple */
bool allTheSame; /* tuple is marked all-the-same? */
bool hasPrefix; /* tuple has a prefix? */
Datum prefixDatum; /* if so, the prefix value */
int nNodes; /* number of nodes in the inner tuple */
Datum *nodeLabels; /* node label values (NULL if none) */
} spgInnerConsistentIn;
typedef struct spgInnerConsistentOut
{
int nNodes; /* number of child nodes to be visited */
int *nodeNumbers; /* their indexes in the node array */
int *levelAdds; /* increment level by this much for each */
Datum *reconstructedValues; /* associated reconstructed values */
} spgInnerConsistentOut;
</programlisting>
<structfield>strategy</> and
<structfield>query</> describe the index search condition.
<structfield>reconstructedValue</> is the value reconstructed for the
parent tuple; it is <literal>(Datum) 0</> at the root level or if the
<function>inner_consistent</> function did not provide a value at the
parent level.
<structfield>level</> is the current inner tuple's level, starting at
zero for the root level.
<structfield>allTheSame</> is true if the current inner tuple is
marked <quote>all-the-same</>; in this case all the nodes have the
same label (if any) and so either all or none of them match the query
(see <xref linkend="spgist-all-the-same">).
<structfield>hasPrefix</> is true if the current inner tuple contains
a prefix; if so,
<structfield>prefixDatum</> is its value.
<structfield>nNodes</> is the number of child nodes contained in the
inner tuple, and
<structfield>nodeLabels</> is an array of their label values, or
NULL if the nodes do not have labels.
</para>
<para>
<structfield>nNodes</> must be set to the number of child nodes that
need to be visited by the search, and
<structfield>nodeNumbers</> must be set to an array of their indexes.
If the operator class keeps track of levels, set
<structfield>levelAdds</> to an array of the level increments
required when descending to each node to be visited. (Often these
increments will be the same for all the nodes, but that's not
necessarily so, so an array is used.)
If value reconstruction is needed, set
<structfield>reconstructedValues</> to an array of the values
reconstructed for each child node to be visited; otherwise, leave
<structfield>reconstructedValues</> as NULL.
Note that the <function>inner_consistent</> function is
responsible for palloc'ing the
<structfield>nodeNumbers</>, <structfield>levelAdds</> and
<structfield>reconstructedValues</> arrays.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>leaf_consistent</></term>
<listitem>
<para>
Returns true if a leaf tuple satisfies a query.
</para>
<para>
The <acronym>SQL</> declaration of the function must look like this:
<programlisting>
CREATE FUNCTION my_leaf_consistent(internal, internal) RETURNS bool ...
</programlisting>
The first argument is a pointer to a <structname>spgLeafConsistentIn</>
C struct, containing input data for the function.
The second argument is a pointer to a <structname>spgLeafConsistentOut</>
C struct, which the function must fill with result data.
<programlisting>
typedef struct spgLeafConsistentIn
{
StrategyNumber strategy; /* operator strategy number */
Datum query; /* operator's RHS value */
Datum reconstructedValue; /* value reconstructed at parent */
int level; /* current level (counting from zero) */
Datum leafDatum; /* datum in leaf tuple */
} spgLeafConsistentIn;
typedef struct spgLeafConsistentOut
{
bool recheck; /* set true if operator must be rechecked */
} spgLeafConsistentOut;
</programlisting>
<structfield>strategy</> and
<structfield>query</> define the index search condition.
<structfield>reconstructedValue</> is the value reconstructed for the
parent tuple; it is <literal>(Datum) 0</> at the root level or if the
<function>inner_consistent</> function did not provide a value at the
parent level.
<structfield>level</> is the current leaf tuple's level, starting at
zero for the root level.
<structfield>leafDatum</> is the key value stored in the current
leaf tuple.
</para>
<para>
The function must return <literal>true</> if the leaf tuple matches the
query, or <literal>false</> if not. In the <literal>true</> case,
<structfield>recheck</> may be set to <literal>true</> if the match
is uncertain and so the operator must be re-applied to the actual heap
tuple to verify the match.
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
All the SP-GiST support methods are normally called in a short-lived
memory context; that is, <varname>CurrentMemoryContext</> will be reset
after processing of each tuple. It is therefore not very important to
worry about pfree'ing everything you palloc. (The <function>config</>
method is an exception: it should try to avoid leaking memory. But
usually the <function>config</> method need do nothing but assign
constants into the passed parameter struct.)
</para>
<para>
If the indexed column is of a collatable data type, the index collation
will be passed to all the support methods, using the standard
<function>PG_GET_COLLATION()</> mechanism.
</para>
</sect1>
<sect1 id="spgist-implementation">
<title>Implementation</title>
<para>
This section covers implementation details and other tricks that are
useful for implementors of <acronym>SP-GiST</acronym> operator classes to
know.
</para>
<sect2 id="spgist-limits">
<title>SP-GiST Limits</title>
<para>
Individual leaf tuples and inner tuples must fit on a single index page
(8KB by default). Therefore, when indexing values of variable-length
data types, long values can only be supported by methods such as suffix
trees, in which each level of the tree includes a prefix that is short
enough to fit on a page, and the final leaf level includes a suffix also
short enough to fit on a page. The operator class should set
<structfield>longValuesOK</> to TRUE only if it is prepared to arrange for
this to happen. Otherwise, the <acronym>SP-GiST</acronym> core will
reject any request to index a value that is too large to fit
on an index page.
</para>
<para>
Likewise, it is the operator class's responsibility that inner tuples
do not grow too large to fit on an index page; this limits the number
of child nodes that can be used in one inner tuple, as well as the
maximum size of a prefix value.
</para>
<para>
Another limitation is that when an inner tuple's node points to a set
of leaf tuples, those tuples must all be in the same index page.
(This is a design decision to reduce seeking and save space in the
links that chain such tuples together.) If the set of leaf tuples
grows too large for a page, a split is performed and an intermediate
inner tuple is inserted. For this to fix the problem, the new inner
tuple <emphasis>must</> divide the set of leaf values into more than one
node group. If the operator class's <function>picksplit</> function
fails to do that, the <acronym>SP-GiST</acronym> core resorts to
extraordinary measures described in <xref linkend="spgist-all-the-same">.
</para>
</sect2>
<sect2 id="spgist-null-labels">
<title>SP-GiST Without Node Labels</title>
<para>
Some tree algorithms use a fixed set of nodes for each inner tuple;
for example, in a quad-tree there are always exactly four nodes
corresponding to the four quadrants around the inner tuple's centroid
point. In such a case the code typically works with the nodes by
number, and there is no need for explicit node labels. To suppress
node labels (and thereby save some space), the <function>picksplit</>
function can return NULL for the <structfield>nodeLabels</> array.
This will in turn result in <structfield>nodeLabels</> being NULL during
subsequent calls to <function>choose</> and <function>inner_consistent</>.
In principle, node labels could be used for some inner tuples and omitted
for others in the same index.
</para>
<para>
When working with an inner tuple having unlabeled nodes, it is an error
for <function>choose</> to return <literal>spgAddNode</>, since the set
of nodes is supposed to be fixed in such cases. Also, there is no
provision for generating an unlabeled node in <literal>spgSplitTuple</>
actions, since it is expected that an <literal>spgAddNode</> action will
be needed as well.
</para>
</sect2>
<sect2 id="spgist-all-the-same">
<title><quote>All-the-same</> Inner Tuples</title>
<para>
The <acronym>SP-GiST</acronym> core can override the results of the
operator class's <function>picksplit</> function when
<function>picksplit</> fails to divide the supplied leaf values into
at least two node categories. When this happens, the new inner tuple
is created with multiple nodes that each have the same label (if any)
that <function>picksplit</> gave to the one node it did use, and the
leaf values are divided at random among these equivalent nodes.
The <literal>allTheSame</> flag is set on the inner tuple to warn the
<function>choose</> and <function>inner_consistent</> functions that the
tuple does not have the node set that they might otherwise expect.
</para>
<para>
When dealing with an <literal>allTheSame</> tuple, a <function>choose</>
result of <literal>spgMatchNode</> is interpreted to mean that the new
value can be assigned to any of the equivalent nodes; the core code will
ignore the supplied <structfield>nodeN</> value and descend into one
of the nodes at random (so as to keep the tree balanced). It is an
error for <function>choose</> to return <literal>spgAddNode</>, since
that would make the nodes not all equivalent; the
<literal>spgSplitTuple</> action must be used if the value to be inserted
doesn't match the existing nodes.
</para>
<para>
When dealing with an <literal>allTheSame</> tuple, the
<function>inner_consistent</> function should return either all or none
of the nodes as targets for continuing the index search, since they are
all equivalent. This may or may not require any special-case code,
depending on how much the <function>inner_consistent</> function normally
assumes about the meaning of the nodes.
</para>
</sect2>
</sect1>
<sect1 id="spgist-examples">
<title>Examples</title>
<para>
The <productname>PostgreSQL</productname> source distribution includes
several examples of index operator classes for
<acronym>SP-GiST</acronym>. The core system currently provides suffix
trees over text columns and two types of trees over points: quad-tree and
k-d tree. Look into <filename>src/backend/access/spgist/</> to see the
code.
</para>
</sect1>
</chapter>
......@@ -237,12 +237,59 @@
</table>
<para>
GIN indexes are similar to GiST indexes in flexibility: they don't have a
fixed set of strategies. Instead the support routines of each operator
SP-GiST indexes are similar to GiST indexes in flexibility: they don't have
a fixed set of strategies. Instead the support routines of each operator
class interpret the strategy numbers according to the operator class's
definition. As an example, the strategy numbers used by the built-in
operator classes for arrays are
shown in <xref linkend="xindex-gin-array-strat-table">.
operator classes for points are shown in <xref
linkend="xindex-spgist-point-strat-table">.
</para>
<table tocentry="1" id="xindex-spgist-point-strat-table">
<title>SP-GiST Point Strategies</title>
<tgroup cols="2">
<thead>
<row>
<entry>Operation</entry>
<entry>Strategy Number</entry>
</row>
</thead>
<tbody>
<row>
<entry>strictly left of</entry>
<entry>1</entry>
</row>
<row>
<entry>strictly right of</entry>
<entry>5</entry>
</row>
<row>
<entry>same</entry>
<entry>6</entry>
</row>
<row>
<entry>contained by</entry>
<entry>8</entry>
</row>
<row>
<entry>strictly below</entry>
<entry>10</entry>
</row>
<row>
<entry>strictly above</entry>
<entry>11</entry>
</row>
</tbody>
</tgroup>
</table>
<para>
GIN indexes are similar to GiST and SP-GiST indexes, in that they don't
have a fixed set of strategies either. Instead the support routines of
each operator class interpret the strategy numbers according to the
operator class's definition. As an example, the strategy numbers used by
the built-in operator classes for arrays are shown in
<xref linkend="xindex-gin-array-strat-table">.
</para>
<table tocentry="1" id="xindex-gin-array-strat-table">
......@@ -434,6 +481,54 @@
</tgroup>
</table>
<para>
SP-GiST indexes require five support functions, as
shown in <xref linkend="xindex-spgist-support-table">.
(For more information see <xref linkend="SPGiST">.)
</para>
<table tocentry="1" id="xindex-spgist-support-table">
<title>SP-GiST Support Functions</title>
<tgroup cols="3">
<thead>
<row>
<entry>Function</entry>
<entry>Description</entry>
<entry>Support Number</entry>
</row>
</thead>
<tbody>
<row>
<entry><function>config</></entry>
<entry>provide basic information about the operator class</entry>
<entry>1</entry>
</row>
<row>
<entry><function>choose</></entry>
<entry>determine how to insert a new value into an inner tuple</entry>
<entry>2</entry>
</row>
<row>
<entry><function>picksplit</></entry>
<entry>determine how to partition a set of values</entry>
<entry>3</entry>
</row>
<row>
<entry><function>inner_consistent</></entry>
<entry>determine which sub-partitions need to be searched for a
query</entry>
<entry>4</entry>
</row>
<row>
<entry><function>leaf_consistent</></entry>
<entry>determine whether key satisfies the
query qualifier</entry>
<entry>5</entry>
</row>
</tbody>
</tgroup>
</table>
<para>
GIN indexes require four support functions, with an optional fifth, as
shown in <xref linkend="xindex-gin-support-table">.
......@@ -495,9 +590,9 @@
of the comparison function for B-trees, a signed integer. The number
and types of the arguments to each support function are likewise
dependent on the index method. For B-tree and hash the comparison and
hashing support functions
take the same input data types as do the operators included in the operator
class, but this is not the case for most GIN and GiST support functions.
hashing support functions take the same input data types as do the
operators included in the operator class, but this is not the case for
most GiST, SP-GiST, and GIN support functions.
</para>
</sect2>
......@@ -876,9 +971,10 @@ ALTER OPERATOR FAMILY integer_ops USING btree ADD
</para>
<para>
GIN and GiST indexes do not have any explicit notion of cross-data-type
operations. The set of operators supported is just whatever the primary
support functions for a given operator class can handle.
GiST, SP-GiST, and GIN indexes do not have any explicit notion of
cross-data-type operations. The set of operators supported is just
whatever the primary support functions for a given operator class can
handle.
</para>
<note>
......@@ -1045,7 +1141,7 @@ SELECT * FROM table WHERE integer_column &lt; 4;
the index is guaranteed to return all the required rows, plus perhaps
some additional rows, which can be eliminated by performing the original
operator invocation. The index methods that support lossy searches
(currently, GiST and GIN) allow the support functions of individual
(currently, GiST, SP-GiST and GIN) allow the support functions of individual
operator classes to set the recheck flag, and so this is essentially an
operator-class feature.
</para>
......
......@@ -8,6 +8,6 @@ subdir = src/backend/access
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
SUBDIRS = common gist hash heap index nbtree transam gin
SUBDIRS = common gist hash heap index nbtree transam gin spgist
include $(top_srcdir)/src/backend/common.mk
......@@ -19,6 +19,7 @@
#include "access/hash.h"
#include "access/nbtree.h"
#include "access/reloptions.h"
#include "access/spgist.h"
#include "catalog/pg_type.h"
#include "commands/defrem.h"
#include "commands/tablespace.h"
......@@ -104,6 +105,14 @@ static relopt_int intRelOpts[] =
},
GIST_DEFAULT_FILLFACTOR, GIST_MIN_FILLFACTOR, 100
},
{
{
"fillfactor",
"Packs spgist index pages only to this percentage",
RELOPT_KIND_SPGIST
},
SPGIST_DEFAULT_FILLFACTOR, SPGIST_MIN_FILLFACTOR, 100
},
{
{
"autovacuum_vacuum_threshold",
......
#-------------------------------------------------------------------------
#
# Makefile--
# Makefile for access/spgist
#
# IDENTIFICATION
# src/backend/access/spgist/Makefile
#
#-------------------------------------------------------------------------
subdir = src/backend/access/spgist
top_builddir = ../../../..
include $(top_builddir)/src/Makefile.global
OBJS = spgutils.o spginsert.o spgscan.o spgvacuum.o \
spgdoinsert.o spgxlog.o \
spgtextproc.o spgquadtreeproc.o spgkdtreeproc.o
include $(top_srcdir)/src/backend/common.mk
src/backend/access/spgist/README
SP-GiST is an abbreviation of space-partitioned GiST. It provides a
generalized infrastructure for implementing space-partitioned data
structures, such as quadtrees, k-d trees, and suffix trees (tries). When
implemented in main memory, these structures are usually designed as a set of
dynamically-allocated nodes linked by pointers. This is not suitable for
direct storing on disk, since the chains of pointers can be rather long and
require too many disk accesses. In contrast, disk based data structures
should have a high fanout to minimize I/O. The challenge is to map tree
nodes to disk pages in such a way that the search algorithm accesses only a
few disk pages, even if it traverses many nodes.
COMMON STRUCTURE DESCRIPTION
Logically, an SP-GiST tree is a set of tuples, each of which can be either
an inner or leaf tuple. Each inner tuple contains "nodes", which are
(label,pointer) pairs, where the pointer (ItemPointerData) is a pointer to
another inner tuple or to the head of a list of leaf tuples. Inner tuples
can have different numbers of nodes (children). Branches can be of different
depth (actually, there is no control or code to support balancing), which
means that the tree is non-balanced. However, leaf and inner tuples cannot
be intermixed at the same level: a downlink from a node of an inner tuple
leads either to one inner tuple, or to a list of leaf tuples.
The SP-GiST core requires that inner and leaf tuples fit on a single index
page, and even more stringently that the list of leaf tuples reached from a
single inner-tuple node all be stored on the same index page. (Restricting
such lists to not cross pages reduces seeks, and allows the list links to be
stored as simple 2-byte OffsetNumbers.) SP-GiST index opclasses should
therefore ensure that not too many nodes can be needed in one inner tuple,
and that inner-tuple prefixes and leaf-node datum values not be too large.
Inner and leaf tuples are stored separately: the former are stored only on
"inner" pages, the latter only on "leaf" pages. Also, there are special
restrictions on the root page. Early in an index's life, when there is only
one page's worth of data, the root page contains an unorganized set of leaf
tuples. After the first page split has occurred, the root is required to
contain exactly one inner tuple.
When the search traversal algorithm reaches an inner tuple, it chooses a set
of nodes to continue tree traverse in depth. If it reaches a leaf page it
scans a list of leaf tuples to find the ones that match the query.
The insertion algorithm descends the tree similarly, except it must choose
just one node to descend to from each inner tuple. Insertion might also have
to modify the inner tuple before it can descend: it could add a new node, or
it could "split" the tuple to obtain a less-specific prefix that can match
the value to be inserted. If it's necessary to append a new leaf tuple to a
list and there is no free space on page, then SP-GiST creates a new inner
tuple and distributes leaf tuples into a set of lists on, perhaps, several
pages.
Inner tuple consists of:
optional prefix value - all successors must be consistent with it.
Example:
suffix tree - prefix value is a common prefix string
quad tree - centroid
k-d tree - one coordinate
list of nodes, where node is a (label, pointer) pair.
Example of a label: a single character for suffix tree
Leaf tuple consists of:
a leaf value
Example:
suffix tree - the rest of string (postfix)
quad and k-d tree - the point itself
ItemPointer to the heap
INSERTION ALGORITHM
Insertion algorithm is designed to keep the tree in a consistent state at
any moment. Here is a simplified insertion algorithm specification
(numbers refer to notes below):
Start with the first tuple on the root page (1)
loop:
if (page is leaf) then
if (enough space)
insert on page and exit (5)
else (7)
call PickSplitFn() (2)
end if
else
switch (chooseFn())
case MatchNode - descend through selected node
case AddNode - add node and then retry chooseFn (3, 6)
case SplitTuple - split inner tuple to prefix and postfix, then
retry chooseFn with the prefix tuple (4, 6)
end if
Notes:
(1) Initially, we just dump leaf tuples into the root page until it is full;
then we split it. Once the root is not a leaf page, it can have only one
inner tuple, so as to keep the amount of free space on the root as large as
possible. Both of these rules are meant to postpone doing PickSplit on the
root for as long as possible, so that the topmost partitioning of the search
space is as good as we can easily make it.
(2) Current implementation allows to do picksplit and insert a new leaf tuple
in one operation, if the new list of leaf tuples fits on one page. It's
always possible for trees with small nodes like quad tree or k-d tree, but
suffix trees may require another picksplit.
(3) Addition of node must keep size of inner tuple small enough to fit on a
page. After addition, inner tuple could become too large to be stored on
current page because of other tuples on page. In this case it will be moved
to another inner page (see notes about page management). When moving tuple to
another page, we can't change the numbers of other tuples on the page, else
we'd make downlink pointers to them invalid. To prevent that, SP-GiST leaves
a "placeholder" tuple, which can be reused later whenever another tuple is
added to the page. See also Concurrency and Vacuum sections below. Right now
only suffix trees could add a node to the tuple; quad trees and k-d trees
make all possible nodes at once in PickSplitFn() call.
(4) Prefix value could only partially match a new value, so the SplitTuple
action allows breaking the current tree branch into upper and lower sections.
Another way to say it is that we can split the current inner tuple into
"prefix" and "postfix" parts, where the prefix part is able to match the
incoming new value. Consider example of insertion into a suffix tree. We use
the following notation, where tuple's id is just for discussion (no such id
is actually stored):
inner tuple: {tuple id}(prefix string)[ comma separated list of node labels ]
leaf tuple: {tuple id}<value>
Suppose we need to insert string 'www.gogo.com' into inner tuple
{1}(www.google.com/)[a, i]
The string does not match the prefix so we cannot descend. We must
split the inner tuple into two tuples:
{2}(www.go)[o] - prefix tuple
|
{3}(gle.com/)[a,i] - postfix tuple
On the next iteration of loop we find that 'www.gogo.com' matches the
prefix, but not any node label, so we add a node [g] to tuple {2}:
NIL (no child exists yet)
|
{2}(www.go)[o, g]
|
{3}(gle.com/)[a,i]
Now we can descend through the [g] node, which will cause us to update
the target string to just 'o.com'. Finally, we'll insert a leaf tuple
bearing that string:
{4}<o.com>
|
{2}(www.go)[o, g]
|
{3}(gle.com/)[a,i]
As we can see, the original tuple's node array moves to postfix tuple without
any changes. Note also that SP-GiST core assumes that prefix tuple is not
larger than old inner tuple. That allows us to store prefix tuple directly
in place of old inner tuple. SP-GiST core will try to store postfix tuple on
the same page if possible, but will use another page if there is not enough
free space (see notes 5 and 6). Currently, quad and k-d trees don't use this
feature, because they have no concept of a prefix being "inconsistent" with
any new value. They grow their depth only by PickSplitFn() call.
(5) If pointer from node of parent is a NIL pointer, algorithm chooses a leaf
page to store on. At first, it tries to use the last-used leaf page with the
largest free space (which we track in each backend) to better utilize disk
space. If that's not large enough, then the algorithm allocates a new page.
(6) Management of inner pages is very similar to management of leaf pages,
described in (5).
(7) Actually, current implementation can move the whole list of leaf tuples
and a new tuple to another page, if the list is short enough. This improves
space utilization, but doesn't change the basis of the algorithm.
CONCURRENCY
While descending the tree, the insertion algorithm holds exclusive lock on
two tree levels at a time, ie both parent and child pages (parent and child
pages can be the same, see notes above). There is a possibility of deadlock
between two insertions if there are cross-referenced pages in different
branches. That is, if inner tuple on page M has a child on page N while
an inner tuple from another branch is on page N and has a child on page M,
then two insertions descending the two branches could deadlock. To prevent
deadlocks we introduce a concept of "triple parity" of pages: if inner tuple
is on page with BlockNumber N, then its child tuples should be placed on the
same page, or else on a page with BlockNumber M where (N+1) mod 3 == M mod 3.
This rule guarantees that tuples on page M will have no children on page N,
since (M+1) mod 3 != N mod 3.
Insertion may also need to take locks on an additional inner and/or leaf page
to add tuples of the right type(s), when there's not enough room on the pages
it descended through. However, we don't care exactly which such page we add
to, so deadlocks can be avoided by conditionally locking the additional
buffers: if we fail to get lock on an additional page, just try another one.
Search traversal algorithm is rather traditional. At each non-leaf level, it
share-locks the page, identifies which node(s) in the current inner tuple
need to be visited, and puts those addresses on a stack of pages to examine
later. It then releases lock on the current buffer before visiting the next
stack item. So only one page is locked at a time, and no deadlock is
possible. But instead, we have to worry about race conditions: by the time
we arrive at a pointed-to page, a concurrent insertion could have replaced
the target inner tuple (or leaf tuple chain) with data placed elsewhere.
To handle that, whenever the insertion algorithm changes a nonempty downlink
in an inner tuple, it places a "redirect tuple" in place of the lower-level
inner tuple or leaf-tuple chain head that the link formerly led to. Scans
(though not insertions) must be prepared to honor such redirects. Only a
scan that had already visited the parent level could possibly reach such a
redirect tuple, so we can remove redirects once all active transactions have
been flushed out of the system.
DEAD TUPLES
Tuples on leaf pages can be in one of four states:
SPGIST_LIVE: normal, live pointer to a heap tuple.
SPGIST_REDIRECT: placeholder that contains a link to another place in the
index. When a chain of leaf tuples has to be moved to another page, a
redirect tuple is inserted in place of the chain's head tuple. The parent
inner tuple's downlink is updated when this happens, but concurrent scans
might be "in flight" from the parent page to the child page (since they
release lock on the parent page before attempting to lock the child).
The redirect pointer serves to tell such a scan where to go. A redirect
pointer is only needed for as long as such concurrent scans could be in
progress. Eventually, it's converted into a PLACEHOLDER dead tuple by
VACUUM, and is then a candidate for replacement. Searches that find such
a tuple (which should never be part of a chain) should immediately proceed
to the other place, forgetting about the redirect tuple. Insertions that
reach such a tuple should raise error, since a valid downlink should never
point to such a tuple.
SPGIST_DEAD: tuple is dead, but it cannot be removed or moved to a
different offset on the page because there is a link leading to it from
some inner tuple elsewhere in the index. (Such a tuple is never part of a
chain, since we don't need one unless there is nothing live left in its
chain.) Searches should ignore such entries. If an insertion action
arrives at such a tuple, it should either replace it in-place (if there's
room on the page to hold the desired new leaf tuple) or replace it with a
redirection pointer to wherever it puts the new leaf tuple.
SPGIST_PLACEHOLDER: tuple is dead, and there are known to be no links to
it from elsewhere. When a live tuple is deleted or moved away, and not
replaced by a redirect pointer, it is replaced by a placeholder to keep
the offsets of later tuples on the same page from changing. Placeholders
can be freely replaced when adding a new tuple to the page, and also
VACUUM will delete any that are at the end of the range of valid tuple
offsets. Both searches and insertions should complain if a link from
elsewhere leads them to a placeholder tuple.
When the root page is also a leaf, all its tuple should be in LIVE state;
there's no need for the others since there are no links and no need to
preserve offset numbers.
Tuples on inner pages can be in LIVE, REDIRECT, or PLACEHOLDER states.
The REDIRECT state has the same function as on leaf pages, to send
concurrent searches to the place where they need to go after an inner
tuple is moved to another page. Expired REDIRECT pointers are converted
to PLACEHOLDER status by VACUUM, and are then candidates for replacement.
DEAD state is not currently possible, since VACUUM does not attempt to
remove unused inner tuples.
VACUUM
VACUUM (or more precisely, spgbulkdelete) performs a single sequential scan
over the entire index. On both leaf and inner pages, we can convert old
REDIRECT tuples into PLACEHOLDER status, and then remove any PLACEHOLDERs
that are at the end of the page (since they aren't needed to preserve the
offsets of any live tuples). On leaf pages, we scan for tuples that need
to be deleted because their heap TIDs match a vacuum target TID.
If we find a deletable tuple that is not at the head of its chain, we
can simply replace it with a PLACEHOLDER, updating the chain links to
remove it from the chain. If it is at the head of its chain, but there's
at least one live tuple remaining in the chain, we move that live tuple
to the head tuple's offset, replacing it with a PLACEHOLDER to preserve
the offsets of other tuples. This keeps the parent inner tuple's downlink
valid. If we find ourselves deleting all live tuples in a chain, we
replace the head tuple with a DEAD tuple and the rest with PLACEHOLDERS.
The parent inner tuple's downlink thus points to the DEAD tuple, and the
rules explained in the previous section keep everything working.
VACUUM doesn't know a-priori which tuples are heads of their chains, but
it can easily figure that out by constructing a predecessor array that's
the reverse map of the nextOffset links (ie, when we see tuple x links to
tuple y, we set predecessor[y] = x). Then head tuples are the ones with
no predecessor.
spgbulkdelete also updates the index's free space map.
Currently, spgvacuumcleanup has nothing to do if spgbulkdelete was
performed; otherwise, it does an spgbulkdelete scan with an empty target
list, so as to clean up redirections and placeholders, update the free
space map, and gather statistics.
LAST USED PAGE MANAGEMENT
List of last used pages contains four pages - a leaf page and three inner
pages, one from each "triple parity" group. This list is stored between
calls on the index meta page, but updates are never WAL-logged to decrease
WAL traffic. Incorrect data on meta page isn't critical, because we could
allocate a new page at any moment.
AUTHORS
Teodor Sigaev <teodor@sigaev.ru>
Oleg Bartunov <oleg@sai.msu.su>
This diff is collapsed.
/*-------------------------------------------------------------------------
*
* spginsert.c
* Externally visible index creation/insertion routines
*
* All the actual insertion logic is in spgdoinsert.c.
*
* Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
* src/backend/access/spgist/spginsert.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/genam.h"
#include "access/spgist_private.h"
#include "catalog/index.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/smgr.h"
#include "utils/memutils.h"
typedef struct
{
SpGistState spgstate; /* SPGiST's working state */
MemoryContext tmpCtx; /* per-tuple temporary context */
} SpGistBuildState;
/* Callback to process one heap tuple during IndexBuildHeapScan */
static void
spgistBuildCallback(Relation index, HeapTuple htup, Datum *values,
bool *isnull, bool tupleIsAlive, void *state)
{
SpGistBuildState *buildstate = (SpGistBuildState *) state;
/* SPGiST doesn't index nulls */
if (*isnull == false)
{
/* Work in temp context, and reset it after each tuple */
MemoryContext oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx);
spgdoinsert(index, &buildstate->spgstate, &htup->t_self, *values);
MemoryContextSwitchTo(oldCtx);
MemoryContextReset(buildstate->tmpCtx);
}
}
/*
* Build an SP-GiST index.
*/
Datum
spgbuild(PG_FUNCTION_ARGS)
{
Relation heap = (Relation) PG_GETARG_POINTER(0);
Relation index = (Relation) PG_GETARG_POINTER(1);
IndexInfo *indexInfo = (IndexInfo *) PG_GETARG_POINTER(2);
IndexBuildResult *result;
double reltuples;
SpGistBuildState buildstate;
Buffer metabuffer,
rootbuffer;
if (RelationGetNumberOfBlocks(index) != 0)
elog(ERROR, "index \"%s\" already contains data",
RelationGetRelationName(index));
/*
* Initialize the meta page and root page
*/
metabuffer = SpGistNewBuffer(index);
rootbuffer = SpGistNewBuffer(index);
Assert(BufferGetBlockNumber(metabuffer) == SPGIST_METAPAGE_BLKNO);
Assert(BufferGetBlockNumber(rootbuffer) == SPGIST_HEAD_BLKNO);
START_CRIT_SECTION();
SpGistInitMetapage(BufferGetPage(metabuffer));
MarkBufferDirty(metabuffer);
SpGistInitBuffer(rootbuffer, SPGIST_LEAF);
MarkBufferDirty(rootbuffer);
if (RelationNeedsWAL(index))
{
XLogRecPtr recptr;
XLogRecData rdata;
/* WAL data is just the relfilenode */
rdata.data = (char *) &(index->rd_node);
rdata.len = sizeof(RelFileNode);
rdata.buffer = InvalidBuffer;
rdata.next = NULL;
recptr = XLogInsert(RM_SPGIST_ID, XLOG_SPGIST_CREATE_INDEX, &rdata);
PageSetLSN(BufferGetPage(metabuffer), recptr);
PageSetTLI(BufferGetPage(metabuffer), ThisTimeLineID);
PageSetLSN(BufferGetPage(rootbuffer), recptr);
PageSetTLI(BufferGetPage(rootbuffer), ThisTimeLineID);
}
END_CRIT_SECTION();
UnlockReleaseBuffer(metabuffer);
UnlockReleaseBuffer(rootbuffer);
/*
* Now insert all the heap data into the index
*/
initSpGistState(&buildstate.spgstate, index);
buildstate.spgstate.isBuild = true;
buildstate.tmpCtx = AllocSetContextCreate(CurrentMemoryContext,
"SP-GiST build temporary context",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
reltuples = IndexBuildHeapScan(heap, index, indexInfo, true,
spgistBuildCallback, (void *) &buildstate);
MemoryContextDelete(buildstate.tmpCtx);
SpGistUpdateMetaPage(index);
result = (IndexBuildResult *) palloc0(sizeof(IndexBuildResult));
result->heap_tuples = result->index_tuples = reltuples;
PG_RETURN_POINTER(result);
}
/*
* Build an empty SPGiST index in the initialization fork
*/
Datum
spgbuildempty(PG_FUNCTION_ARGS)
{
Relation index = (Relation) PG_GETARG_POINTER(0);
Page page;
/* Construct metapage. */
page = (Page) palloc(BLCKSZ);
SpGistInitMetapage(page);
/* Write the page. If archiving/streaming, XLOG it. */
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_METAPAGE_BLKNO,
(char *) page, true);
if (XLogIsNeeded())
log_newpage(&index->rd_smgr->smgr_rnode.node, INIT_FORKNUM,
SPGIST_METAPAGE_BLKNO, page);
/* Likewise for the root page. */
SpGistInitPage(page, SPGIST_LEAF);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_HEAD_BLKNO,
(char *) page, true);
if (XLogIsNeeded())
log_newpage(&index->rd_smgr->smgr_rnode.node, INIT_FORKNUM,
SPGIST_HEAD_BLKNO, page);
/*
* An immediate sync is required even if we xlog'd the pages, because the
* writes did not go through shared buffers and therefore a concurrent
* checkpoint may have moved the redo pointer past our xlog record.
*/
smgrimmedsync(index->rd_smgr, INIT_FORKNUM);
PG_RETURN_VOID();
}
/*
* Insert one new tuple into an SPGiST index.
*/
Datum
spginsert(PG_FUNCTION_ARGS)
{
Relation index = (Relation) PG_GETARG_POINTER(0);
Datum *values = (Datum *) PG_GETARG_POINTER(1);
bool *isnull = (bool *) PG_GETARG_POINTER(2);
ItemPointer ht_ctid = (ItemPointer) PG_GETARG_POINTER(3);
#ifdef NOT_USED
Relation heapRel = (Relation) PG_GETARG_POINTER(4);
IndexUniqueCheck checkUnique = (IndexUniqueCheck) PG_GETARG_INT32(5);
#endif
SpGistState spgstate;
MemoryContext oldCtx;
MemoryContext insertCtx;
/* SPGiST doesn't index nulls */
if (*isnull)
PG_RETURN_BOOL(false);
insertCtx = AllocSetContextCreate(CurrentMemoryContext,
"SP-GiST insert temporary context",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
oldCtx = MemoryContextSwitchTo(insertCtx);
initSpGistState(&spgstate, index);
spgdoinsert(index, &spgstate, ht_ctid, *values);
SpGistUpdateMetaPage(index);
MemoryContextSwitchTo(oldCtx);
MemoryContextDelete(insertCtx);
/* return false since we've not done any unique check */
PG_RETURN_BOOL(false);
}
/*-------------------------------------------------------------------------
*
* spgkdtreeproc.c
* implementation of k-d tree over points for SP-GiST
*
*
* Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
* src/backend/access/spgist/spgkdtreeproc.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/gist.h" /* for RTree strategy numbers */
#include "access/spgist.h"
#include "catalog/pg_type.h"
#include "utils/builtins.h"
#include "utils/geo_decls.h"
Datum
spg_kd_config(PG_FUNCTION_ARGS)
{
/* spgConfigIn *cfgin = (spgConfigIn *) PG_GETARG_POINTER(0); */
spgConfigOut *cfg = (spgConfigOut *) PG_GETARG_POINTER(1);
cfg->prefixType = FLOAT8OID;
cfg->labelType = VOIDOID; /* we don't need node labels */
cfg->longValuesOK = false;
PG_RETURN_VOID();
}
static int
getSide(double coord, bool isX, Point *tst)
{
double tstcoord = (isX) ? tst->x : tst->y;
if (coord == tstcoord)
return 0;
else if (coord > tstcoord)
return 1;
else
return -1;
}
Datum
spg_kd_choose(PG_FUNCTION_ARGS)
{
spgChooseIn *in = (spgChooseIn *) PG_GETARG_POINTER(0);
spgChooseOut *out = (spgChooseOut *) PG_GETARG_POINTER(1);
Point *inPoint = DatumGetPointP(in->datum);
double coord;
if (in->allTheSame)
elog(ERROR, "allTheSame should not occur for k-d trees");
Assert(in->hasPrefix);
coord = DatumGetFloat8(in->prefixDatum);
Assert(in->nNodes == 2);
out->resultType = spgMatchNode;
out->result.matchNode.nodeN =
(getSide(coord, in->level % 2, inPoint) > 0) ? 0 : 1;
out->result.matchNode.levelAdd = 1;
out->result.matchNode.restDatum = PointPGetDatum(inPoint);
PG_RETURN_VOID();
}
typedef struct SortedPoint
{
Point *p;
int i;
} SortedPoint;
static int
x_cmp(const void *a, const void *b)
{
SortedPoint *pa = (SortedPoint *) a;
SortedPoint *pb = (SortedPoint *) b;
if (pa->p->x == pb->p->x)
return 0;
return (pa->p->x > pb->p->x) ? 1 : -1;
}
static int
y_cmp(const void *a, const void *b)
{
SortedPoint *pa = (SortedPoint *) a;
SortedPoint *pb = (SortedPoint *) b;
if (pa->p->y == pb->p->y)
return 0;
return (pa->p->y > pb->p->y) ? 1 : -1;
}
Datum
spg_kd_picksplit(PG_FUNCTION_ARGS)
{
spgPickSplitIn *in = (spgPickSplitIn *) PG_GETARG_POINTER(0);
spgPickSplitOut *out = (spgPickSplitOut *) PG_GETARG_POINTER(1);
int i;
int middle;
SortedPoint *sorted;
double coord;
sorted = palloc(sizeof(*sorted) * in->nTuples);
for (i = 0; i < in->nTuples; i++)
{
sorted[i].p = DatumGetPointP(in->datums[i]);
sorted[i].i = i;
}
qsort(sorted, in->nTuples, sizeof(*sorted),
(in->level % 2) ? x_cmp : y_cmp);
middle = in->nTuples >> 1;
coord = (in->level % 2) ? sorted[middle].p->x : sorted[middle].p->y;
out->hasPrefix = true;
out->prefixDatum = Float8GetDatum(coord);
out->nNodes = 2;
out->nodeLabels = NULL; /* we don't need node labels */
out->mapTuplesToNodes = palloc(sizeof(int) * in->nTuples);
out->leafTupleDatums = palloc(sizeof(Datum) * in->nTuples);
/*
* Note: points that have coordinates exactly equal to coord may get
* classified into either node, depending on where they happen to fall
* in the sorted list. This is okay as long as the inner_consistent
* function descends into both sides for such cases. This is better
* than the alternative of trying to have an exact boundary, because
* it keeps the tree balanced even when we have many instances of the
* same point value. So we should never trigger the allTheSame logic.
*/
for (i = 0; i < in->nTuples; i++)
{
Point *p = sorted[i].p;
int n = sorted[i].i;
out->mapTuplesToNodes[n] = (i < middle) ? 0 : 1;
out->leafTupleDatums[n] = PointPGetDatum(p);
}
PG_RETURN_VOID();
}
Datum
spg_kd_inner_consistent(PG_FUNCTION_ARGS)
{
spgInnerConsistentIn *in = (spgInnerConsistentIn *) PG_GETARG_POINTER(0);
spgInnerConsistentOut *out = (spgInnerConsistentOut *) PG_GETARG_POINTER(1);
Point *query;
BOX *boxQuery;
double coord;
query = DatumGetPointP(in->query);
Assert(in->hasPrefix);
coord = DatumGetFloat8(in->prefixDatum);
if (in->allTheSame)
elog(ERROR, "allTheSame should not occur for k-d trees");
Assert(in->nNodes == 2);
out->nodeNumbers = (int *) palloc(sizeof(int) * 2);
out->levelAdds = (int *) palloc(sizeof(int) * 2);
out->levelAdds[0] = 1;
out->levelAdds[1] = 1;
out->nNodes = 0;
switch (in->strategy)
{
case RTLeftStrategyNumber:
out->nNodes = 1;
out->nodeNumbers[0] = 0;
if ((in->level % 2) == 0 || FPge(query->x, coord))
{
out->nodeNumbers[1] = 1;
out->nNodes++;
}
break;
case RTRightStrategyNumber:
out->nNodes = 1;
out->nodeNumbers[0] = 1;
if ((in->level % 2) == 0 || FPle(query->x, coord))
{
out->nodeNumbers[1] = 0;
out->nNodes++;
}
break;
case RTSameStrategyNumber:
if (in->level % 2)
{
if (FPle(query->x, coord))
{
out->nodeNumbers[out->nNodes] = 0;
out->nNodes++;
}
if (FPge(query->x, coord))
{
out->nodeNumbers[out->nNodes] = 1;
out->nNodes++;
}
}
else
{
if (FPle(query->y, coord))
{
out->nodeNumbers[out->nNodes] = 0;
out->nNodes++;
}
if (FPge(query->y, coord))
{
out->nodeNumbers[out->nNodes] = 1;
out->nNodes++;
}
}
break;
case RTBelowStrategyNumber:
out->nNodes = 1;
out->nodeNumbers[0] = 0;
if ((in->level % 2) == 1 || FPge(query->y, coord))
{
out->nodeNumbers[1] = 1;
out->nNodes++;
}
break;
case RTAboveStrategyNumber:
out->nNodes = 1;
out->nodeNumbers[0] = 1;
if ((in->level % 2) == 1 || FPle(query->y, coord))
{
out->nodeNumbers[1] = 0;
out->nNodes++;
}
break;
case RTContainedByStrategyNumber:
/*
* For this operator, the query is a box not a point. We cheat to
* the extent of assuming that DatumGetPointP won't do anything
* that would be bad for a pointer-to-box.
*/
boxQuery = DatumGetBoxP(in->query);
out->nNodes = 1;
if (in->level % 2)
{
if (FPlt(boxQuery->high.x, coord))
out->nodeNumbers[0] = 0;
else if (FPgt(boxQuery->low.x, coord))
out->nodeNumbers[0] = 1;
else
{
out->nodeNumbers[0] = 0;
out->nodeNumbers[1] = 1;
out->nNodes = 2;
}
}
else
{
if (FPlt(boxQuery->high.y, coord))
out->nodeNumbers[0] = 0;
else if (FPgt(boxQuery->low.y, coord))
out->nodeNumbers[0] = 1;
else
{
out->nodeNumbers[0] = 0;
out->nodeNumbers[1] = 1;
out->nNodes = 2;
}
}
break;
default:
elog(ERROR, "unrecognized strategy number: %d", in->strategy);
break;
}
PG_RETURN_VOID();
}
/*
* spg_kd_leaf_consistent() is the same as spg_quad_leaf_consistent(),
* since we support the same operators and the same leaf data type.
* So we just borrow that function.
*/
/*-------------------------------------------------------------------------
*
* spgquadtreeproc.c
* implementation of quad tree over points for SP-GiST
*
*
* Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
* src/backend/access/spgist/spgquadtreeproc.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/gist.h" /* for RTree strategy numbers */
#include "access/spgist.h"
#include "catalog/pg_type.h"
#include "utils/builtins.h"
#include "utils/geo_decls.h"
Datum
spg_quad_config(PG_FUNCTION_ARGS)
{
/* spgConfigIn *cfgin = (spgConfigIn *) PG_GETARG_POINTER(0); */
spgConfigOut *cfg = (spgConfigOut *) PG_GETARG_POINTER(1);
cfg->prefixType = POINTOID;
cfg->labelType = VOIDOID; /* we don't need node labels */
cfg->longValuesOK = false;
PG_RETURN_VOID();
}
#define SPTEST(f, x, y) \
DatumGetBool(DirectFunctionCall2(f, PointPGetDatum(x), PointPGetDatum(y)))
/*
* Determine which quadrant a point falls into, relative to the centroid.
*
* Quadrants are identified like this:
*
* 4 | 1
* ----+-----
* 3 | 2
*
* Points on one of the axes are taken to lie in the lowest-numbered
* adjacent quadrant.
*/
static int2
getQuadrant(Point *centroid, Point *tst)
{
if ((SPTEST(point_above, tst, centroid) ||
SPTEST(point_horiz, tst, centroid)) &&
(SPTEST(point_right, tst, centroid) ||
SPTEST(point_vert, tst, centroid)))
return 1;
if (SPTEST(point_below, tst, centroid) &&
(SPTEST(point_right, tst, centroid) ||
SPTEST(point_vert, tst, centroid)))
return 2;
if ((SPTEST(point_below, tst, centroid) ||
SPTEST(point_horiz, tst, centroid)) &&
SPTEST(point_left, tst, centroid))
return 3;
if (SPTEST(point_above, tst, centroid) &&
SPTEST(point_left, tst, centroid))
return 4;
elog(ERROR, "getQuadrant: impossible case");
return 0;
}
Datum
spg_quad_choose(PG_FUNCTION_ARGS)
{
spgChooseIn *in = (spgChooseIn *) PG_GETARG_POINTER(0);
spgChooseOut *out = (spgChooseOut *) PG_GETARG_POINTER(1);
Point *inPoint = DatumGetPointP(in->datum),
*centroid;
if (in->allTheSame)
{
out->resultType = spgMatchNode;
/* nodeN will be set by core */
out->result.matchNode.levelAdd = 0;
out->result.matchNode.restDatum = PointPGetDatum(inPoint);
PG_RETURN_VOID();
}
Assert(in->hasPrefix);
centroid = DatumGetPointP(in->prefixDatum);
Assert(in->nNodes == 4);
out->resultType = spgMatchNode;
out->result.matchNode.nodeN = getQuadrant(centroid, inPoint) - 1;
out->result.matchNode.levelAdd = 0;
out->result.matchNode.restDatum = PointPGetDatum(inPoint);
PG_RETURN_VOID();
}
#ifdef USE_MEDIAN
static int
x_cmp(const void *a, const void *b, void *arg)
{
Point *pa = *(Point **) a;
Point *pb = *(Point **) b;
if (pa->x == pb->x)
return 0;
return (pa->x > pb->x) ? 1 : -1;
}
static int
y_cmp(const void *a, const void *b, void *arg)
{
Point *pa = *(Point **) a;
Point *pb = *(Point **) b;
if (pa->y == pb->y)
return 0;
return (pa->y > pb->y) ? 1 : -1;
}
#endif
Datum
spg_quad_picksplit(PG_FUNCTION_ARGS)
{
spgPickSplitIn *in = (spgPickSplitIn *) PG_GETARG_POINTER(0);
spgPickSplitOut *out = (spgPickSplitOut *) PG_GETARG_POINTER(1);
int i;
Point *centroid;
#ifdef USE_MEDIAN
/* Use the median values of x and y as the centroid point */
Point **sorted;
sorted = palloc(sizeof(*sorted) * in->nTuples);
for (i = 0; i < in->nTuples; i++)
sorted[i] = DatumGetPointP(in->datums[i]);
centroid = palloc(sizeof(*centroid));
qsort(sorted, in->nTuples, sizeof(*sorted), x_cmp);
centroid->x = sorted[in->nTuples >> 1]->x;
qsort(sorted, in->nTuples, sizeof(*sorted), y_cmp);
centroid->y = sorted[in->nTuples >> 1]->y;
#else
/* Use the average values of x and y as the centroid point */
centroid = palloc0(sizeof(*centroid));
for (i = 0; i < in->nTuples; i++)
{
centroid->x += DatumGetPointP(in->datums[i])->x;
centroid->y += DatumGetPointP(in->datums[i])->y;
}
centroid->x /= in->nTuples;
centroid->y /= in->nTuples;
#endif
out->hasPrefix = true;
out->prefixDatum = PointPGetDatum(centroid);
out->nNodes = 4;
out->nodeLabels = NULL; /* we don't need node labels */
out->mapTuplesToNodes = palloc(sizeof(int) * in->nTuples);
out->leafTupleDatums = palloc(sizeof(Datum) * in->nTuples);
for (i = 0; i < in->nTuples; i++)
{
Point *p = DatumGetPointP(in->datums[i]);
int quadrant = getQuadrant(centroid, p) - 1;
out->leafTupleDatums[i] = PointPGetDatum(p);
out->mapTuplesToNodes[i] = quadrant;
}
PG_RETURN_VOID();
}
/* Subroutine to fill out->nodeNumbers[] for spg_quad_inner_consistent */
static void
setNodes(spgInnerConsistentOut *out, bool isAll, int first, int second)
{
if (isAll)
{
out->nNodes = 4;
out->nodeNumbers[0] = 0;
out->nodeNumbers[1] = 1;
out->nodeNumbers[2] = 2;
out->nodeNumbers[3] = 3;
}
else
{
out->nNodes = 2;
out->nodeNumbers[0] = first - 1;
out->nodeNumbers[1] = second - 1;
}
}
Datum
spg_quad_inner_consistent(PG_FUNCTION_ARGS)
{
spgInnerConsistentIn *in = (spgInnerConsistentIn *) PG_GETARG_POINTER(0);
spgInnerConsistentOut *out = (spgInnerConsistentOut *) PG_GETARG_POINTER(1);
Point *query,
*centroid;
BOX *boxQuery;
query = DatumGetPointP(in->query);
Assert(in->hasPrefix);
centroid = DatumGetPointP(in->prefixDatum);
if (in->allTheSame)
{
/* Report that all nodes should be visited */
int i;
out->nNodes = in->nNodes;
out->nodeNumbers = (int *) palloc(sizeof(int) * in->nNodes);
for (i = 0; i < in->nNodes; i++)
out->nodeNumbers[i] = i;
PG_RETURN_VOID();
}
Assert(in->nNodes == 4);
out->nodeNumbers = (int *) palloc(sizeof(int) * 4);
switch (in->strategy)
{
case RTLeftStrategyNumber:
setNodes(out, SPTEST(point_left, centroid, query), 3, 4);
break;
case RTRightStrategyNumber:
setNodes(out, SPTEST(point_right, centroid, query), 1, 2);
break;
case RTSameStrategyNumber:
out->nNodes = 1;
out->nodeNumbers[0] = getQuadrant(centroid, query) - 1;
break;
case RTBelowStrategyNumber:
setNodes(out, SPTEST(point_below, centroid, query), 2, 3);
break;
case RTAboveStrategyNumber:
setNodes(out, SPTEST(point_above, centroid, query), 1, 4);
break;
case RTContainedByStrategyNumber:
/*
* For this operator, the query is a box not a point. We cheat to
* the extent of assuming that DatumGetPointP won't do anything
* that would be bad for a pointer-to-box.
*/
boxQuery = DatumGetBoxP(in->query);
if (DatumGetBool(DirectFunctionCall2(box_contain_pt,
PointerGetDatum(boxQuery),
PointerGetDatum(centroid))))
{
/* centroid is in box, so descend to all quadrants */
setNodes(out, true, 0, 0);
}
else
{
/* identify quadrant(s) containing all corners of box */
Point p;
int i,
r = 0;
p = boxQuery->low;
r |= 1 << (getQuadrant(centroid, &p) - 1);
p.y = boxQuery->high.y;
r |= 1 << (getQuadrant(centroid, &p) - 1);
p = boxQuery->high;
r |= 1 << (getQuadrant(centroid, &p) - 1);
p.x = boxQuery->low.x;
r |= 1 << (getQuadrant(centroid, &p) - 1);
/* we must descend into those quadrant(s) */
out->nNodes = 0;
for (i = 0; i < 4; i++)
{
if (r & (1 << i))
{
out->nodeNumbers[out->nNodes] = i;
out->nNodes++;
}
}
}
break;
default:
elog(ERROR, "unrecognized strategy number: %d", in->strategy);
break;
}
PG_RETURN_VOID();
}
Datum
spg_quad_leaf_consistent(PG_FUNCTION_ARGS)
{
spgLeafConsistentIn *in = (spgLeafConsistentIn *) PG_GETARG_POINTER(0);
spgLeafConsistentOut *out = (spgLeafConsistentOut *) PG_GETARG_POINTER(1);
Point *query = DatumGetPointP(in->query);
Point *datum = DatumGetPointP(in->leafDatum);
bool res;
/* all tests are exact */
out->recheck = false;
switch (in->strategy)
{
case RTLeftStrategyNumber:
res = SPTEST(point_left, datum, query);
break;
case RTRightStrategyNumber:
res = SPTEST(point_right, datum, query);
break;
case RTSameStrategyNumber:
res = SPTEST(point_eq, datum, query);
break;
case RTBelowStrategyNumber:
res = SPTEST(point_below, datum, query);
break;
case RTAboveStrategyNumber:
res = SPTEST(point_above, datum, query);
break;
case RTContainedByStrategyNumber:
/*
* For this operator, the query is a box not a point. We cheat to
* the extent of assuming that DatumGetPointP won't do anything
* that would be bad for a pointer-to-box.
*/
res = SPTEST(box_contain_pt, query, datum);
break;
default:
elog(ERROR, "unrecognized strategy number: %d", in->strategy);
res = false;
break;
}
PG_RETURN_BOOL(res);
}
/*-------------------------------------------------------------------------
*
* spgscan.c
* routines for scanning SP-GiST indexes
*
*
* Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
* src/backend/access/spgist/spgscan.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/relscan.h"
#include "access/spgist_private.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "utils/datum.h"
#include "utils/memutils.h"
typedef struct ScanStackEntry
{
Datum reconstructedValue; /* value reconstructed from parent */
int level; /* level of items on this page */
ItemPointerData ptr; /* block and offset to scan from */
} ScanStackEntry;
/* Free a ScanStackEntry */
static void
freeScanStackEntry(SpGistScanOpaque so, ScanStackEntry *stackEntry)
{
if (!so->state.attType.attbyval &&
DatumGetPointer(stackEntry->reconstructedValue) != NULL)
pfree(DatumGetPointer(stackEntry->reconstructedValue));
pfree(stackEntry);
}
/* Free the entire stack */
static void
freeScanStack(SpGistScanOpaque so)
{
ListCell *lc;
foreach(lc, so->scanStack)
{
freeScanStackEntry(so, (ScanStackEntry *) lfirst(lc));
}
list_free(so->scanStack);
so->scanStack = NIL;
}
/* Initialize scanStack with a single entry for the root page */
static void
resetSpGistScanOpaque(SpGistScanOpaque so)
{
ScanStackEntry *startEntry = palloc0(sizeof(ScanStackEntry));
ItemPointerSet(&startEntry->ptr, SPGIST_HEAD_BLKNO, FirstOffsetNumber);
freeScanStack(so);
so->scanStack = list_make1(startEntry);
so->nPtrs = so->iPtr = 0;
}
Datum
spgbeginscan(PG_FUNCTION_ARGS)
{
Relation rel = (Relation) PG_GETARG_POINTER(0);
int keysz = PG_GETARG_INT32(1);
/* ScanKey scankey = (ScanKey) PG_GETARG_POINTER(2); */
IndexScanDesc scan;
SpGistScanOpaque so;
scan = RelationGetIndexScan(rel, keysz, 0);
so = (SpGistScanOpaque) palloc0(sizeof(SpGistScanOpaqueData));
initSpGistState(&so->state, scan->indexRelation);
so->tempCxt = AllocSetContextCreate(CurrentMemoryContext,
"SP-GiST search temporary context",
ALLOCSET_DEFAULT_MINSIZE,
ALLOCSET_DEFAULT_INITSIZE,
ALLOCSET_DEFAULT_MAXSIZE);
resetSpGistScanOpaque(so);
scan->opaque = so;
PG_RETURN_POINTER(scan);
}
Datum
spgrescan(PG_FUNCTION_ARGS)
{
IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
SpGistScanOpaque so = (SpGistScanOpaque) scan->opaque;
ScanKey scankey = (ScanKey) PG_GETARG_POINTER(1);
if (scankey && scan->numberOfKeys > 0)
{
memmove(scan->keyData, scankey,
scan->numberOfKeys * sizeof(ScanKeyData));
}
resetSpGistScanOpaque(so);
PG_RETURN_VOID();
}
Datum
spgendscan(PG_FUNCTION_ARGS)
{
IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
SpGistScanOpaque so = (SpGistScanOpaque) scan->opaque;
MemoryContextDelete(so->tempCxt);
PG_RETURN_VOID();
}
Datum
spgmarkpos(PG_FUNCTION_ARGS)
{
elog(ERROR, "SPGiST does not support mark/restore");
PG_RETURN_VOID();
}
Datum
spgrestrpos(PG_FUNCTION_ARGS)
{
elog(ERROR, "SPGiST does not support mark/restore");
PG_RETURN_VOID();
}
/*
* Test whether a leaf datum satisfies all the scan keys
*
* *recheck is set true if any of the operators are lossy
*/
static bool
spgLeafTest(SpGistScanOpaque so, Datum leafDatum,
int level, Datum reconstructedValue,
bool *recheck)
{
bool result = true;
spgLeafConsistentIn in;
spgLeafConsistentOut out;
MemoryContext oldCtx;
int i;
*recheck = false;
/* set up values that are the same for all quals */
in.reconstructedValue = reconstructedValue;
in.level = level;
in.leafDatum = leafDatum;
/* Apply each leaf consistent function, working in the temp context */
oldCtx = MemoryContextSwitchTo(so->tempCxt);
for (i = 0; i < so->numberOfKeys; i++)
{
in.strategy = so->keyData[i].sk_strategy;
in.query = so->keyData[i].sk_argument;
out.recheck = false;
result = DatumGetBool(FunctionCall2Coll(&so->state.leafConsistentFn,
so->keyData[i].sk_collation,
PointerGetDatum(&in),
PointerGetDatum(&out)));
*recheck |= out.recheck;
if (!result)
break;
}
MemoryContextSwitchTo(oldCtx);
return result;
}
/*
* Walk the tree and report all tuples passing the scan quals to the storeRes
* subroutine.
*
* If scanWholeIndex is true, we'll do just that. If not, we'll stop at the
* next page boundary once we have reported at least one tuple.
*/
static void
spgWalk(Relation index, SpGistScanOpaque so, bool scanWholeIndex,
void (*storeRes) (SpGistScanOpaque, ItemPointer, bool))
{
Buffer buffer = InvalidBuffer;
bool reportedSome = false;
while (scanWholeIndex || !reportedSome)
{
ScanStackEntry *stackEntry;
BlockNumber blkno;
OffsetNumber offset;
Page page;
/* Pull next to-do item from the list */
if (so->scanStack == NIL)
break; /* there are no more pages to scan */
stackEntry = (ScanStackEntry *) linitial(so->scanStack);
so->scanStack = list_delete_first(so->scanStack);
redirect:
/* Check for interrupts, just in case of infinite loop */
CHECK_FOR_INTERRUPTS();
blkno = ItemPointerGetBlockNumber(&stackEntry->ptr);
offset = ItemPointerGetOffsetNumber(&stackEntry->ptr);
if (buffer == InvalidBuffer)
{
buffer = ReadBuffer(index, blkno);
LockBuffer(buffer, BUFFER_LOCK_SHARE);
}
else if (blkno != BufferGetBlockNumber(buffer))
{
UnlockReleaseBuffer(buffer);
buffer = ReadBuffer(index, blkno);
LockBuffer(buffer, BUFFER_LOCK_SHARE);
}
/* else new pointer points to the same page, no work needed */
page = BufferGetPage(buffer);
if (SpGistPageIsLeaf(page))
{
SpGistLeafTuple leafTuple;
OffsetNumber max = PageGetMaxOffsetNumber(page);
bool recheck = false;
if (blkno == SPGIST_HEAD_BLKNO)
{
/* When root is a leaf, examine all its tuples */
for (offset = FirstOffsetNumber; offset <= max; offset++)
{
leafTuple = (SpGistLeafTuple)
PageGetItem(page, PageGetItemId(page, offset));
if (leafTuple->tupstate != SPGIST_LIVE)
{
/* all tuples on root should be live */
elog(ERROR, "unexpected SPGiST tuple state: %d",
leafTuple->tupstate);
}
Assert(ItemPointerIsValid(&leafTuple->heapPtr));
if (spgLeafTest(so,
SGLTDATUM(leafTuple, &so->state),
stackEntry->level,
stackEntry->reconstructedValue,
&recheck))
{
storeRes(so, &leafTuple->heapPtr, recheck);
reportedSome = true;
}
}
}
else
{
/* Normal case: just examine the chain we arrived at */
while (offset != InvalidOffsetNumber)
{
Assert(offset >= FirstOffsetNumber && offset <= max);
leafTuple = (SpGistLeafTuple)
PageGetItem(page, PageGetItemId(page, offset));
if (leafTuple->tupstate != SPGIST_LIVE)
{
if (leafTuple->tupstate == SPGIST_REDIRECT)
{
/* redirection tuple should be first in chain */
Assert(offset == ItemPointerGetOffsetNumber(&stackEntry->ptr));
/* transfer attention to redirect point */
stackEntry->ptr = ((SpGistDeadTuple) leafTuple)->pointer;
Assert(ItemPointerGetBlockNumber(&stackEntry->ptr) != SPGIST_METAPAGE_BLKNO);
goto redirect;
}
if (leafTuple->tupstate == SPGIST_DEAD)
{
/* dead tuple should be first in chain */
Assert(offset == ItemPointerGetOffsetNumber(&stackEntry->ptr));
/* No live entries on this page */
Assert(leafTuple->nextOffset == InvalidOffsetNumber);
break;
}
/* We should not arrive at a placeholder */
elog(ERROR, "unexpected SPGiST tuple state: %d",
leafTuple->tupstate);
}
Assert(ItemPointerIsValid(&leafTuple->heapPtr));
if (spgLeafTest(so,
SGLTDATUM(leafTuple, &so->state),
stackEntry->level,
stackEntry->reconstructedValue,
&recheck))
{
storeRes(so, &leafTuple->heapPtr, recheck);
reportedSome = true;
}
offset = leafTuple->nextOffset;
}
}
}
else /* page is inner */
{
SpGistInnerTuple innerTuple;
SpGistNodeTuple node;
int i;
innerTuple = (SpGistInnerTuple) PageGetItem(page,
PageGetItemId(page, offset));
if (innerTuple->tupstate != SPGIST_LIVE)
{
if (innerTuple->tupstate == SPGIST_REDIRECT)
{
/* transfer attention to redirect point */
stackEntry->ptr = ((SpGistDeadTuple) innerTuple)->pointer;
Assert(ItemPointerGetBlockNumber(&stackEntry->ptr) != SPGIST_METAPAGE_BLKNO);
goto redirect;
}
elog(ERROR, "unexpected SPGiST tuple state: %d",
innerTuple->tupstate);
}
if (so->numberOfKeys == 0)
{
/*
* This case cannot happen at the moment, because we don't
* set pg_am.amoptionalkey for SP-GiST. In order for full
* index scans to produce correct answers, we'd need to
* index nulls, which we don't.
*/
Assert(false);
#ifdef NOT_USED
/*
* A full index scan could be done approximately like this,
* but note that reconstruction of indexed values would be
* impossible unless the API for inner_consistent is changed.
*/
SGITITERATE(innerTuple, i, node)
{
if (ItemPointerIsValid(&node->t_tid))
{
ScanStackEntry *newEntry = palloc(sizeof(ScanStackEntry));
newEntry->ptr = node->t_tid;
newEntry->level = -1;
newEntry->reconstructedValue = (Datum) 0;
so->scanStack = lcons(newEntry, so->scanStack);
}
}
#endif
}
else
{
spgInnerConsistentIn in;
spgInnerConsistentOut out;
SpGistNodeTuple *nodes;
int *andMap;
int *levelAdds;
Datum *reconstructedValues;
int j,
nMatches = 0;
MemoryContext oldCtx;
/* use temp context for calling inner_consistent */
oldCtx = MemoryContextSwitchTo(so->tempCxt);
/* set up values that are the same for all scankeys */
in.reconstructedValue = stackEntry->reconstructedValue;
in.level = stackEntry->level;
in.allTheSame = innerTuple->allTheSame;
in.hasPrefix = (innerTuple->prefixSize > 0);
in.prefixDatum = SGITDATUM(innerTuple, &so->state);
in.nNodes = innerTuple->nNodes;
in.nodeLabels = spgExtractNodeLabels(&so->state, innerTuple);
/* collect node pointers */
nodes = (SpGistNodeTuple *) palloc(sizeof(SpGistNodeTuple) * in.nNodes);
SGITITERATE(innerTuple, i, node)
{
nodes[i] = node;
}
andMap = (int *) palloc0(sizeof(int) * in.nNodes);
levelAdds = (int *) palloc0(sizeof(int) * in.nNodes);
reconstructedValues = (Datum *) palloc0(sizeof(Datum) * in.nNodes);
for (j = 0; j < so->numberOfKeys; j++)
{
in.strategy = so->keyData[j].sk_strategy;
in.query = so->keyData[j].sk_argument;
memset(&out, 0, sizeof(out));
FunctionCall2Coll(&so->state.innerConsistentFn,
so->keyData[j].sk_collation,
PointerGetDatum(&in),
PointerGetDatum(&out));
/* If allTheSame, they should all or none of 'em match */
if (innerTuple->allTheSame)
if (out.nNodes != 0 && out.nNodes != in.nNodes)
elog(ERROR, "inconsistent inner_consistent results for allTheSame inner tuple");
nMatches = 0;
for (i = 0; i < out.nNodes; i++)
{
int nodeN = out.nodeNumbers[i];
andMap[nodeN]++;
if (andMap[nodeN] == j + 1)
nMatches++;
if (out.levelAdds)
levelAdds[nodeN] = out.levelAdds[i];
if (out.reconstructedValues)
reconstructedValues[nodeN] = out.reconstructedValues[i];
}
/* quit as soon as all nodes have failed some qual */
if (nMatches == 0)
break;
}
MemoryContextSwitchTo(oldCtx);
if (nMatches > 0)
{
for (i = 0; i < in.nNodes; i++)
{
if (andMap[i] == so->numberOfKeys &&
ItemPointerIsValid(&nodes[i]->t_tid))
{
ScanStackEntry *newEntry;
/* Create new work item for this node */
newEntry = palloc(sizeof(ScanStackEntry));
newEntry->ptr = nodes[i]->t_tid;
newEntry->level = stackEntry->level + levelAdds[i];
/* Must copy value out of temp context */
newEntry->reconstructedValue =
datumCopy(reconstructedValues[i],
so->state.attType.attbyval,
so->state.attType.attlen);
so->scanStack = lcons(newEntry, so->scanStack);
}
}
}
}
}
/* done with this scan stack entry */
freeScanStackEntry(so, stackEntry);
/* clear temp context before proceeding to the next one */
MemoryContextReset(so->tempCxt);
}
if (buffer != InvalidBuffer)
UnlockReleaseBuffer(buffer);
}
/* storeRes subroutine for getbitmap case */
static void
storeBitmap(SpGistScanOpaque so, ItemPointer heapPtr, bool recheck)
{
tbm_add_tuples(so->tbm, heapPtr, 1, recheck);
so->ntids++;
}
Datum
spggetbitmap(PG_FUNCTION_ARGS)
{
IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
TIDBitmap *tbm = (TIDBitmap *) PG_GETARG_POINTER(1);
SpGistScanOpaque so = (SpGistScanOpaque) scan->opaque;
/* Copy scankey to *so so we don't need to pass it around separately */
so->numberOfKeys = scan->numberOfKeys;
so->keyData = scan->keyData;
so->tbm = tbm;
so->ntids = 0;
spgWalk(scan->indexRelation, so, true, storeBitmap);
PG_RETURN_INT64(so->ntids);
}
/* storeRes subroutine for gettuple case */
static void
storeGettuple(SpGistScanOpaque so, ItemPointer heapPtr, bool recheck)
{
Assert(so->nPtrs < MaxIndexTuplesPerPage);
so->heapPtrs[so->nPtrs] = *heapPtr;
so->recheck[so->nPtrs] = recheck;
so->nPtrs++;
}
Datum
spggettuple(PG_FUNCTION_ARGS)
{
IndexScanDesc scan = (IndexScanDesc) PG_GETARG_POINTER(0);
ScanDirection dir = (ScanDirection) PG_GETARG_INT32(1);
SpGistScanOpaque so = (SpGistScanOpaque) scan->opaque;
if (dir != ForwardScanDirection)
elog(ERROR, "SP-GiST only supports forward scan direction");
/* Copy scankey to *so so we don't need to pass it around separately */
so->numberOfKeys = scan->numberOfKeys;
so->keyData = scan->keyData;
for (;;)
{
if (so->iPtr < so->nPtrs)
{
/* continuing to return tuples from a leaf page */
scan->xs_ctup.t_self = so->heapPtrs[so->iPtr];
scan->xs_recheck = so->recheck[so->iPtr];
so->iPtr++;
PG_RETURN_BOOL(true);
}
so->iPtr = so->nPtrs = 0;
spgWalk(scan->indexRelation, so, false, storeGettuple);
if (so->nPtrs == 0)
break; /* must have completed scan */
}
PG_RETURN_BOOL(false);
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment