Skip to content
Snippets Groups Projects
  1. Oct 15, 2015
    • Robert Haas's avatar
      Allow FDWs to push down quals without breaking EvalPlanQual rechecks. · 5fc4c26d
      Robert Haas authored
      This fixes a long-standing bug which was discovered while investigating
      the interaction between the new join pushdown code and the EvalPlanQual
      machinery: if a ForeignScan appears on the inner side of a paramaterized
      nestloop, an EPQ recheck would re-return the original tuple even if
      it no longer satisfied the pushed-down quals due to changed parameter
      values.
      
      This fix adds a new member to ForeignScan and ForeignScanState and a
      new argument to make_foreignscan, and requires changes to FDWs which
      push down quals to populate that new argument with a list of quals they
      have chosen to push down.  Therefore, I'm only back-patching to 9.5,
      even though the bug is not new in 9.5.
      
      Etsuro Fujita, reviewed by me and by Kyotaro Horiguchi.
      5fc4c26d
  2. May 15, 2015
    • Simon Riggs's avatar
      TABLESAMPLE, SQL Standard and extensible · f6d208d6
      Simon Riggs authored
      Add a TABLESAMPLE clause to SELECT statements that allows
      user to specify random BERNOULLI sampling or block level
      SYSTEM sampling. Implementation allows for extensible
      sampling functions to be written, using a standard API.
      Basic version follows SQLStandard exactly. Usable
      concrete use cases for the sampling API follow in later
      commits.
      
      Petr Jelinek
      
      Reviewed by Michael Paquier and Simon Riggs
      f6d208d6
    • Simon Riggs's avatar
      Separate block sampling functions · 83e176ec
      Simon Riggs authored
      Refactoring ahead of tablesample patch
      
      Requested and reviewed by Michael Paquier
      
      Petr Jelinek
      83e176ec
  3. May 10, 2015
    • Tom Lane's avatar
      Code review for foreign/custom join pushdown patch. · 1a8a4e5c
      Tom Lane authored
      Commit e7cb7ee1 included some design
      decisions that seem pretty questionable to me, and there was quite a lot
      of stuff not to like about the documentation and comments.  Clean up
      as follows:
      
      * Consider foreign joins only between foreign tables on the same server,
      rather than between any two foreign tables with the same underlying FDW
      handler function.  In most if not all cases, the FDW would simply have had
      to apply the same-server restriction itself (far more expensively, both for
      lack of caching and because it would be repeated for each combination of
      input sub-joins), or else risk nasty bugs.  Anyone who's really intent on
      doing something outside this restriction can always use the
      set_join_pathlist_hook.
      
      * Rename fdw_ps_tlist/custom_ps_tlist to fdw_scan_tlist/custom_scan_tlist
      to better reflect what they're for, and allow these custom scan tlists
      to be used even for base relations.
      
      * Change make_foreignscan() API to include passing the fdw_scan_tlist
      value, since the FDW is required to set that.  Backwards compatibility
      doesn't seem like an adequate reason to expect FDWs to set it in some
      ad-hoc extra step, and anyway existing FDWs can just pass NIL.
      
      * Change the API of path-generating subroutines of add_paths_to_joinrel,
      and in particular that of GetForeignJoinPaths and set_join_pathlist_hook,
      so that various less-used parameters are passed in a struct rather than
      as separate parameter-list entries.  The objective here is to reduce the
      probability that future additions to those parameter lists will result in
      source-level API breaks for users of these hooks.  It's possible that this
      is even a small win for the core code, since most CPU architectures can't
      pass more than half a dozen parameters efficiently anyway.  I kept root,
      joinrel, outerrel, innerrel, and jointype as separate parameters to reduce
      code churn in joinpath.c --- in particular, putting jointype into the
      struct would have been problematic because of the subroutines' habit of
      changing their local copies of that variable.
      
      * Avoid ad-hocery in ExecAssignScanProjectionInfo.  It was probably all
      right for it to know about IndexOnlyScan, but if the list is to grow
      we should refactor the knowledge out to the callers.
      
      * Restore nodeForeignscan.c's previous use of the relcache to avoid
      extra GetFdwRoutine lookups for base-relation scans.
      
      * Lots of cleanup of documentation and missed comments.  Re-order some
      code additions into more logical places.
      1a8a4e5c
  4. Mar 22, 2015
    • Tom Lane's avatar
      Allow foreign tables to participate in inheritance. · cb1ca4d8
      Tom Lane authored
      Foreign tables can now be inheritance children, or parents.  Much of the
      system was already ready for this, but we had to fix a few things of
      course, mostly in the area of planner and executor handling of row locks.
      
      As side effects of this, allow foreign tables to have NOT VALID CHECK
      constraints (and hence to accept ALTER ... VALIDATE CONSTRAINT), and to
      accept ALTER SET STORAGE and ALTER SET WITH/WITHOUT OIDS.  Continuing to
      disallow these things would've required bizarre and inconsistent special
      cases in inheritance behavior.  Since foreign tables don't enforce CHECK
      constraints anyway, a NOT VALID one is a complete no-op, but that doesn't
      mean we shouldn't allow it.  And it's possible that some FDWs might have
      use for SET STORAGE or SET WITH OIDS, though doubtless they will be no-ops
      for most.
      
      An additional change in support of this is that when a ModifyTable node
      has multiple target tables, they will all now be explicitly identified
      in EXPLAIN output, for example:
      
       Update on pt1  (cost=0.00..321.05 rows=3541 width=46)
         Update on pt1
         Foreign Update on ft1
         Foreign Update on ft2
         Update on child3
         ->  Seq Scan on pt1  (cost=0.00..0.00 rows=1 width=46)
         ->  Foreign Scan on ft1  (cost=100.00..148.03 rows=1170 width=46)
         ->  Foreign Scan on ft2  (cost=100.00..148.03 rows=1170 width=46)
         ->  Seq Scan on child3  (cost=0.00..25.00 rows=1200 width=46)
      
      This was done mainly to provide an unambiguous place to attach "Remote SQL"
      fields, but it is useful for inherited updates even when no foreign tables
      are involved.
      
      Shigeru Hanada and Etsuro Fujita, reviewed by Ashutosh Bapat and Kyotaro
      Horiguchi, some additional hacking by me
      cb1ca4d8
  5. Mar 06, 2015
  6. Feb 21, 2015
  7. Jan 06, 2015
  8. Dec 17, 2014
    • Tom Lane's avatar
      Allow CHECK constraints to be placed on foreign tables. · fc2ac1fb
      Tom Lane authored
      As with NOT NULL constraints, we consider that such constraints are merely
      reports of constraints that are being enforced by the remote server (or
      other underlying storage mechanism).  Their only real use is to allow
      planner optimizations, for example in constraint-exclusion checks.  Thus,
      the code changes here amount to little more than removal of the error that
      was formerly thrown for applying CHECK to a foreign table.
      
      (In passing, do a bit of cleanup of the ALTER FOREIGN TABLE reference page,
      which had accumulated some weird decisions about ordering etc.)
      
      Shigeru Hanada and Etsuro Fujita, reviewed by Kyotaro Horiguchi and
      Ashutosh Bapat.
      fc2ac1fb
  9. Jul 14, 2014
  10. May 06, 2014
    • Bruce Momjian's avatar
      pgindent run for 9.4 · 0a783200
      Bruce Momjian authored
      This includes removing tabs after periods in C comments, which was
      applied to back branches, so this change should not effect backpatching.
      0a783200
  11. Apr 22, 2014
  12. Apr 18, 2014
    • Peter Eisentraut's avatar
      Create function prototype as part of PG_FUNCTION_INFO_V1 macro · e7128e8d
      Peter Eisentraut authored
      Because of gcc -Wmissing-prototypes, all functions in dynamically
      loadable modules must have a separate prototype declaration.  This is
      meant to detect global functions that are not declared in header files,
      but in cases where the function is called via dfmgr, this is redundant.
      Besides filling up space with boilerplate, this is a frequent source of
      compiler warnings in extension modules.
      
      We can fix that by creating the function prototype as part of the
      PG_FUNCTION_INFO_V1 macro, which such modules have to use anyway.  That
      makes the code of modules cleaner, because there is one less place where
      the entry points have to be listed, and creates an additional check that
      functions have the right prototype.
      
      Remove now redundant prototypes from contrib and other modules.
      e7128e8d
  13. Mar 04, 2014
    • Andrew Dunstan's avatar
      Provide a FORCE NULL option to COPY in CSV mode. · 3b5e03dc
      Andrew Dunstan authored
      This forces an input field containing the quoted null string to be
      returned as a NULL. Without this option, only unquoted null strings
      behave this way. This helps where some CSV producers insist on quoting
      every field, whether or not it is needed. The option takes a list of
      fields, and only applies to those columns. There is an equivalent
      column-level option added to file_fdw.
      
      Ian Barwick, with some tweaking by Andrew Dunstan, reviewed by Payal
      Singh.
      3b5e03dc
  14. Jan 07, 2014
  15. May 29, 2013
  16. Mar 10, 2013
    • Tom Lane's avatar
      Support writable foreign tables. · 21734d2f
      Tom Lane authored
      This patch adds the core-system infrastructure needed to support updates
      on foreign tables, and extends contrib/postgres_fdw to allow updates
      against remote Postgres servers.  There's still a great deal of room for
      improvement in optimization of remote updates, but at least there's basic
      functionality there now.
      
      KaiGai Kohei, reviewed by Alexander Korotkov and Laurenz Albe, and rather
      heavily revised by Tom Lane.
      21734d2f
  17. Feb 27, 2013
    • Heikki Linnakangas's avatar
      Add support for piping COPY to/from an external program. · 3d009e45
      Heikki Linnakangas authored
      This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding
      psql \copy syntax. Like with reading/writing files, the backend version is
      superuser-only, and in the psql version, the program is run in the client.
      
      In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you
      the stdin/stdout is quoted, it's now interpreted as a filename. For example,
      "\copy foo from 'stdin'" now reads from a file called 'stdin', not from
      standard input. Before this, there was no way to specify a filename called
      stdin, stdout, pstdin or pstdout.
      
      This creates a new function in pgport, wait_result_to_str(), which can
      be used to convert the exit status of a process, as returned by wait(3),
      to a human-readable string.
      
      Etsuro Fujita, reviewed by Amit Kapila.
      3d009e45
  18. Feb 06, 2013
  19. Jan 23, 2013
    • Alvaro Herrera's avatar
      Improve concurrency of foreign key locking · 0ac5ad51
      Alvaro Herrera authored
      This patch introduces two additional lock modes for tuples: "SELECT FOR
      KEY SHARE" and "SELECT FOR NO KEY UPDATE".  These don't block each
      other, in contrast with already existing "SELECT FOR SHARE" and "SELECT
      FOR UPDATE".  UPDATE commands that do not modify the values stored in
      the columns that are part of the key of the tuple now grab a SELECT FOR
      NO KEY UPDATE lock on the tuple, allowing them to proceed concurrently
      with tuple locks of the FOR KEY SHARE variety.
      
      Foreign key triggers now use FOR KEY SHARE instead of FOR SHARE; this
      means the concurrency improvement applies to them, which is the whole
      point of this patch.
      
      The added tuple lock semantics require some rejiggering of the multixact
      module, so that the locking level that each transaction is holding can
      be stored alongside its Xid.  Also, multixacts now need to persist
      across server restarts and crashes, because they can now represent not
      only tuple locks, but also tuple updates.  This means we need more
      careful tracking of lifetime of pg_multixact SLRU files; since they now
      persist longer, we require more infrastructure to figure out when they
      can be removed.  pg_upgrade also needs to be careful to copy
      pg_multixact files over from the old server to the new, or at least part
      of multixact.c state, depending on the versions of the old and new
      servers.
      
      Tuple time qualification rules (HeapTupleSatisfies routines) need to be
      careful not to consider tuples with the "is multi" infomask bit set as
      being only locked; they might need to look up MultiXact values (i.e.
      possibly do pg_multixact I/O) to find out the Xid that updated a tuple,
      whereas they previously were assured to only use information readily
      available from the tuple header.  This is considered acceptable, because
      the extra I/O would involve cases that would previously cause some
      commands to block waiting for concurrent transactions to finish.
      
      Another important change is the fact that locking tuples that have
      previously been updated causes the future versions to be marked as
      locked, too; this is essential for correctness of foreign key checks.
      This causes additional WAL-logging, also (there was previously a single
      WAL record for a locked tuple; now there are as many as updated copies
      of the tuple there exist.)
      
      With all this in place, contention related to tuples being checked by
      foreign key rules should be much reduced.
      
      As a bonus, the old behavior that a subtransaction grabbing a stronger
      tuple lock than the parent (sub)transaction held on a given tuple and
      later aborting caused the weaker lock to be lost, has been fixed.
      
      Many new spec files were added for isolation tester framework, to ensure
      overall behavior is sane.  There's probably room for several more tests.
      
      There were several reviewers of this patch; in particular, Noah Misch
      and Andres Freund spent considerable time in it.  Original idea for the
      patch came from Simon Riggs, after a problem report by Joel Jacobson.
      Most code is from me, with contributions from Marti Raudsepp, Alexander
      Shulgin, Noah Misch and Andres Freund.
      
      This patch was discussed in several pgsql-hackers threads; the most
      important start at the following message-ids:
      	AANLkTimo9XVcEzfiBR-ut3KVNDkjm2Vxh+t8kAmWjPuv@mail.gmail.com
      	1290721684-sup-3951@alvh.no-ip.org
      	1294953201-sup-2099@alvh.no-ip.org
      	1320343602-sup-2290@alvh.no-ip.org
      	1339690386-sup-8927@alvh.no-ip.org
      	4FE5FF020200002500048A3D@gw.wicourts.gov
      	4FEAB90A0200002500048B7D@gw.wicourts.gov
      0ac5ad51
  20. Jan 01, 2013
  21. Nov 12, 2012
  22. Aug 30, 2012
    • Alvaro Herrera's avatar
      Split tuple struct defs from htup.h to htup_details.h · c219d9b0
      Alvaro Herrera authored
      This reduces unnecessary exposure of other headers through htup.h, which
      is very widely included by many files.
      
      I have chosen to move the function prototypes to the new file as well,
      because that means htup.h no longer needs to include tupdesc.h.  In
      itself this doesn't have much effect in indirect inclusion of tupdesc.h
      throughout the tree, because it's also required by execnodes.h; but it's
      something to explore in the future, and it seemed best to do the htup.h
      change now while I'm busy with it.
      c219d9b0
  23. Jul 12, 2012
    • Tom Lane's avatar
      Skip text->binary conversion of unnecessary columns in contrib/file_fdw. · a36088bc
      Tom Lane authored
      When reading from a text- or CSV-format file in file_fdw, the datatype
      input routines can consume a significant fraction of the runtime.
      Often, the query does not need all the columns, so we can get a useful
      speed boost by skipping I/O conversion for unnecessary columns.
      
      To support this, add a "convert_selectively" option to the core COPY code.
      This is undocumented and not accessible from SQL (for now, anyway).
      
      Etsuro Fujita, reviewed by KaiGai Kohei
      a36088bc
  24. Jun 10, 2012
  25. May 17, 2012
  26. Apr 19, 2012
    • Tom Lane's avatar
      Revise parameterized-path mechanism to fix assorted issues. · 5b7b5518
      Tom Lane authored
      This patch adjusts the treatment of parameterized paths so that all paths
      with the same parameterization (same set of required outer rels) for the
      same relation will have the same rowcount estimate.  We cache the rowcount
      estimates to ensure that property, and hopefully save a few cycles too.
      Doing this makes it practical for add_path_precheck to operate without
      a rowcount estimate: it need only assume that paths with different
      parameterizations never dominate each other, which is close enough to
      true anyway for coarse filtering, because normally a more-parameterized
      path should yield fewer rows thanks to having more join clauses to apply.
      
      In add_path, we do the full nine yards of comparing rowcount estimates
      along with everything else, so that we can discard parameterized paths that
      don't actually have an advantage.  This fixes some issues I'd found with
      add_path rejecting parameterized paths on the grounds that they were more
      expensive than not-parameterized ones, even though they yielded many fewer
      rows and hence would be cheaper once subsequent joining was considered.
      
      To make the same-rowcounts assumption valid, we have to require that any
      parameterized path enforce *all* join clauses that could be obtained from
      the particular set of outer rels, even if not all of them are useful for
      indexing.  This is required at both base scans and joins.  It's a good
      thing anyway since the net impact is that join quals are checked at the
      lowest practical level in the join tree.  Hence, discard the original
      rather ad-hoc mechanism for choosing parameterization joinquals, and build
      a better one that has a more principled rule for when clauses can be moved.
      The original rule was actually buggy anyway for lack of knowledge about
      which relations are part of an outer join's outer side; getting this right
      requires adding an outer_relids field to RestrictInfo.
      5b7b5518
  27. Apr 06, 2012
    • Tom Lane's avatar
      Dept of second thoughts: improve the API for AnalyzeForeignTable. · cea49fe8
      Tom Lane authored
      If we make the initially-called function return the table physical-size
      estimate, acquire_inherited_sample_rows will be able to use that to
      allocate numbers of samples among child tables, when the day comes that
      we want to support foreign tables in inheritance trees.
      cea49fe8
    • Tom Lane's avatar
      Allow statistics to be collected for foreign tables. · 263d9de6
      Tom Lane authored
      ANALYZE now accepts foreign tables and allows the table's FDW to control
      how the sample rows are collected.  (But only manual ANALYZEs will touch
      foreign tables, for the moment, since among other things it's not very
      clear how to handle remote permissions checks in an auto-analyze.)
      
      contrib/file_fdw is extended to support this.
      
      Etsuro Fujita, reviewed by Shigeru Hanada, some further tweaking by me.
      263d9de6
  28. Mar 09, 2012
    • Tom Lane's avatar
      Revise FDW planning API, again. · b1495393
      Tom Lane authored
      Further reflection shows that a single callback isn't very workable if we
      desire to let FDWs generate multiple Paths, because that forces the FDW to
      do all work necessary to generate a valid Plan node for each Path.  Instead
      split the former PlanForeignScan API into three steps: GetForeignRelSize,
      GetForeignPaths, GetForeignPlan.  We had already bit the bullet of breaking
      the 9.1 FDW API for 9.2, so this shouldn't cause very much additional pain,
      and it's substantially more flexible for complex FDWs.
      
      Add an fdw_private field to RelOptInfo so that the new functions can save
      state there rather than possibly having to recalculate information two or
      three times.
      
      In addition, we'd not thought through what would be needed to allow an FDW
      to set up subexpressions of its choice for runtime execution.  We could
      treat ForeignScan.fdw_private as an executable expression but that seems
      likely to break existing FDWs unnecessarily (in particular, it would
      restrict the set of node types allowable in fdw_private to those supported
      by expression_tree_walker).  Instead, invent a separate field fdw_exprs
      which will receive the postprocessing appropriate for expression trees.
      (One field is enough since it can be a list of expressions; also, we assume
      the corresponding expression state tree(s) will be held within fdw_state,
      so we don't need to add anything to ForeignScanState.)
      
      Per review of Hanada Shigeru's pgsql_fdw patch.  We may need to tweak this
      further as we continue to work on that patch, but to me it feels a lot
      closer to being right now.
      b1495393
  29. Mar 08, 2012
    • Tom Lane's avatar
      Add GetForeignColumnOptions() to foreign.c, and add some documentation. · 9088d1b9
      Tom Lane authored
      GetForeignColumnOptions provides some abstraction for accessing
      column-specific FDW options, on a par with the access functions that were
      already provided here for other FDW-related information.
      
      Adjust file_fdw.c to use GetForeignColumnOptions instead of equivalent
      hand-rolled code.
      
      In addition, add some SGML documentation for the functions exported by
      foreign.c that are meant for use by FDW authors.
      
      (This is the fdw_helper portion of the proposed pgsql_fdw patch.)
      
      Hanada Shigeru, reviewed by KaiGai Kohei
      9088d1b9
  30. Mar 05, 2012
    • Tom Lane's avatar
      Redesign PlanForeignScan API to allow multiple paths for a foreign table. · 6b289942
      Tom Lane authored
      The original API specification only allowed an FDW to create a single
      access path, which doesn't seem like a terribly good idea in hindsight.
      Instead, move the responsibility for building the Path node and calling
      add_path() into the FDW's PlanForeignScan function.  Now, it can do that
      more than once if appropriate.  There is no longer any need for the
      transient FdwPlan struct, so get rid of that.
      
      Etsuro Fujita, Shigeru Hanada, Tom Lane
      6b289942
  31. Jan 02, 2012
  32. Oct 12, 2011
    • Tom Lane's avatar
      Throw a useful error message if an extension script file is fed to psql. · 458857cc
      Tom Lane authored
      We have seen one too many reports of people trying to use 9.1 extension
      files in the old-fashioned way of sourcing them in psql.  Not only does
      that usually not work (due to failure to substitute for MODULE_PATHNAME
      and/or @extschema@), but if it did work they'd get a collection of loose
      objects not an extension.  To prevent this, insert an \echo ... \quit
      line that prints a suitable error message into each extension script file,
      and teach commands/extension.c to ignore lines starting with \echo.
      That should not only prevent any adverse consequences of loading a script
      file the wrong way, but make it crystal clear to users that they need to
      do it differently now.
      
      Tom Lane, following an idea of Andrew Dunstan's.  Back-patch into 9.1
      ... there is not going to be much value in this if we wait till 9.2.
      458857cc
  33. Sep 16, 2011
  34. Jul 06, 2011
  35. Jul 04, 2011
  36. Apr 25, 2011
    • Peter Eisentraut's avatar
      Support "make check" in contrib · f8ebe3bc
      Peter Eisentraut authored
      Added a new option --extra-install to pg_regress to arrange installing
      the respective contrib directory into the temporary installation.
      This is currently not yet supported for Windows MSVC builds.
      
      Updated the .gitignore files for contrib modules to ignore the
      leftovers of a temp-install check run.
      
      Changed the exit status of "make check" in a pgxs build (which still
      does nothing) to 0 from 1.
      
      Added "make check" in contrib to top-level "make check-world".
      f8ebe3bc
  37. Apr 10, 2011
  38. Feb 21, 2011
    • Itagaki Takahiro's avatar
      Add ENCODING option to COPY TO/FROM and file_fdw. · 3cba8240
      Itagaki Takahiro authored
      File encodings can be specified separately from client encoding.
      If not specified, client encoding is used for backward compatibility.
      
      Cases when the encoding doesn't match client encoding are slower
      than matched cases because we don't have conversion procs for other
      encodings. Performance improvement would be be a future work.
      
      Original patch by Hitoshi Harada, and modified by me.
      3cba8240
Loading