From f7672c8ce26582f250f7dce543a998ac9d1d6665 Mon Sep 17 00:00:00 2001
From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Wed, 7 May 2014 21:38:41 -0400
Subject: [PATCH] Avoid buffer bloat in libpq when server is consistently
faster than client.
If the server sends a long stream of data, and the server + network are
consistently fast enough to force the recv() loop in pqReadData() to
iterate until libpq's input buffer is full, then upon processing the last
incomplete message in each bufferload we'd usually double the buffer size,
due to supposing that we didn't have enough room in the buffer to finish
collecting that message. After filling the newly-enlarged buffer, the
cycle repeats, eventually resulting in an out-of-memory situation (which
would be reported misleadingly as "lost synchronization with server").
Of course, we should not enlarge the buffer unless we still need room
after discarding already-processed messages.
This bug dates back quite a long time: pqParseInput3 has had the behavior
since perhaps 2003, getCopyDataMessage at least since commit 70066eb1a1ad
in 2008. Probably the reason it's not been isolated before is that in
common environments the recv() loop would always be faster than the server
(if on the same machine) or faster than the network (if not); or at least
it wouldn't be slower consistently enough to let the buffer ramp up to a
problematic size. The reported cases involve Windows, which perhaps has
different timing behavior than other platforms.
Per bug #7914 from Shin-ichi Morita, though this is different from his
proposed solution. Back-patch to all supported branches.
---
src/interfaces/libpq/fe-misc.c | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index e156bb2155b..1575675ad94 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -352,6 +352,7 @@ pqCheckOutBufferSpace(size_t bytes_needed, PGconn *conn)
int newsize = conn->outBufSize;
char *newbuf;
+ /* Quick exit if we have enough space */
if (bytes_needed <= (size_t) newsize)
return 0;
@@ -415,6 +416,37 @@ pqCheckInBufferSpace(size_t bytes_needed, PGconn *conn)
int newsize = conn->inBufSize;
char *newbuf;
+ /* Quick exit if we have enough space */
+ if (bytes_needed <= (size_t) newsize)
+ return 0;
+
+ /*
+ * Before concluding that we need to enlarge the buffer, left-justify
+ * whatever is in it and recheck. The caller's value of bytes_needed
+ * includes any data to the left of inStart, but we can delete that in
+ * preference to enlarging the buffer. It's slightly ugly to have this
+ * function do this, but it's better than making callers worry about it.
+ */
+ bytes_needed -= conn->inStart;
+
+ if (conn->inStart < conn->inEnd)
+ {
+ if (conn->inStart > 0)
+ {
+ memmove(conn->inBuffer, conn->inBuffer + conn->inStart,
+ conn->inEnd - conn->inStart);
+ conn->inEnd -= conn->inStart;
+ conn->inCursor -= conn->inStart;
+ conn->inStart = 0;
+ }
+ }
+ else
+ {
+ /* buffer is logically empty, reset it */
+ conn->inStart = conn->inCursor = conn->inEnd = 0;
+ }
+
+ /* Recheck whether we have enough space */
if (bytes_needed <= (size_t) newsize)
return 0;
--
GitLab