Changes needed to build at all:

- Check for SSL_new in configure, now that SSL_library_init is a macro.
- Do not access struct members directly. This includes some new code in
  pgcrypto, to use the resource owner mechanism to ensure that we don't
  leak OpenSSL handles, now that we can't embed them in other structs
  anymore.
- RAND_SSLeay() -> RAND_OpenSSL()

Changes that were needed to silence deprecation warnings, but were not
strictly necessary:

- RAND_pseudo_bytes() -> RAND_bytes().
- SSL_library_init() and OpenSSL_config() -> OPENSSL_init_ssl()
- ASN1_STRING_data() -> ASN1_STRING_get0_data()
- DH_generate_parameters() -> DH_generate_parameters()
- Locking callbacks are not needed with OpenSSL 1.1.0 anymore. (Good
  riddance!)

Also change references to SSLEAY_VERSION_NUMBER with OPENSSL_VERSION_NUMBER,
for the sake of consistency. OPENSSL_VERSION_NUMBER has existed since time
immemorial.

Fix SSL test suite to work with OpenSSL 1.1.0. CA certificates must have
the "CA:true" basic constraint extension now, or OpenSSL will refuse them.
Regenerate the test certificates with that. The "openssl" binary, used to
generate the certificates, is also now more picky, and throws an error
if an X509 extension is specified in "req_extensions", but that section
is empty.

Backpatch to all supported branches, per popular demand. In back-branches,
we still support OpenSSL 0.9.7 and above. OpenSSL 0.9.6 should still work
too, but I didn't test it. In master, we only support 0.9.8 and above.

Patch by Andreas Karlsson, with additional changes by me.

Discussion: <20160627151604.GD1051@msg.df7cb.de>

This has been the predominant outcome.  When the output of decrypting
with a wrong key coincidentally resembled an OpenPGP packet header,
pgcrypto could instead report "Corrupt data", "Not text data" or
"Unsupported compression algorithm".  The distinct "Corrupt data"
message added no value.  The latter two error messages misled when the
decrypted payload also exhibited fundamental integrity problems.  Worse,
error message variance in other systems has enabled cryptologic attacks;
see RFC 4880 section "14. Security Considerations".  Whether these
pgcrypto behaviors are likewise exploitable is unknown.

In passing, document that pgcrypto does not resist side-channel attacks.
Back-patch to 9.0 (all supported versions).

Security: CVE-2015-3167

This improves on commit bbfd7eda by
making two simple changes:

* pg_attribute_noreturn now takes parentheses, ie pg_attribute_noreturn().
Likewise pg_attribute_unused(), pg_attribute_packed().  This reduces
pgindent's tendency to misformat declarations involving them.

* attributes are now always attached to function declarations, not
definitions.  Previously some places were taking creative shortcuts,
which were not merely candidates for bad misformatting by pgindent
but often were outright wrong anyway.  (It does little good to put a
noreturn annotation where callers can't see it.)  In any case, if
we would like to believe that these macros can be used with non-gcc
compilers, we should avoid gratuitous variance in usage patterns.

I also went through and manually improved the formatting of a lot of
declarations, and got rid of excessively repetitive (and now obsolete
anyway) comments informing the reader what pg_attribute_printf is for.

Until now __attribute__() was defined to be empty for all compilers but
gcc. That's problematic because it prevents using it in other compilers;
which is necessary e.g. for atomics portability.  It's also just
generally dubious to do so in a header as widely included as c.h.

Instead add pg_attribute_format_arg, pg_attribute_printf,
pg_attribute_noreturn macros which are implemented in the compilers that
understand them. Also add pg_attribute_noreturn and pg_attribute_packed,
but don't provide fallbacks, since they can affect functionality.

This means that external code that, possibly unwittingly, relied on
__attribute__ defined to be empty on !gcc compilers may now run into
warnings or errors on those compilers. But there shouldn't be many
occurances of that and it's hard to work around...

Discussion: 54B58BA3.8040302@ohmu.fi
Author: Oskari Saarenmaa, with some minor changes by me.

This includes removing tabs after periods in C comments, which was
applied to back branches, so this change should not effect backpatching.

Specifically, on-stack memset() might be removed, so:

	* Replace memset() with px_memset()
	* Add px_memset to copy_crlf()
	* Add px_memset to pgp-s2k.c

Patch by Marko Kreen

Report by PVS-Studio

Backpatch through 8.4.

commit-fest.

Add __attribute__ decorations for printf format checking to the places that
were missing them.  Fix the resulting warnings.  Add
-Wmissing-format-attribute to the standard set of warnings for GCC, so these
don't happen again.

The warning fixes here are relatively harmless.  The one serious problem
discovered by this was already committed earlier in
cf15fb5c.

provided by Andrew.

padded encryption scheme.  Formerly it would try to access res[(unsigned) -1],
which resulted in core dumps on 64-bit machines, and was certainly trouble
waiting to happen on 32-bit machines (though in at least the known case
it was harmless because that byte would be overwritten after return).
Per report from Ken Colson; fix by Marko Kreen.

right, there seems precious little reason to have a pile of hand-maintained
endianness definitions in src/include/port/*.h.  Get rid of those, and make
the couple of places that used them depend on WORDS_BIGENDIAN instead.

the pubkey functions a bit.  The actual RSA-specific code
there is tiny, most of the patch consists of reorg of the
pubkey code, as lots of it was written as elgamal-only.

---------------------------------------------------------------------------

The SHLIB section was copy-pasted from somewhere and contains
several unnecessary libs.  This cleans it up a bit.

 -lcrypt
   we don't use system crypt()

 -lssl, -lssleay32
   no SSL here

 -lz in win32 section
   already added on previous line

 -ldes
   The chance anybody has it is pretty low.
   And the chance pgcrypto works with it is even lower.

Also trim the win32 section.

---------------------------------------------------------------------------

It is already disabled in Makefile, remove code too.

---------------------------------------------------------------------------

I was bit hasty making the random exponent 'k' a prime.  Further researh
shows that Elgamal encryption has no specific needs in respect to k,
any random number is fine.

It is bit different for signing, there it needs to be 'relatively prime'
to p - 1,  that means GCD(k, p-1) == 1, which is also a lot lighter than
full primality.  As we don't do signing, this can be ignored.

This brings major speedup to Elgamal encryption.

---------------------------------------------------------------------------

o  pgp_mpi_free: Accept NULLs
o  pgp_mpi_cksum: result should be 16bit
o  Remove function name from error messages - to be similar to other
   SQL functions, and it does not match anyway the called function
o  remove couple junk lines

---------------------------------------------------------------------------

o  Support for RSA encryption
o  Big reorg to better separate generic and algorithm-specific code.
o  Regression tests for RSA.

---------------------------------------------------------------------------

o  Tom stuck a CVS id into file.  I doubt the usefulness of it,
   but if it needs to be in the file then rather at the end.
   Also tag it as comment for asciidoc.
o  Mention bytea vs. text difference
o  Couple clarifications

---------------------------------------------------------------------------

There is a choice whether to update it with pgp functions or
remove it.  I decided to remove it, updating is pointless.

I've tried to keep the core of pgcrypto relatively independent
from main PostgreSQL, to make it easy to use externally if needed,
and that is good.  Eg. that made development of PGP functions much
nicer.

But I have no plans to release it as generic library, so keeping such
doc
up-to-date is waste of time.  If anyone is interested in using it in
other products, he can probably bother to read the source too.

Commented source is another thing - I'll try to make another pass
over code to see if there is anything non-obvious that would need
more comments.

---------------------------------------------------------------------------

Marko Kreen

of password-based encryption from RFC2440 (OpenPGP).

The goal of this code is to be more featureful encryption solution
than current encrypt(), which only functionality is running cipher
over data.

Compared to encrypt(), pgp_encrypt() does following:

* It uses the equvialent of random Inital Vector to get cipher
  into random state before it processes user data
* Stores SHA-1 of the data into result so any modification
  will be detected.
* Remembers if data was text or binary - thus it can decrypt
  to/from text data.  This was a major nuisance for encrypt().
* Stores info about used algorithms with result, so user needs
  not remember them - more user friendly!
* Uses String2Key algorithms (similar to crypt()) with random salt
  to generate full-length binary key to be used for encrypting.
* Uses standard format for data - you can feed it to GnuPG, if needed.

Optional features (off by default):

* Can use separate session key - user data will be encrypted
  with totally random key, which will be encrypted with S2K
  generated key and attached to result.
* Data compression with zlib.
* Can convert between CRLF<->LF line-endings - to get fully
  RFC2440-compliant behaviour.  This is off by default as
  pgcrypto does not know the line-endings of user data.

Interface is simple:


    pgp_encrypt(data text, key text) returns bytea
    pgp_decrypt(data text, key text) returns text
    pgp_encrypt_bytea(data bytea, key text) returns bytea
    pgp_decrypt_bytea(data bytea, key text) returns bytea

To change parameters (cipher, compression, mdc):

    pgp_encrypt(data text, key text, parms text) returns bytea
    pgp_decrypt(data text, key text, parms text) returns text
    pgp_encrypt_bytea(data bytea, key text, parms text) returns bytea
    pgp_decrypt_bytea(data bytea, key text, parms text) returns bytea

Parameter names I lifted from gpg:

   pgp_encrypt('message', 'key', 'compress-algo=1,cipher-algo=aes256')

For text data, pgp_encrypt simply encrypts the PostgreSQL internal data.

This maps to RFC2440 data type 't' - 'extenally specified encoding'.
But this may cause problems if data is dumped and reloaded into database
which as different internal encoding.  My next goal is to implement data
type 'u' - which means data is in UTF-8 encoding by converting internal
encoding to UTF-8 and back.  And there wont be any compatibility
problems with current code, I think its ok to submit this without UTF-8
encoding by converting internal encoding to UTF-8 and back.  And there
wont be any compatibility problems with current code, I think its ok to
submit this without UTF-8 support.


Here is v4 of PGP encrypt.  This depends on previously sent
Fortuna-patch, as it uses the px_add_entropy function.

- New function: pgp_key_id() for finding key id's.
- Add SHA1 of user data and key into RNG pools.  We need to get
  randomness from somewhere, and it is in user best interests
  to contribute.
- Regenerate pgp-armor test for SQL_ASCII database.
- Cleanup the key handling so that the pubkey support is less
  hackish.

Marko Kreen

- Move openssl random provider to openssl.c and builtin provider
  to internal.c
- Make px_random_bytes use Fortuna, instead of giving error.
- Retarget random.c to aquiring system randomness, for initial seeding
  of Fortuna.  There is ATM 2 functions for Windows,
  reader from /dev/urandom and the regular time()/getpid() silliness.

Marko Kreen

Reserve px_get_random_bytes() for strong randomness,
add new function px_get_pseudo_random_bytes() for
weak randomness and use it in gen_salt().

On openssl case, use RAND_pseudo_bytes() for
px_get_pseudo_random_bytes().

Final result is that is user has not configured random
souce but kept the 'silly' one, gen_salt() keeps
working, but pgp_encrypt() will throw error.

Marko Kreen

* Use error codes instead of -1
* px_strerror for new error codes
* calling convention change for px_gen_salt - return error code
* use px_strerror in pgcrypto.c

Marko Kreen

  It was a bad style to begin with, and now several loops can be clearer.
* pgcrypto.c: Fix function comments
* crypt-gensalt.c, crypt-blowfish.c: stop messing with errno
* openssl.c: use px_free instead pfree
* px.h: make redefining px_alloc/px_realloc/px_free easier

Marko Kreen

make sure the $Id tags are converted to $PostgreSQL as well ...

  everywhere.

At least it was now detected correctly.

marko

  produces garbage.

I learned the hard way that

        #if UNDEFINED_1 == UNDEFINED_2
        #error "gcc is idiot"
        #endif

prints "gcc is idiot" ...

Affected are MD5/SHA1 in internal library, and also HMAC-MD5/HMAC-SHA1/
crypt-md5 which use them.  Blowfish is ok, also Rijndael on at
least x86.

Big thanks to Daniel Holtzman who send me a build log which
contained warning:

        md5.c:246: warning: `X' defined but not used

Yes, gcc is that helpful...

Please apply this.

--
marko

failures on FreeBSD.  This patch replaces uint -> unsigned.

This was reported by Daniel Holtzman against 0.4pre3 standalone
package, but it needs fixing in contrib/pgcrypto too.

Marko Kreen

initdb/regression tests pass.

tests pass.

salt generation code.  He also urged using better random source
and making possible to choose using bcrypt and xdes rounds more
easily.  So, here's patch:

* For all salt generation, use Solar Designer's own code.  This
  is mostly due fact that his code is more fit for get_random_bytes()
  style interface.
* New function: gen_salt(type, rounds).  This lets specify iteration
  count for algorithm.
* random.c: px_get_random_bytes() function.
  Supported randomness soure: /dev/urandom, OpenSSL PRNG, libc random()
  Default: /dev/urandom.
* Draft description of C API for pgcrypto functions.

New files: API, crypt-gensalt.c, random.c

Marko Kreen