Passwords: they just seem to work. You connect to your PostgreSQL database and you are prompted for your password. You type in the correct character combination, and presto! you're in, safe and sound. But what if I told you that all was not as it seemed. What if I told you there was a better, safer way to use passwords with PostgreSQL? What if I told you it was imperative that you upgraded, too? PostgreSQL 10 introduced SCRAM (Salted Challenge Response Authentication Mechanism), introduced in RFC 5802, as a way to securely authenticate passwords. The SCRAM algorithm lets a client and server validate a password without ever sending the password, whether plaintext or a hashed form of it, to each other, using a series of cryptographic methods. In this talk, we will look at: * A history of the evolution of password storage and authentication in PostgreSQL * How SCRAM works with a step-by-step deep dive into the algorithm (and convince you why you need to upgrade!) * SCRAM channel binding, which helps prevent MITM attacks during authentication * How to safely set and modify your passwords, as well as how to upgrade to SCRAM-SHA-256 (which we will do live!) all of which will be explained by some adorable elephants and hippos! At the end of this talk, you will understand how SCRAM works, how to ensure your PostgreSQL drivers supports it, how to upgrade your passwords to using SCRAM-SHA-256, and why you want to tell other PostgreSQL password mechanisms to SCRAM!

1. Get Your Insecure Passwords
to SCRAM
Jonathan S. Katz PGCon 2020 May 6, 2020

2. aka "A Tale of Two Hippos"
2

3. " …and an Elephant"
3

4. " …Guest Starring Blue Elephant"
4

5. • Vice President of Platform Engineering,
Crunchy Data
• Previously: Engineering leadership in startups
• Longtime PostgreSQL community contributor
• Advocacy & various committees for PGDG
• @postgresql + .org content
• Director, PgUS
• Conference organization + speaking
• @jkatz05
About Me
5

6. Market Leading
Data Security
Crunchy Data is the leader in
PostgreSQL security. Common Criteria
certification and essential security
enhancements make Crunchy Certified
PostgreSQL the trusted open source
PostgreSQL distribution for
the enterprise.
Cloud Ready
Data Management
Whether deploying to public or private
clouds, Crunchy Data provides market
leading, open source, Kubernetes-
based technology solutions, giving
your team the choice and flexibility for
how you deploy your data.
Leader in Open Source
Enterprise PostgreSQL
Crunchy Data gives organizations the
technology, support, and confidence to
enjoy the power and efficiency of open
source PostgreSQL.

7. • You’ll understand the evolution of how password management in
PostgreSQL
• If you use PostgreSQL’s password mechanisms in production, you should
want to start using SCRAM-SHA-256
• You will know how to upgrade your systems to use SCRAM-SHA-256
By The End of This Talk
7

8. Let’s Log Into PostgreSQL!

9. 9
$ psql -h localhost app
Password for user jkatz:
psql: FATAL: password authentication failed
for user "jkatz"

10. 10
$ psql -h localhost –U jkatz app
Password for user jkatz:
psql (12.2)
Type "help" for help.
app=#

11. It Just Works!

12. …but, what just works?

13. • It’s very easy to take password management for granted when it just works
• Before version 10, offered two methods of storing passwords and
validating passwords
• plain
• crypt
• md5
How Do Passwords Work in PostgreSQL?
13

14. Some Helpers
14
Clients Servers

15. Meet grayhippo
15
This is "grayhippo"
grayhippo loves PostgreSQL,
and loves storing large
amounts of data.
grayhippo's password is
"datalake"

16. Meet redhippo
16
This is "redhippo"
redhippo also loves
PostgreSQL. redhippo likes
to play pretend, and in
particular, likes to pretend to
be grayhippo.

17. • Stored in plain text
• Communicated in plain text
• Method was kept available while drivers
were updating to “md5” method.
• Dropped in PostgreSQL 10*
• “Plaintext” validation is still requested for some
methods, such as LDAP
PostgreSQL Plaintext Passwords
17
password

18. Plaintext Password Authentication Flow
18
Hi! I want to
authenticate.
I’m “grayhippo”

19. Plaintext Password Authentication Flow
19
Hi grayhippo!
Please tell me
your password

20. Plaintext Password Authentication Flow
20
My password
is “datalake”

21. Plaintext Password Authentication Flow
21
Confirmed!
You can
access
PostgreSQL
as “grayhippo”

22. And all was well.

23. …except we know it’s not.

24. The Eavesdropper
24
My password
is “datalake”

25. What Could Go Wrong?
25
Hi! I want to
authenticate.
I’m “grayhippo”

26. …
26
Hi grayhippo!
Please tell me
your password

27. …
27
My password
is “datalake”

28. …
28
Confirmed!
You can
access
PostgreSQL
as “grayhippo”

29. …oh
29
LOL! I got your
data

30. What About TLS?

31. • Supersedes Securely Socket Layers (SSL)
• PostgreSQL refers to “SSL” everywhere in the configuration
• Performs a “secure handshake” between two parties and encrypts all
traffic between them
• Use the “hostssl” authentication type to require TLS connections
• Can be used as a method of authentication in PostgreSQL…but that’s a
different talk
• (PostgreSQL 12: clientert=verify-full – allows for two-factor authentication using
certificates + another authentication method!)
Transport Layer Security
31

32. The Eavesdropper…With TLS
32
My password
is “datalake”
T
L
S

33. 33
…

34. 34
💡

35. If At First You Don’t Succeed…
35
Hey, can you
get me a dump
of the entire
database?

36. …
36
pg_dumpall

37. 37
--
-- Roles
--
CREATE ROLE grayhippo;ALTER ROLE grayhippo WITH NOSUPERUSER
INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION
NOBYPASSRLS PASSWORD 'datalake';
…and if we inspect the file

38. Don’t Forget Manners!
38
Thanks! Oh your
password is in this
file, but don’t worry I
won’t do anything
bad.

39. Perhaps MD5 Can Do Better

40. • Derived by computing md5 hash of the
combination of the plaintext password
and the username
• Stored in hexadecimal format with "md5"
prepended to it
• Communicated using the PostgreSQL
"md5 protocol"
• PostgreSQL server sends a 4 byte random
salt
• Connecting client computes md5 hash of
password + username, then appends the 4
byte salt and computes another md5 hash
PostgreSQL MD5 Password Authentication
40
"md5" +
md5(password + username)

41. MD5 Authentication Flow
41
Hi! I want to
authenticate.
I’m “grayhippo”

42. MD5 Authentication Flow
42
Hi grayhippo!
Let’s use the MD5
method. Here are
some random
bytes: “r4nd”

43. MD5 Authentication Flow
43
"md5" + md5("datalake" + "grayhippo") =
md55133cd3d6af5b408fa0bd7fbbde1dedd

44. MD5 Authentication Flow
44
"md5" + md5("md55133cd3d6af5b408fa0bd7fbbde1dedd " + "r4nd") =
md558dc9038c1b6284fabdb4eefa421a881

45. MD5 Authentication Flow
45
"md558dc9038c1b6284fabdb4eefa421a881"

46. MD5 Authentication Flow
46
Confirmed!
You can
access
PostgreSQL
as “grayhippo”

47. Is redhippo Out Of Luck?

48. Does Eavesdropping Work?
48
“md558dc9038c1b6284fabdb4eefa421a881”

49. 49
Hi! I want to
authenticate.
I’m “grayhippo”

50. 50
Hi grayhippo!
Let’s use the MD5
method. Here are
some random
bytes: "n0n0"

51. 51
“md558dc9038c1b6284fabdb4eefa421a881”

52. 52
No, that's
incorrect for
grayhippo,
sorry.

53. Nope!
53
Meh.

54. If At First You Don’t Succeed…
54
Hey, can you
get me a dump
of the entire
database?

55. …
55
pg_dumpall

56. 56
--
-- Roles
--
CREATE ROLE grayhippo;ALTER ROLE grayhippo WITH NOSUPERUSER
INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION
NOBYPASSRLS PASSWORD 'md55133cd3d6af5b408fa0bd7fbbde1dedd';
…and if we inspect the file

57. 57
Hi! I want to
authenticate.
I’m “grayhippo”

58. 58
Hi grayhippo!
Let’s use the MD5
method. Here are
some random
bytes: "n0n0"

59. 59
"md5" + md5("md55133cd3d6af5b408fa0bd7fbbde1dedd " + "n0n0") =
md5863c562e50cbee7c19662191fe737157

60. 60
"md5863c562e50cbee7c19662191fe737157"

61. 61
Confirmed! You
can access
PostgreSQL as
“grayhippo”

62. 62
Really? LOL!

63. • CREATE ROLE grayhippo PASSWORD 'datalake';
• If redhippo can read the server logs, redhippos now has the plain text password
• Alternatives
• Via psql: Use password
• Via other: pre-hash the password with the formula: "md5" + md5(password + username)
• …but both of these methods still log the md5 hashed password to the server logs
• SELECT rolname, rolpassword FROM pg_authid;
• Requires a privileged user, but if redhippo has said privileges…
Other Ways redhippo Can Get the Password
63

64. These Password Methods
Need to SCRAM!

65. • Defined in RFC 5802
• Part of the Simple Authentication and Security Layer (SASL) family (RFC 4422)
• PostgreSQL uses RFC7767 which specifies the use of the SHA-256 hashing algorithm
• Designed to be easy to implement and use all standard, accepted methods
• SHA-256 (RFC 6234)
• HMAC (RFC 2014)
• SASLprep (RFC 4013), based on stringprep (RFC 3434)
• …and we'll talk about how PostgreSQL does this
• Allows for two parties to verify they both know a secret without
exchanging the secret
Salted Challenge Response Authentication Mechanism
65

66. Two Parties Can Verify They
Both Know a Secret
Without Exchanging the
Secret

68. • In PostgreSQL need to build a "SCRAM verifier"
• A client can build the SCRAM verifier and transmit it to the server.
• Even in this form, an eavesdropper will be unable to access the secret
Creating a Password For SCRAM
68
<DIGEST>$<ITERATIONS>:<SALT>$<STORED_KEY>:<SERVER_KE
Y>

69. • PostgreSQL 10 introduced support for "SCRAM-SHA-256"
• PostgreSQL 11 introduced support for "SCRAM-SHA-256-PLUS", which is
used for channel binding (more on that later)
Building a SCRAM Secret - DIGEST
69
"SCRAM-SHA-256"

70. • Iterations is the number of times the HMAC signature is computed with
hashing function (SCRAM-SHA-256)
• We'll start talking about the full computation of this in two slides
• By default, PostgreSQL uses 4096 iterations. If you built your own secret,
you can customize the number of iterations.
Building a SCRAM Secret - ITERATIONS
70
"4096"

71. • A cryptographically secure randomly generated value
• PostgreSQL defaults to 16 bytes
• Ultimately stored in base64 representation
Building a SCRAM Secret - SALT
71
"s+1VLTv5oCfNEymVKi01Fw=="

72. • Before creating the hashed password, it needs to be normalized using SASLPrep*
• *SASLprep is for UTF-8 encoded strings, but PostgreSQL accepts multiple encodings…so we perform
a modified SASLprep
• If password is ASCII or is not UTF-8 encoded, this step is skipped
• 4 Steps:
• Step 1: Remove non-ASCII space characters. If password ends up being empty, then return original
password
• Step 2: Normalize the password using NFKC form. If empty, then return original password
• Step 3: Check for prohibited characters. If any are detected, then return original password
• Step 4: Bi-directional characters. If any prohibited Bi-directional characters detected, return original
password. If there is a "RandALCat" character and either a) at least one "LCat" character OR b) does
not end with a "RandALCat" character, return original password
• If we get through that, return normalized password!
Building a SCRAM Secret – SASLPrep the Password
72
It's still
"datalake"

73. • The first iteration for generating the salted password uses the following
formula:
• HMAC using SHA-256 with "password" as the key signing a message with "salt"
concatenated with a 32-bit value with final bit set to 1
• Store this value as an aggregator
• For the remaining iterations:
• Calculate the HMAC using SHA-256 with "password" as the key signing a message
that is composed of HMAC calculated from the previous iteration
• XOR this HMAC with the aggregator (aggregator = aggregator XOR HMAC)
Building a SCRAM Secret – Generate the Salted Password
73
In base64 for this example, the value is:
"oJZnL+tf7yE1QWp5fykec7xEgw1cwfyr5Jh3SudbWio="

74. • The SHA-256 hash of the "Client Key", stored in a base64 representation
• …the "Client Key" is a HMAC using the salted password as the key and
"Client Key" as the message
Building a SCRAM Secret – STORED KEY
74
"z+tqsnBaCmgSJQBLf1cPOmq7n80PW3aRjzeuAGkkuq4="

75. Stored Key
75
Plaintext Password
Salted Hashed
Password Client Key
Salt
Hash
Iterations
HMAC HMAC
"Client Key"
SHA-256 Stored Key

76. • A HMAC using the salted password as the key and "Server Key" as the
message
Building a SCRAM Secret – SERVER KEY
76
"9ZWHsv+XZAmrpkiEnDPm05wg55jLRrITWspTqsz1zD4="

77. Server Key
77
Plaintext Password
Salted Hashed
Password
Server Key
Salt
Hash
Iterations
HMAC HMAC
"Server Key"

78. • You can just use "password" from psql
• If your PostgreSQL driver supports it, its password creation facilities
• It may delegate it to libpq, which has said facilities
Building a SCRAM Secret – "Easy Button"
78
That sounds
much simpler…

79. Send the SCRAM Secret to PostgreSQL
79
ALTER ROLE grayhippo
PASSWORD 'SCRAM-SHA-
256$4096:s+1VLTv5oCfNEymVKi01Fw==$z+tqsnBaCmgSJQBLf1cPOmq7n
80PW3aRjzeuAGkkuq4=:9ZWHsv+XZAmrpkiEnDPm05wg55jLRrITWspTqsz
1zD4=';
(We'll let redhippo listen)

80. • Identifying information sent:
• Stored Key: A hash of a HMAC using the computed salted password and "Client Key"
• Server Key: A HMAC using the computed salted password and "Server Key"
• In other words, we did not send the original secret in any recognizable form
• redhippo can still try to brute force the password or try other offline attacks,
but with a high number of iterations (and good password selection), this is
hard.
We Don't Care That redhippo Listens?
80
Just wait until my
quantum computer
boots up…

81. What About Verification?

82. • The goal of SCRAM authentication is for two parties to verify that each ones
knows the shared secret
• To prevent against replay attacks, one-time nonces are used for the session
being authenticated
SCRAM Authentication
82
Just to save space,
pretend I'm listening
on the next few
slides

83. SCRAM Authentication Flow
83
Hi! I want to
authenticate.
I’m “grayhippo”

84. SCRAM Authentication Flow
84
Hi grayhippo! Let’s use the SASL
method. Here are some available
methods: "SCRAM-SHA-256"

85. SCRAM Authentication Flow
85
OK. I choose "SCRAM-SHA-256".
Here is a nonce I generated:
"FVazuC8Hjl46XPPCs2L9RFhq"

86. SCRAM Authentication Flow
86
Cool. Here is my nonce added to your nonce:
"FVazuC8Hjl46XPPCs2L9RFhqMxMjEzh5txq6gZv6iuW813Aa"
Here is the number of iterations to use to computed a salted password: "4096"
Here is the salt for you to use: "s+1VLTv5oCfNEymVKi01Fw=="
Send me proof that you know the password.

87. This is the magical step!

88. SCRAM Authentication Flow: Generating Proof
88
My password is "datalake" so from that I am going to computed
the salted password using the "SHA-256" hashing method, the
"s+1VLTv5oCfNEymVKi01Fw==" salt, and applying the method
for "4096" iterations.
Basically, like generating the salted hash password.
Which was
"oJZnL+tf7yE1QWp5fykec7xEgw1cwfyr5Jh3SudbWio="
in base64.

89. Recall: Client Key
89
Plaintext Password
Salted Hashed
Password
Client Key
Salt
Hash
Iterations
HMAC HMAC
"Client Key"

90. Recall: Stored Key
90
Plaintext Password
Salted Hashed
Password Client Key
Salt
Hash
Iterations
HMAC HMAC
"Client Key"
SHA-256 Stored Key

91. SCRAM Authentication Flow: Generating Proof
91
From that I can derive the client key (see password generation)
which is
"eq9WtJSH7PLTxsdv8SJQqJA4+6SU3f03nWE1TW4Ppqg="
And the stored key which is
"z+tqsnBaCmgSJQBLf1cPOmq7n80PW3aRjzeuAGkkuq4="
I can create the client signature by taking a HMAC using the SHA-
256 hashing method with the stored key and the message
containing information from the authorization sessions headers

92. Client Signature
92
Stored Key Client Signature
$AUTHENTICATION_MESSAGE
HMAC
“Authentication Message” is composed of some
of the one-time information shared between the
server and client

93. SCRAM Authentication Flow: Generating Proof
93
And I can generate the client proof
by calculating
client key XOR client signature

94. SCRAM Authentication Flow: Sending Proof
94
I'm sending you the client proof that I built up. It's:
"MhvH1xAaVVhUrp4d5TCyjIiMsmlNQQAFTgI57AksP7g="
Also, here is our combined nonce.

95. SCRAM Authentication Flow: Server Verification
95
Okay, I can compute the client signature
because I know the stored key and I have
the authentication session info, too.
I can get what you think the client key by
calculating
client proof XOR client signature
If the SHA-256 digest of your client key
matches the stored key, then I know you
know the password!
Plaintext
Password
Hashed
Password
Client
Key
Salt
Iterati
ons
"Client Key"
Stored
Key

96. SCRAM Authentication Flow: Server Verification
96
But…I still need to prove to you that I know
the password.
I'm going to send you my server signature
which is an HMAC using the SHA-256
method using my stored server key and the
authorization session info as the message.

97. Server Signature
97
Server Key Server Signature
$AUTHENTICATION_MESSAGE
HMAC
“Authentication Message” is composed of some
of the one-time information shared between the
server and client

98. SCRAM Authentication Flow
98
Here's my server signature:
"DOCb6oniIvWOCY0aDWKGtaUtUip3wgcQQA1PO5dp1dU="

99. Technically, grayhippo is
Authenticated At This Point.
But, can grayhippo trust the
PostgreSQL server?

100. SCRAM Authentication Flow: Client Verification
100
First let me generate the server key. This is just like
generating the SCRAM secret, and I have the salted
password available.
I can then generate an HMAC using the SHA-256
method and the server key and the authorization
session info as the message.
If this value matches your server signature, then I
know you know my password!

101. We Never Exchanged
the Secret!

102. • Identifying information sent:
• Stored Key: A hash of the HMAC using the computed salted password and "Client Key"
• Server Key: A HMAC using the computed salted password and "Server Key"
• A combined client/server nonce that can only be used once for this session
• Information about the session from the authorization headers
Recall What We Transmitted
102
Grr…

104. Looks Like redhippo is Stuck!

105. But…What About a
Rogue Server?

106. • Recall: As the final step of SCRAM, the server sends a server signature to
the client.
• The client is authenticated, but what if the server is not who you think it is?
Case #1: Server "Claims" To Know Secret
106
Here's my server
signature, you're
authenticated now!

107. Case #1: Server "Claims" To Know Secret
107
Awesome, here is all of
my secret information

108. • This is why the client must also verify the server, to ensure the server
actually knows the client's password.
Case #1: Server "Claims" To Know Secret
108
Thanks! I'm going to go
sell this for a lot of
money and buy up so
many fruits and
vegetables!

109. • Recall: We can use TLS to secure the connection endpoints between the
client and server
• But…how do we know that the server sending the signature is the same
instance we originally connected to?
Case #2: Elephant-in-the-Middle Attack
109
TLS

110. Case #2: Elephant-in-the-Middle Attack
110
TLS
OK. I choose "SCRAM-SHA-
256".
Here is a nonce I generated:
Hmm…I want to get
in on this…

111. Case #2: Elephant-in-the-Middle Attack
111
TLS
OK, here's a bunch of stuff that
looks normal because your
connection is already of a secure
socket so this must appear to be
normal…nothing weird going on
here…

112. Enter Channel Binding

113. • Introduced in PostgreSQL 11, channel binding ensures that the TLS
handshake is still the same when the client and server are identifying each
other
• Specified for SCRAM in RFC 5802, TLS channel bindings in RFC 5929
• PostgreSQL uses "tls-server-end-point", which uses a hash of the server certificate to bind
the channels
• A server that supports channel binding advertises it as "SCRAM-SHA-256-
PLUS"
• A client then requests to use channel binding (and in PostgreSQL 13 client
can require to use channel binding!)
Channel Binding
113

114. Channel Binding
114
TLS
OK. I choose "SCRAM-SHA-256-PLUS".
Here is a channel binding header with
your certificate. And the nonce
Oh no…I won't be
able to prove that I
can validate that
certificate…

115. Upgrading to SCRAM

116. • As of this presentation, all client drivers and libraries support SCRAM
except for the Swift driver
• https://wiki.postgresql.org/wiki/List_of_drivers#Drivers
• In postgresql.conf set password_encryption to scram-sha-256
• Keep md5 as your authentication method in pg_hba.conf until all your
users have re-hashed their passwords
• ...have your users re-hash their passwords. Easiest way is password
• Once all of your users have re-hashed their password, go back to
pg_hba.conf and switch your authentication method to scram-sha-256
Upgrading to SCRAM
116

117. • As of this presentation, all client drivers and libraries support SCRAM
except for the Swift driver
https://wiki.postgresql.org/wiki/List_of_drivers#Drivers
Driver Support for SCRAM
117

118. Conclusion

119. • It's good to question "it just works" as this can lead to better solutions
• Beyond PostgreSQL, SCRAM is a general purpose solution. Consider
implementing it in your other applications where you need to verify secrets
• Example PostgreSQL Password Creator:
https://gist.github.com/jkatz/e0a1f52f66fa03b732945f6eb94d9c21
• Example (poorly written) SCRAM implementation with PostgreSQL:
https://gist.github.com/jkatz/7444eda78a6fff18ab5d74c024e3761d
• …oh, upgrade your passwords to use SCRAM. Now.
• Also look into using "clientcert=verify-full" if you're running PostgreSQL 12 – two factor
authentication!
Not All Password Methods Are the Same
119

120. Thank You!
120

121. Thank You
Jonathan S. Katz
jonathan.katz@crunchydata.com
@jkatz05