Looking ahead at PostgreSQL 15

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Looking ahead at PostgreSQL 15 Jonathan Katz, Principal Product Manager Technical Jim Mlodgenski, Principal Database Engineer AWS RDS Open Source

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon Relational Database Service (Amazon RDS) Fully managed relational database service Spend time innovating and building new apps, not managing infrastructure • Schema design • Query construction • Query optimization Automatic failover Backup and recovery Isolation and security Industry compliance Push-button scaling Automated patching and upgrades Advanced monitoring Routine maintenance You AWS

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon RDS Set up, operate, and scale a relational database in the cloud with just a few clicks Available and durable Automatic Multi-AZ data replication, with automated backup, snapshots, and failover Easy to administer Easily deploy and maintain hardware, OS, and DB software, with built-in monitoring Performant and scalable Scale compute and storage with a few clicks, plus minimal downtime for your application Secure and compliant Data encryption at rest and in transit, with industry compliance and assurance programs PostgreSQL-Compatible Edition MySQL-Compatible Edition

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • A PostgreSQL “major release” is an annual feature release • PostgreSQL 15 release cycle: • Release cycle begins: July 2021 • Feature freeze: March/April 2022 • Beta: May 2022 • General availability: Late Q3 / Early Q4 2022 • During beta: • Very unlikely that new functionality is added • Some functionality may be removed 4 PostgreSQL major release process

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • PostgreSQL releases average ~175 new features • Review beta release notes: https://www.postgresql.org/docs/15/release-15.html • As we go through the new PostgreSQL features in this talk, we will: • Look at use cases for each feature • Review previous work • Describe the new functionality • See examples How to explore PostgreSQL 15

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. What is “conditional SQL?” • “Conditional SQL” is when a statement takes an additional action based on the outcome of a previous statement • Prior to PostgreSQL 9.5, two ways of doing this both prone to race conditions • Application layer • Procedural Language (e.g. PL/pgSQL) • Additional syntax + overhead • Ease of use with ORMs “Add a new credit card to this account. However, if the credit card number already exists, update the expiration date.”

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 9.5: INSERT … ON CONFLICT CREATE TABLE active_accounts ( account_id int PRIMARY KEY, last_active_at timestamptz NOT NULL ); INSERT INTO active_accounts VALUES (1, CURRENT_TIMESTAMP) ON CONFLICT (account_id) DO UPDATE SET last_active_at = CURRENT_TIMESTAMP; TABLE active_accounts; account_id | last_active_at ------------+------------------------------- 1 | 2022-04-29 12:12:25.626644-04

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 9.5: INSERT … ON CONFLICT INSERT INTO active_accounts VALUES (1, CURRENT_TIMESTAMP) ON CONFLICT (account_id) DO UPDATE SET last_active_at = CURRENT_TIMESTAMP; TABLE active_accounts; account_id | last_active_at ------------+------------------------------- 1 | 2022-05-02 11:45:06.317415-04 (1 row)

10.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 15: MERGE TRUNCATE TABLE active_accounts; -- clear out the data for this demo -- store each account activity in a log table CREATE TABLE activity_log ( account_id int NOT NULL, active_at timestamptz NOT NULL ); INSERT INTO activity_log VALUES (1, CURRENT_TIMESTAMP - '15 days'::interval), (1, CURRENT_TIMESTAMP - '10 days'::interval), (1, CURRENT_TIMESTAMP - '8 days'::interval);

11.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 15: MERGE MERGE INTO active_accounts USING ( SELECT account_id, max(active_at) AS last_active_at FROM activity_log GROUP BY account_id ) alog ON active_accounts.account_id = alog.account_id WHEN NOT MATCHED AND alog.last_active_at >= CURRENT_TIMESTAMP - '14 days'::interval THEN INSERT VALUES (alog.account_id, alog.last_active_at) WHEN MATCHED AND alog.last_active_at < CURRENT_TIMESTAMP - '14 days'::interval THEN DELETE WHEN MATCHED AND active_accounts.last_active_at < alog.last_active_at THEN UPDATE SET last_active_at = alog.last_active_at;

12.

13.

14.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 15: MERGE WHEN NOT MATCHED AND alog.last_active_at >= CURRENT_TIMESTAMP - '14 days'::interval THEN INSERT VALUES (alog.account_id, alog.last_active_at) WHEN MATCHED AND alog.last_active_at < CURRENT_TIMESTAMP - '14 days'::interval THEN DELETE WHEN MATCHED AND active_accounts.last_active_at < alog.last_active_at THEN UPDATE SET last_active_at = alog.last_active_at; Conditions and actions. If no row is found and the most recent activity was within 14 days: INSERT If a row is found but most recent activity older than 14 days: DELETE If a row is found and most recent activity is newer: UPDATE

15.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 15: MERGE -- execute MERGE command TABLE active_accounts; account_id | last_active_at ------------+------------------------------- 1 | 2022-04-24 11:20:59.445867-04 INSERT INTO activity_log VALUES (1, CURRENT_TIMESTAMP); -- execute MERGE command TABLE active_accounts; account_id | last_active_at ------------+------------------------------- 1 | 2022-05-02 11:21:36.55613-04

16.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. PostgreSQL 15: MERGE DELETE FROM activity_log WHERE active_at >= CURRENT_TIMESTAMP - '14 days'::interval; -- execute MERGE command TABLE active_accounts; account_id | last_active_at ------------+---------------- (0 rows)

17.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 17 INSERT … ON CONFLICT vs Merge INSERT INTO active_accounts SELECT g, CURRENT_TIMESTAMP FROM generate_series(1, 10000000) g ON CONFLICT (account_id) DO UPDATE SET last_active_at = CURRENT_TIMESTAMP; INSERT 0 10000000 Time: 36295.022 ms (00:36.295) INSERT 0 10000000 Time: 66868.616 ms (01:06.869) MERGE INTO active_accounts USING (SELECT g, CURRENT_TIMESTAMP as t FROM generate_series(1, 10000000) g) AS a ON active_accounts.account_id = a.g WHEN NOT MATCHED THEN INSERT VALUES (a.g, a.t) WHEN MATCHED THEN UPDATE SET last_active_at = a.t; MERGE 10000000 Time: 17938.168 ms (00:17.938) MERGE 10000000 Time: 495383.872 ms (08:15.384) SET work_mem = '256MB’; MERGE 10000000 Time: 54970.733 ms (00:54.971)

18.

19.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • 9.2 (2012): Support for JSON stored as text type • 9.3: Introspection / extraction functionality • 9.4: Support for JSON as binary type (JSONB) and indexing • 9.5: JSONB building functions (2017: SQL/JSON standard published) • 10: Fulltext search for JSON(B) documents • 11: JSON(B) transforms from PL/Python / PL/Perl • 12: SQL/JSON path language • 13: jsonpath.datetime • 14 (2021): JSON subscripting syntax (e.g. a[‘b’][‘c’]) A brief history of PostgreSQL and JSON

20.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Published as SQL standard extension with SQL:2016 • SQL standard for interfacing with JSON • Provide better interoperability between database engines • Previous PostgreSQL releases have supported functionality similar to SQL/JSON 20 SQL/JSON

21.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Constructors • JSON(), JSON_SCALAR(), JSON_ARRAY(), JSON_ARRAYAGG(), JSON_OBJECT(), JSON_OBJECTAGG() • Return "json" type by default; for "jsonb" use "RETURNING jsonb" • Query functions • JSON_EXISTS, JSON_VALUE, JSON_QUERY • Helpful for introspection and writing constraints • JSON table • Converts JSON into a PostgreSQL table • IS JSON • "SELECT value IS JSON;" 21 SQL/JSON and PostgreSQL 15

22.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 22 SQL/JSON constructors =$ SELECT pg_typeof(JSON('{"a": 1}')); -- returns "json" =$ SELECT pg_typeof(JSON('{"a": 1}' RETURNING jsonb)); -- returns "jsonb" =$ SELECT JSON(1); ERROR: cannot cast type integer to json =$ SELECT JSON(JSON_SCALAR(1)); json ------ 1

23.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 23 SQL/JSON constructors =$ SELECT JSON_ARRAYAGG(x ORDER BY x DESC RETURNING jsonb) FROM generate_series(1,5) x; json_arrayagg ----------------- [5, 4, 3, 2, 1] =$ SELECT JSON_OBJECT(x: x+1, x*2: x^2 RETURNING jsonb) FROM generate_series(1,5); json_object -------------------- {"1": 2, "2": 1} {"2": 3, "4": 4} {"3": 4, "6": 9} {"4": 5, "8": 16} {"5": 6, "10": 25}

24.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 24 SQL/JSON query functions =$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }'), '$.a.aa'); ERROR: JSON_EXISTS() is not yet implemented for json type =$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }' RETURNING jsonb), '$.a.aa'); json_exists ------------- t =$ SELECT JSON_EXISTS(JSON('{"a": { "aa": 1 } }' RETURNING jsonb), '$.b'); json_exists ------------- f

25.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 25 SQL/JSON query functions =$ SELECT JSON_QUERY( JSON_ARRAY( JSON_OBJECT('id': 1), JSON_OBJECT('id': 2), JSON_OBJECT('id': 3) RETURNING jsonb ), '$[$i].id' PASSING x AS i ) FROM generate_series(0, 2) x; json_query ------------ 1 2 3

26.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 26 SQL/JSON table CREATE TABLE users AS SELECT * FROM JSON_TABLE( JSON_ARRAY( JSON_OBJECT('id': 1, 'name': 'abc', 'created_on': '2022-04-30'), JSON_OBJECT('id': 2, 'name': 'def', 'created_on': '2022-05-01'), JSON_OBJECT('id': 3, 'name': 'ghi', 'created_on': '2022-05-02') RETURNING jsonb ), '$[*]' COLUMNS ( id int, name text, created_on date ) );

27.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 27 SQL/JSON table TABLE users; id | name | created_on ----+------+------------ 1 | abc | 2022-04-30 2 | def | 2022-05-01 3 | ghi | 2022-05-02

28.

29.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • "recovery_prefetch" parameter • During recovery, enables prefetching of blocks that are referenced in the WAL that are not yet in the buffer pool • Works for crash recovery, replication, and PITR • Extensibility • Custom WAL resource managers • Allows table access method extensions to participate in logical decoding • Remove exclusive backup mode Note: Some custom backup scripts may need to change 30 PostgreSQL 15 Backup and recovery improvements

30.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 31 pg_walinspect =# SELECT pg_current_wal_lsn(); pg_current_wal_lsn -------------------- 16/F2392B60 (1 row) =# INSERT INTO active_accounts VALUES (-1, now()); INSERT 0 1 =# SELECT start_lsn, xid, resource_manager, record_type, record_length FROM pg_get_wal_records_info_till_end_of_wal('16/F2392B60'); start_lsn | xid | resource_manager | record_type | record_length -------------+-----+------------------+-------------------+--------------- 16/F2392B60 | 789 | Heap | INSERT | 71 16/F2392BA8 | 789 | Btree | INSERT_LEAF | 64 … (7 rows) • Provides SQL functions that allow view access to the contents of write-ahead log

31.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Allows for WAL archiving to be handled by a custom library instead of a shell command using “archive_command” • Will often be considerably more robust and performant • Example module provided by contrib/basic_archive # postgresql.conf archive_mode = 'on' archive_library = 'basic_archive' basic_archive.archive_directory = '/path/to/archive/directory' 32 Archive modules

32.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • WAL compression (zstd + lz4) • Server-side compression for pg_basebackup • gzip, zstd, lz4 • Add zstd + lz4 for client-side • pg_receivewal (lz4 only) 33 Compression

33.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 1 2 4 8 16 32 64 128 256 512 transaction per second connections WAL compression off pglz lz4 zstd • PostgreSQL has the ability to compress full page writes • lz4 and zstd added to the existing pglz algorithm 34 WAL compression - performance 31% 41%

34.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 0 20 40 60 80 100 120 140 160 1 2 4 8 16 32 64 128 256 512 size (GB) connections WAL compression off pglz lz4 zstd • PostgreSQL has the ability to compress full page writes • lz4 and zstd added to the existing pglz algorithm 35 WAL compression – size-on-disk 71% 79%

35.

36.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Logical replication allows for changes to be streamed to another database independent of the physical filesystem • Several use cases: • Change-data capture (CDC) • Streaming analytics / BI • Major version upgrades • Multi-writeable databases • Logical replication currently limited to data changes 37 Logical replication overview

37.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • 9.4: Logical decoding, output plugins, replication slots • 10: Logical replication of INSERT/UPDATE/DELETE • 13: Logical replication of partitions • 14: Performance improvements; stream in-flight transactions A brief history of PostgreSQL and logical replication

38.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • ALTER SUBSCRPTION … SKIP • Skip a transaction (e.g. on a conflict) and then resume replay • Can specify log sequence number (LSN) • Requires superuser privileges • Temporarily disable streaming replication • ALTER SUBSCRIPTION … SET disable_on_error = true; • Fix conflict on publisher side • ALTER SUBSCRIPTION ... ENABLE; 39 PostgreSQL 15 logical replication improvements

39.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Support for two-phase commit / prepared transactions • Sends transaction to subscriber when PREPARED TRANSACTION called • Publish all tables in schema • Previous was all tables in a database • Publish a subset of rows/columns in a table 40 PostgreSQL 15 logical replication improvements

40.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 41 PostgreSQL 15 Logical replication examples -- publish all tables in schema "app" =# CREATE SCHEMA app; =# CREATE TABLE app.users ( id int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, username text); =# CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA app; -- publish only a subset of columns on a table =# CREATE TABLE abc(x int, y int, z int); =# ALTER TABLE abc REPLICA IDENTITY FULL; =# CREATE PUBLICATION pub2 FOR TABLE abc (x); -- publish only non-test data =# CREATE TABLE logs (id uuid PRIMARY KEY, log jsonb, is_test bool); =# CREATE PUBLICATION pub3 FOR TABLE logs WHERE (NOT is_test);

41.

42.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Improved in-memory sort performance • Parallel SELECT DISTINCT • postgres_fdw parallel commit • Commit transactions executed on remote PostgreSQL asynchronously • Optimization for high CPU core arm64 processors (e.g. Graviton) • Tests show 10-20% speedup on 48+ cores 43 PostgreSQL 15 performance highlights

43.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • CREATE revoked from PUBLIC in public schema in new databases • By default only database owners can create objects in default schema • Mitigation for CVE-2018-1058 Note: This may break some scripts run as a non-superuser • SECURITY INVOKER views • Uses permissions of user executing view, not owner 44 PostgreSQL 15 security highlights

44.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • jsonlog log format • Structured logging and compatibility with log analysis tools • More regular expression functions • regexp_count: counts instances of pattern matches • regexp_instr: returns position that matches pattern • regexp_like: returns true if pattern matched; similar to ~ operator • dconfig • Displays non-default PostgreSQL configurations • Search for configuration names/value 45 PostgreSQL 15 general highlights

45.

46.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Significant work on simplifying developer experience • More space-saving options for backups and archives • More flexibility with logical replication • General improvements that help with daily tasks 47 PostgreSQL 15 early takes

47.

© 2022, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • PostgreSQL commit logs • Blogs • Hubert "depesz" Lubaczewski – "Waiting for PostgreSQL 15" series • Presentations • Magnus Hagander – "What's new in PostgreSQL 15" • pgPedia • https://pgpedia.info/postgresql-versions/postgresql-15.html 48 References

48.