2024.2: Faster with Shared Memory Between PostgreSQL and TServer Layers

In the latest YugabyteDB Long Term Support release (2024.2), the parameter pg_client_use_shared_memory is now set to true by default, as the feature has reached General Availability (GA). The release note provides the following description of the feature:

Improved query performance. Implemented shared memory for PostgreSQL to TServer communication to minimize latency and enhance query performance by reducing communication overhead. This enhancement is part of the Enhanced PostgreSQL compatibility mode.

The communication between the PostgreSQL query layer (YSQL) and the storage and transaction layer (DocDB) is designed for scalability and separation of concerns. They interact via read and write operations through the network layer to allow horizontal scalability. The new feature reduces unnecessary network calls and context switches by utilizing shared memory buffers when the calls are within the same server.

Now, let’s perform a basic PgBench test using a Docker container.

I started a container using the 2024.2 image:

-bash-4.2# docker run --rm -it yugabytedb/yugabyte:2024.2.0.0-b145 bash
[root@a69af15bf76f yugabyte]#

I started YugabyteDB with the feature disabled:

[root@a69af15bf76f yugabyte]# yugabyted start --tserver_flags="pg_client_use_shared_memory=false"
Starting yugabyted...
✅ YugabyteDB Started
✅ UI ready
✅ Data placement constraint successfully verified

+---------------------------------------------------------------------------------------------------------+
|                                                yugabyted                                                |
+---------------------------------------------------------------------------------------------------------+
| Status              : Running.                                                                          |
| Replication Factor  : 1                                                                                 |
| YugabyteDB UI       : http://172.17.0.2:15433                                                           |
| JDBC                : jdbc:postgresql://172.17.0.2:5433/yugabyte?user=yugabyte&password=yugabyte        |
| YSQL                : bin/ysqlsh -h 172.17.0.2  -U yugabyte -d yugabyte                                 |
| YCQL                : bin/ycqlsh 172.17.0.2 9042 -u cassandra                                           |
| Data Dir            : /root/var/data                                                                    |
| Log Dir             : /root/var/logs                                                                    |
| Universe UUID       : 89a893d9-fe7a-42ae-87f7-b20960e8fc28                                              |
+---------------------------------------------------------------------------------------------------------+

[root@a69af15bf76f yugabyte]# export PGHOST=$(hostname)

I initialized PgBench tables:

[root@a69af15bf76f yugabyte]# postgres/bin/ysql_bench -i
dropping old tables...
NOTICE:  table "ysql_bench_accounts" does not exist, skipping
NOTICE:  table "ysql_bench_branches" does not exist, skipping
NOTICE:  table "ysql_bench_history" does not exist, skipping
NOTICE:  table "ysql_bench_tellers" does not exist, skipping
creating tables (with primary keys)...
generating data...
100000 of 100000 tuples (100%) done (elapsed 2.22 s, remaining 0.00 s)
done.

I ran a 60-minute simple update with ten connections:

[root@a69af15bf76f yugabyte]# postgres/bin/ysql_bench -nN -c5 -P 5 -T 60
progress: 5.0 s, 854.6 tps, lat 5.738 ms stddev 2.271
progress: 10.0 s, 877.0 tps, lat 5.702 ms stddev 0.684
progress: 15.0 s, 876.8 tps, lat 5.703 ms stddev 0.510
progress: 20.0 s, 874.4 tps, lat 5.718 ms stddev 0.585
progress: 25.0 s, 876.2 tps, lat 5.707 ms stddev 0.526
progress: 30.0 s, 873.2 tps, lat 5.725 ms stddev 0.662
progress: 35.0 s, 870.4 tps, lat 5.744 ms stddev 0.750
progress: 40.0 s, 870.2 tps, lat 5.747 ms stddev 0.800
progress: 45.0 s, 870.8 tps, lat 5.741 ms stddev 0.619
progress: 50.0 s, 873.2 tps, lat 5.727 ms stddev 0.603
progress: 55.0 s, 874.0 tps, lat 5.721 ms stddev 0.522
progress: 60.0 s, 868.6 tps, lat 5.755 ms stddev 0.587
transaction type: <builtin: simple update>
scaling factor: 1
query mode: simple
number of clients: 5
number of threads: 1
batch size: 1024
duration: 60 s
number of transactions actually processed: 52302
maximum number of tries: 1
latency average = 5.727 ms
latency stddev = 0.886 ms
tps = 871.595315 (including connections establishing)
tps = 871.863476 (excluding connections establishing)

I restarted YugabyteDB with the new feature enabled:

[root@a69af15bf76f yugabyte]# yugabyted stop
Stopped yugabyted using config /root/var/conf/yugabyted.conf.
[root@a69af15bf76f yugabyte]# yugabyted start --tserver_flags="pg_client_use_shared_memory=true"
Starting yugabyted...
✅ YugabyteDB Started
✅ UI ready
✅ Data placement constraint successfully verified

+----------------------------------------------------------------------------------------------------+
|                                             yugabyted                                              |
+----------------------------------------------------------------------------------------------------+
| Status         : Bootstrapping.                                                                    |
| YugabyteDB UI  : http://172.17.0.2:15433                                                           |
| JDBC           : jdbc:postgresql://172.17.0.2:5433/yugabyte?user=yugabyte&password=yugabyte        |
| YSQL           : bin/ysqlsh -h 172.17.0.2  -U yugabyte -d yugabyte                                 |
| YCQL           : bin/ycqlsh 172.17.0.2 9042 -u cassandra                                           |
| Data Dir       : /root/var/data                                                                    |
| Log Dir        : /root/var/logs                                                                    |
| Universe UUID  : 89a893d9-fe7a-42ae-87f7-b20960e8fc28                                              |
+----------------------------------------------------------------------------------------------------+

I ran the same test:

[root@a69af15bf76f yugabyte]# postgres/bin/ysql_bench -i
dropping old tables...
creating tables (with primary keys)...
generating data...
100000 of 100000 tuples (100%) done (elapsed 2.41 s, remaining 0.00 s)
done.
[root@a69af15bf76f yugabyte]# postgres/bin/ysql_bench -nN -c5 -P 5 -T 60
progress: 5.0 s, 1013.5 tps, lat 4.841 ms stddev 1.739
progress: 10.0 s, 1036.9 tps, lat 4.821 ms stddev 0.531
progress: 15.0 s, 1032.4 tps, lat 4.843 ms stddev 0.473
progress: 20.0 s, 1029.0 tps, lat 4.859 ms stddev 0.556
progress: 25.0 s, 1024.0 tps, lat 4.883 ms stddev 0.492
progress: 30.0 s, 1014.0 tps, lat 4.930 ms stddev 1.047
progress: 35.0 s, 1024.2 tps, lat 4.882 ms stddev 0.706
progress: 40.0 s, 1027.8 tps, lat 4.865 ms stddev 0.468
progress: 45.0 s, 1026.7 tps, lat 4.870 ms stddev 0.459
progress: 50.0 s, 1024.5 tps, lat 4.880 ms stddev 0.459
progress: 55.0 s, 1027.6 tps, lat 4.866 ms stddev 0.465
progress: 60.0 s, 1023.6 tps, lat 4.884 ms stddev 0.590
transaction type: <builtin: simple update>
scaling factor: 1
query mode: simple
number of clients: 5
number of threads: 1
batch size: 1024
duration: 60 s
number of transactions actually processed: 61526
maximum number of tries: 1
latency average = 4.869 ms
latency stddev = 0.755 ms
tps = 1025.368918 (including connections establishing)
tps = 1025.681142 (excluding connections establishing)

Throughput is 1.8 times higher, and response time is 1.8 times faster.