In YugabyteDB, a PostgreSQL backend process refers to the individual processes that manage SQL client connections in a PostgreSQL-compatible way.
Built on a distributed architecture, YugabyteDB supports PostgreSQL client drivers and SQL queries, providing seamless compatibility.
While these backend processes function similarly to those in PostgreSQL, they are integrated into YugabyteDB’s distributed system. They play a role in managing data across the multi-node, distributed environment of YugabyteDB.
Each connection to a YugabyteDB cluster consumes CPU and memory, making it crucial to consider the number of connections required by your application.
To manage resource usage and prevent excessive consumption, YugabyteDB uses the max_connections setting to limit the number of connections per node in the cluster. This helps avoid overwhelming the deployment’s resources due to runaway connection behavior.
To check the total memory usage of all PostgreSQL backend processes on a node, you can use the following script:
for pid in $(ps -eo pid,comm | grep '[p]ostgres' | awk '{print $1}'); do pss=$(awk '/Pss:/ {total += $2} END {print total}' /proc/$pid/smaps 2>/dev/null); total_pss=$((total_pss + pss)); done
Note: pss (Proportional Set Size), is the amount of memory shared with other processes, accounted in a way that the amount is divided evenly between the processes that share it. I.e., if a process has 10 MB all to itself and 10 MB shared with another process its PSS will be 15 MB.
Example:
[root@yugabytedb ~]# ps -eo pid,comm | grep '[p]ostgres'
3646053 postgres
3646092 postgres
3646094 postgres
3646097 postgres
3646105 postgres
[root@yugabytedb ~]# for pid in $(ps -eo pid,comm | grep '[p]ostgres' | awk '{print $1}'); do pss=$(awk '/Pss:/ {total += $2} END {print total}' /proc/$pid/smaps 2>/dev/null); total_pss=$((total_pss + pss)); done
[root@yugabytedb ~]# echo $total_pss
527371
[root@yugabytedb ~]# echo $((total_pss / 1024)) -- In MiB
515
Let’s create alot more connections!
[root@yugabytedb ~]# cat start_300.sh
#!/bin/bash
for i in {1..300}; do
ysqlsh -h 127.0.0.1 -U yugabyte -c "SELECT 1;" > /dev/null 2>&1 &
done
wait
[root@yugabytedb ~]# ./start_300.sh &
[602] 3680817
[root@yugabytedb ~]# ps -eo pid,comm | grep '[p]ostgres' | wc -l
305
[root@yugabytedb ~]# for pid in $(ps -eo pid,comm | grep '[p]ostgres' | awk '{print $1}'); do pss=$(awk '/Pss:/ {total += $2} END {print total}' /proc/$pid/smaps 2>/dev/null); total_pss=$((total_pss + pss)); done
[root@yugabytedb ~]# echo $total_pss
2261143
[root@yugabytedb ~]# echo $((total_pss / 1024)) -- In MiB
2208 -- In MiB
[root@yugabytedb ~]# echo $((total_pss / 1048576)) -- In GiB
2 -- In GiB
Note that all the connections in the above test were idle. In other words, they are open connections that aren’t executing any queries or operations, but they remain active and maintain a session with the database.
An active workload will definitely gobble up way more memory than our simple baseline experiment.
To avoid overprovisioning a node with connections in YugabyteDB, it’s important to manage and limit the number of connections effectively. Too many connections can lead to resource exhaustion, including CPU, memory, and network issues.
Here are some strategies to help prevent over-provisioning:
- Tune the
max_connectionsParameter - Connection Pooling
- Connection Timeouts
- Connection Limits per User
- Use the new YugabyteDB Built in Connection Manager! [ RECOMMENDED!!! ]
Have Fun!
