In a distributed database like YugabyteDB, high availability and fault tolerance are core strength, but only if data replication is healthy.
One key metric to watch is under-replicated tablets. Ignoring this metric can lead to degraded read/write availability and potential data loss in failure scenarios.
What Are Under-Replicated Tablets?
YugabyteDB shards data into tablets, which are the smallest unit of data distribution and replication. Each tablet is typically replicated across multiple nodes using Raft consensus, ensuring that even if a node fails, the data remains available.
An under-replicated tablet is one where the number of healthy replicas has dropped below the configured replication factor (usually 3). This means Raft cannot safely make progress, and the tablet becomes vulnerable to additional failures.
Common Causes of Under-Replicated Tablets
Node Failures or Network Partitions: When a node goes down or becomes unreachable, any tablet replicas on that node are considered unavailable.
Slow or Overloaded Nodes: Nodes that are too slow to respond may be marked as unhealthy, reducing the effective replica count.
Insufficient Cluster Size: If the number of nodes is less than the replication factor, there simply aren’t enough targets to maintain all replicas.
Rebalancing or Maintenance Operations: During rolling upgrades or node replacements, temporary under-replication can occur.
Why Monitoring Matters
Failing to monitor under-replicated tablets can result in:
Data loss if another node fails.
Write unavailability for affected tablets (Raft requires majority consensus).
Increased recovery times, especially if the situation goes unnoticed.
YugabyteDB exposes a health check API (/api/v1/health-check) that includes a list of under-replicated tablets. Monitoring this endpoint and alerting on non-zero results is crucial for production environments.
Example:
Let’s say we have a simple three-node cluster spread across multiple regions:
yugabyte=# SELECT host, cloud, region, zone FROM yb_servers() ORDER BY cloud, region, zone;
host | cloud | region | zone
-----------+-------+-----------+------------
127.0.0.1 | aws | us-east-1 | us-east-1a
127.0.0.2 | aws | us-east-2 | us-east-2a
127.0.0.3 | aws | us-west-2 | us-west-2a
(3 rows)
This cluster is resilient to a regional outage, as it’s an RF3 setup with one copy of each tablet distributed across the three regions.
yugabyte=# CREATE TABLE some_table(id INT PRIMARY KEY);
CREATE TABLE
yugabyte=# INSERT INTO some_table SELECT generate_series(1, 100000);
INSERT 0 100000
If we check the health endpoint right now, we’ll see there are no under-replicated tablets.
yugabyte=# \! curl -s http://127.0.0.1:7000/api/v1/health-check | jq -r '.under_replicated_tablets[]'
yugabyte=#
But once we stop the YugabyteDB processes on host 127.0.0.3, that changes!
yugabyte=# -- Wait about a minute...
yugabyte=# \! curl -s http://127.0.0.1:7000/api/v1/health-check | jq -r '.under_replicated_tablets[]'
3e167157845a4d7e8e2c46d27f5c3324
77fa7e5b69094eb6a3eb50d0167735a2
0aab0e68aaf141c9aed56f78a30b099e
4cdc52fd283e40d89a9babb4c5132245
2fa977764ed94ea99881dc107dd31bb0
34e97f43162544459152998ccafe550a
def8c7ed40934f3a96b64db0cf21f9e9
c9269a134a3e4052ae79468213ce166c
b12ed2e0b2c14116ac150074cc8c4c1d
725d8f259e0443b3b77a47e5840e7f68
58f9d36b8f0c4f459e5bdebe06d299c5
31a3b66963854fe588851563050caa43
Now that we’re seeing under-replicated tablets, let’s find out which tables they belong to. We’ll start by loading the tablet IDs into a table, which will make it easier to run exploratory queries and investigate further.
yugabyte=# CREATE TEMP TABLE under_replicated_tablets(tablet_id TEXT) ON COMMIT PRESERVE ROWS;
CREATE TABLE
yugabyte=# \COPY under_replicated_tablets FROM PROGRAM 'curl -s http://127.0.0.1:7000/api/v1/health-check | jq -r ''.under_replicated_tablets[]''' WITH (ROWS_PER_TRANSACTION 0);
COPY 12
With the under_replicated_tablets loaded, we can join it with the yb_local_tablets system table to retrieve the table names associated with the under-replicated tablets:
yugabyte=# SELECT DISTINCT table_type, namespace_name, ysql_schema_name, table_name
yugabyte-# FROM yb_local_tablets a
yugabyte-# JOIN under_replicated_tablets b USING (tablet_id)
yugabyte-# ORDER BY table_type, namespace_name, ysql_schema_name, table_name;
table_type | namespace_name | ysql_schema_name | table_name
------------+----------------+------------------+--------------
System | system | | transactions
YCQL | system | | metrics
YSQL | yugabyte | public | some_table
(3 rows)
From the output above, we can see all the affected tables, including the one created in today’s tip, that are at risk. Immediate action should be taken to restore the unavailable node and recover full replication.
Best Practices for Monitoring Under-Replicated Tablets
Use Prometheus/Grafana to track under-replicated tablet metrics.
Set up alerts for immediate operator attention. (YBA has a built in alert for this: Under-replicated tablets)
Automate node recovery where possible.
Regularly test fault tolerance to ensure resilience.
Have Fun!
