DocDB is the underlying document storage engine of YugabyteDB and is built on top of a highly customized and optimized version of RocksDB, a log-structured merge tree (LSM)-based key-value store.
From time to time, it may be necessary to review storage layer activity in the YugabyteDB TServer log.
Enabling low-level tracing temporarily can provide valuable insights for debugging this layer, but it’s important not to leave it enabled continuously, as it will result in a significant increase in logging activity.
To enable DocDB storage layer actvity logging, use the following yb-ts-cli commands on each node of youe cluster (node restarts are not needed):
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force enable_tracing 1
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force collect_end_to_end_traces 1
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force rpc_slow_query_threshold_ms 100
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force tracing_level 1
Now DDL command that effect the stroage layer will generate a slew of new infornative messages in the TServer log…
# ysqlsh -h $(hostname -I | awk '{print $1}') -c "SELECT now();" -c "DROP TABLE t;"
now
------------------------------
2024-04-22 12:24:25.30127+00
(1 row)
DROP TABLE
Example records from the TServer log resulting from the DROP command above:
I0422 12:24:25.409153 27650 ts_tablet_manager.cc:3142] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I0422 12:24:25.409166 27650 docdb_rocksdb_util.cc:326] FLAGS_rocksdb_max_background_flushes was not set, automatically configuring 1 max background flushes
I0422 12:24:25.409173 27650 docdb_rocksdb_util.cc:361] FLAGS_rocksdb_max_background_compactions was not set, automatically configuring 2 background compactions.
I0422 12:24:25.409178 27650 docdb_rocksdb_util.cc:374] FLAGS_rocksdb_base_background_compactions was not set, automatically configuring 2 base background compactions.
I0422 12:24:25.409184 27650 docdb_rocksdb_util.cc:616] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Write buffer size: 134217728
I0422 12:24:25.409196 27650 tablet_metadata.cc:774] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Destroying regular db at: /root/yb01/data/yb-data/tserver/data/rocksdb/table-000033bd000030008000000000004000/tablet-98f8ad90fcf4427cba5c0799af3cfdee
I0422 12:24:25.409746 27650 tablet_metadata.cc:780] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Successfully destroyed regular DB at: /root/yb01/data/yb-data/tserver/data/rocksdb/table-000033bd000030008000000000004000/tablet-98f8ad90fcf4427cba5c0799af3cfdee
I0422 12:24:25.409966 27650 tablet_metadata.cc:797] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Successfully destroyed provisional records DB at: /root/yb01/data/yb-data/tserver/data/rocksdb/table-000033bd000030008000000000004000/tablet-98f8ad90fcf4427cba5c0799af3cfdee.intents
I0422 12:24:25.417534 27650 ts_tablet_manager.cc:3152] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Tablet deleted. Last logged OpId: 1.1
I0422 12:24:25.417563 27650 log.cc:1615] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Deleting WAL dir /root/yb01/data/yb-data/tserver/wals/table-000033bd000030008000000000004000/tablet-98f8ad90fcf4427cba5c0799af3cfdee
I0422 12:24:25.418270 27650 consensus_meta.cc:124] T 98f8ad90fcf4427cba5c0799af3cfdee Deleting consensus metadata
I0422 12:24:25.418406 27650 tablet_bootstrap_if.cc:96] T 98f8ad90fcf4427cba5c0799af3cfdee P d71460c486504207a09b348d26cbf2c0: Deleted tablet blocks from disk
I0422 12:24:25.418440 27650 ts_tablet_manager.cc:2769] Unregister data/wal directory assignment map for table: 000033bd000030008000000000004000 and tablet 98f8ad90fcf4427cba5c0799af3cfdee
I0422 12:24:25.418454 27650 ts_tablet_manager.cc:3226] Deleted transition in progress deleting tablet for tablet 98f8ad90fcf4427cba5c0799af3cfdee
I0422 12:24:25.453619 27653 client-internal.cc:665] Deleted table 000033bd000030008000000000004000
I0422 12:24:25.969446 27631 tablet_server.cc:1152] Invalidating the entire PgTableCache cache since catalog version incremented
Don’t forget to turn off the DocDB storage layer logging!
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force enable_tracing 0
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force collect_end_to_end_traces 0
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force rpc_slow_query_threshold_ms 10000
yb-ts-cli --server_address=$(hostname -I | awk '{print $1}') set_flag -force tracing_level 0
Have Fun!