Online index creation in YugabyteDB is powerful… you can add columns, create indexes, and keep the application running without blocking reads or writes.
But when a table is large (hundreds of GB or hundreds of millions of rows), index backfill can take a long time, and default settings are intentionally conservative to protect live workloads.
This tip summarizes why backfill can be slow, what makes it retry work, and, most importantly, how to safely speed it up.
🔍 What Is Index Backfill?
When you run CREATE INDEX on a table that already contains data:
YugabyteDB scans the existing rows in the base table.
It writes corresponding index entries into the new or rebuilt index.
Meanwhile, new writes to the table are also copied into the index to maintain consistency.
This is why index creation is online… queries continue to run normally.
🐢 Why Can Index Backfill Take a Long Time?
1️⃣ Large amount of data
Backfill must read and write every existing row. On a 143 GB table, this may be hundreds of millions of rows. Every row must be scanned, transformed, and written into the index’s distributed storage.
2️⃣ Conservative defaults
For safety, YugabyteDB limits backfill aggressiveness:
| Setting | Default | Meaning |
|---|---|---|
backfill_index_write_batch_size | 128 | Rows per batch |
num_concurrent_backfills_allowed | auto-tuned (~8 per node on >16 cores) | Parallel backfill workers |
| Several backfill RPC timeouts | Relatively small | Protect cluster from long-running/stalled operations |
Safe defaults prevent overload but slow backfill dramatically on large datasets.
3️⃣ Timeouts that trigger retries
A batch might run slowly due to:
● leader changes
● network hiccups
● saturated CPU/IO
● uneven tablet distribution
If the batch does not finish before its RPC deadline, the master retries the batch… often restarting work from the beginning.
Large batches make retries more expensive, and repeated retries can cause hours of unnecessary delay.
4️⃣ Distributed nature of index tables
Since index tablets may live on different nodes, the system will:
● read from one set of nodes,
● write to others,
● coordinate RPCs across the cluster.
This can become a bottleneck on:
● clusters under heavy load
● large-scale deployments
● environments with noisy neighbors or variable network latency
5️⃣ Transaction “safe time” delays
Before backfill starts, YugabyteDB ensures no long-running transactions can see a partial index. If transactions are stuck or slow to commit, backfill may stall waiting for a “safe time.”
Controlled by:index_backfill_wait_for_old_txns_ms
🚀 How to Make Index Backfill Much Faster
(Updated: 09/03/2026)
Here are the most effective tuning options, with practical guidance from real-world scenarios.
1️⃣ Carefully Increase backfill_index_write_batch_size
Default: 128
Common tuning range: 2000–4000
Increasing backfill_index_write_batch_size can significantly improve index-backfill throughput when the cluster has sufficient headroom. Clusters have seen 2× improvement in index build time by increasing this value.
If a large base table is backfilling into an index with only a few tablets, many source tablets can funnel writes into just one or a few index tablet leaders. A larger batch size can amplify this fan-in, leading to frequent flushes, L0 SST-file buildup, compaction pressure, and potentially temporary write rejections. For large hash-sharded indexes, consider pre-splitting the index with
SPLIT INTO so the backfill work is distributed from the beginning. See Don’t Forget to Pre-Split Large Indexes for more details.
Larger batches must also be paired with larger RPC timeouts (next section). Otherwise, batches may time out and retry, negating the performance improvement.
2️⃣ Tune backfill RPC and grace-period timeouts when needed
When increasing backfill_index_write_batch_size (for example, to 1024 or 2048) each backfill batch can take longer to complete. If you begin seeing server-side timeout errors, review the related timeout settings and tune them as needed rather than increasing them blindly.
Keep in mind that larger RPC timeouts only help prevent otherwise healthy backfill batches from timing out prematurely. They do not address write fan-in, compaction pressure, or an under-sharded index. If the new index has too few tablets, address the tablet distribution and/or backfill rate first.
Recommended values:
# YB-Master
ysql_index_backfill_rpc_timeout_ms = 300000
# 5 minutes
# YB-TServer
backfill_index_timeout_grace_margin_ms = -1
# Automatic; YSQL baseline is at least 180000 ms (3 minutes)
backfill_index_client_rpc_timeout_ms = 86400000
# 1 day; increase only if the overall backfill legitimately needs longer
Why this matters:
- ● Gives larger backfill batches enough time to complete
- ● Avoids premature server-side backfill failures
- ● Keeps the grace margin appropriately sized for each batch
- ● Avoids accidentally reducing the much longer client-side timeout
The default
backfill_index_client_rpc_timeout_ms is
86400000 milliseconds — 1 day. Reducing it to values such as
30000 or 60000 would shorten the timeout to only 30–60 seconds and could cause
long-running index backfills to fail prematurely.Likewise, consider leaving
backfill_index_timeout_grace_margin_ms
at its default of -1 unless you have a specific reason to override it. For YSQL,
YugabyteDB automatically applies a baseline grace margin of at least 3 minutes.
3️⃣ Understand and leverage parallelism
YugabyteDB automatically configures num_concurrent_backfills_allowed as:
num_concurrent_backfills_allowed = min(8, num_cores / 2)
Meaning:
● A node with 16+ cores → up to 8 parallel backfill workers
● A 3-node cluster can achieve up to 24 parallel tasks
Each task spawns a dedicated PG backend.
If the cluster is not resource-constrained, the default parallelism is generally sufficient.
4️⃣ Control the Backfill Rate When Needed
If index backfill is putting too much pressure on the cluster, don’t rely on backfill_index_write_batch_size as the primary throttle. That setting controls how much work is sent in each batch, but it is not the best control for deliberately slowing the overall backfill workload.
Instead, use backfill_index_rate_rows_per_sec to control the rate at which rows are backfilled, and consider reducing num_concurrent_backfills_allowed if multiple index backfills are competing for resources at the same time. These are the more appropriate controls when the goal is to deliberately reduce backfill pressure.
For example, if you observe high disk utilization, sustained compaction pressure, increasing SST-file counts, or write rejections while an index is being built, consider lowering the backfill rate rather than simply reducing the batch size.
backfill_index_rate_rows_per_sec = <lower rate as needed>
num_concurrent_backfills_allowed = <reduce concurrency as needed>
The right values depend on the workload and available cluster headroom, so tune gradually while monitoring the system.
Rate limiting helps control pressure. It does not fix tablet fan-in.
Slowing the backfill can give compaction more time to keep up, but if many base-table tablets are feeding writes into only one or a few index tablets, the underlying bottleneck still exists. For large hash-sharded indexes, check the index tablet count and consider pre-splitting with SPLIT INTO before starting the backfill.A good tuning sequence is therefore: make sure the index has an appropriate tablet distribution first, then tune batch size for throughput, and use backfill rate or concurrency controls when you need to reduce pressure on the cluster.
5️⃣ Avoid noisy or unstable cluster conditions
Backfill is sensitive to:
● leadership flapping
● high CPU (>85%)
● memory pressure
● slow storage
● network jitter
Best practice:
● Run backfills during low-traffic windows
● Stabilize leadership first
● Ensure nodes aren’t heavily loaded
6️⃣ Troubleshooting common backfill stalls
If you see:
ERROR: timed out waiting for postgres backends to catch up
DETAIL: 1 backends on database 13515 are still behind catalog version 2.
You can find lagging PG backends with:
SELECT *
FROM pg_stat_activity
WHERE backend_type != 'walsender'
AND catalog_version < <target_version>
AND datid = <db_oid>;
datid, and 2 is the target_version. Slow catalog-version advancement usually means:
● Overloaded PG backend
● Timeout too small for the configured batch size
If this happens, reduce the batch size (e.g., from 4000 → 2000) or increase timeouts.
🧪 Recommended Starting Point for Large Table Backfills
For clusters with adequate CPU and stable workload:
backfill_index_write_batch_size = 2000 # or 4000 if cluster is very stable
ysql_index_backfill_rpc_timeout_ms = 300000
backfill_index_timeout_grace_margin_ms = 60000
backfill_index_client_rpc_timeout_ms = 300000
If you see retries or catalog lag:
● Reduce batch size (2000 → 1000)
● Increase timeouts further
● Reduce system load
If you are working with a very large table that has only a small number of tablets, or need to build indexes on partitioned tables, see:
That tip covers additional techniques including fetch-size tuning, write concurrency, partition-by-partition index creation, and results from more aggressive backfill testing.
📌 Final Guidance
If you’re adding columns and recreating or backfilling indexes on a large table:
● Expect the operation to be CPU/IO intensive
● Plan a tuning pass before running it on production-sized data
● Start with higher batch sizes + tuned timeouts
● Monitor for backfill retries or catalog lag
● Validate that the cluster has enough headroom for parallel backfills
With correct tuning, even huge backfills can complete several times faster while still running online.
Have Fun!
