When preparing for a YugabyteDB upgrade or investigating a behavior change, a common question comes up:
- Did the default value of this GFlag change between my current version and the version I’m upgrading to?
You could search the documentation or GitHub, or even stand up two clusters and compare their /varz output. But if what you really want to know is what default value was compiled into a specific yb-master or yb-tserver binary, you can ask the binaries directly.
YugabyteDB binaries support the gflags option:
--helpxml
This emits the flags compiled into the binary as structured XML, including their hard-coded default values. Combine that with official YugabyteDB Docker images and you can compare two releases without ever starting a YugabyteDB cluster.
This technique compares the compiled default of a GFlag in specific
yb-master and/or yb-tserver binaries. It does not tell you what a running YugabyteDB universe is currently using after configuration overrides, AutoFlag promotion, or other runtime changes.
Why Not Just Check the Documentation?
The documentation is still a great place to start, but binary-to-binary comparison becomes especially useful when you need an answer for an exact release build.
Documentation is generally organized around release families rather than every patch and build, so a patch-level change may not be obvious. Some advanced or internal flags may also be difficult to find, and source-code searches introduce another question: Which exact commit or backport made it into the released build?
The released binary removes that ambiguity. If you want to know what was compiled into 2024.1.3.0-bXX, inspect that exact binary.
This approach also avoids the “two cluster” problem. You do not need to provision two environments, start Masters and TServers, query them, and tear everything down. Docker provides the binaries, --helpxml prints their flag metadata, and the process exits.
The --helpxml Trick
For example, inside a YugabyteDB installation you can run:
bin/yb-tserver --helpxml
The returned XML contains entries such as:
webserver_port
9000
That means we can programmatically ask each release binary:
- What is your compiled default for this exact GFlag?
We can do the same against yb-master, which is important because a flag may exist in one process, both processes, or have different defaults between them.
The Script: compare_docker_flags.sh
The following script automates the entire process. It accepts two YugabyteDB versions, resolves base versions to Docker build tags, pulls the required images, runs yb-master and/or yb-tserver with --helpxml, extracts the requested flag, compares the values, and optionally removes images that it downloaded.
#!/bin/bash
# Usage:
# ./compare_docker_flags.sh [--cleanup] \
# [master|tserver|both]
#
# Example:
# ./compare_docker_flags.sh --cleanup \
# webserver_port 2.20.10.0 2024.1.3.0 master
CLEANUP=0
while [[ "$#" -gt 0 ]]; do
case $1 in
--cleanup)
CLEANUP=1
shift
;;
-*)
echo -e "\e[31m[!] Unknown option: $1\e[0m"
exit 1
;;
*)
break
;;
esac
done
FLAG=$1
V1=$2
V2=$3
COMP=${4:-both}
if [ "$#" -lt 3 ] || [ "$#" -gt 4 ]; then
echo -e "\e[31m[!] Error: Invalid number of arguments.\e[0m"
echo "Usage: $0 [--cleanup] [master|tserver|both]"
exit 1
fi
if [[ ! "$FLAG" =~ ^[a-zA-Z_][a-zA-Z0-9_-]*$ ]]; then
echo -e "\e[31m[!] Error: '$FLAG' does not look like a valid flag name.\e[0m"
exit 1
fi
for v in "$V1" "$V2"; do
if [[ ! "$v" =~ ^[0-9] ]]; then
echo -e "\e[31m[!] Error: Invalid version string '$v'. Versions must start with a number.\e[0m"
exit 1
fi
done
if [[ "$COMP" != "master" &&
"$COMP" != "tserver" &&
"$COMP" != "both" ]]; then
echo -e "\e[31m[!] Error: Invalid component parameter '$COMP'. Use master, tserver, or both.\e[0m"
exit 1
fi
resolve_tag() {
local base_version=$1
if [[ "$base_version" == *-b* ]]; then
echo "$base_version"
return
fi
local resolved_tag
resolved_tag=$(
curl -s \
"https://hub.docker.com/v2/repositories/yugabytedb/yugabyte/tags?page_size=100&name=${base_version}" |
grep -o '"name":"[^"]*"' |
cut -d'"' -f4 |
grep -E "^${base_version}(\.[0-9]+)*-b[0-9]+$" |
sort -V |
tail -n 1
)
if [ -n "$resolved_tag" ]; then
echo "$resolved_tag"
else
echo "$base_version"
fi
}
get_flag_from_binary() {
local image=$1
local binary=$2
docker run --rm "$image" \
"bin/$binary" --helpxml 2>/dev/null |
sed 's//\n/g' |
grep "${FLAG} " |
grep -o "[^<]* " |
head -n 1 |
sed 's/<[^>]*>//g' |
xargs
}
get_docker_default() {
local image=$1
local val_tserver=""
local val_master=""
if [[ "$COMP" == "tserver" ||
"$COMP" == "both" ]]; then
val_tserver=$(get_flag_from_binary "$image" "yb-tserver")
fi
if [[ "$COMP" == "master" ||
"$COMP" == "both" ]]; then
val_master=$(get_flag_from_binary "$image" "yb-master")
fi
local out=""
if [ -n "$val_tserver" ]; then
out="${val_tserver} [yb-tserver]"
fi
if [ -n "$val_master" ]; then
if [ -n "$out" ]; then
out="${out} | ${val_master} [yb-master]"
else
out="${val_master} [yb-master]"
fi
fi
if [ -z "$out" ]; then
echo "(Flag not found in binary)"
else
echo "$out"
fi
}
echo "Querying compiled binaries via Docker ($COMP)..."
echo "Flag: $FLAG"
echo "----------------------------------------"
TAG1=$(resolve_tag "$V1")
TAG2=$(resolve_tag "$V2")
IMG1="yugabytedb/yugabyte:${TAG1}"
IMG2="yugabytedb/yugabyte:${TAG2}"
echo "Resolved $V1 -> $TAG1"
if [ "$V1" != "$V2" ]; then
echo "Resolved $V2 -> $TAG2"
fi
PULLED_IMAGES=()
for img in "$IMG1" "$IMG2"; do
if ! docker image inspect "$img" >/dev/null 2>&1; then
echo "Pulling image $img (this may take a minute)..."
docker pull -q "$img" >/dev/null
if [[ ! " ${PULLED_IMAGES[*]} " =~ " ${img} " ]]; then
PULLED_IMAGES+=("$img")
fi
fi
done
VAL1=$(get_docker_default "$IMG1")
VAL2=$(get_docker_default "$IMG2")
printf "Result Env 1 (%-10s) : %s\n" "$V1" "$VAL1"
printf "Result Env 2 (%-10s) : %s\n" "$V2" "$VAL2"
echo "----------------------------------------"
if [ "$VAL1" != "$VAL2" ]; then
echo -e "\e[31m[!] DIFFERENCE DETECTED\e[0m"
else
echo -e "\e[32m[β] IDENTICAL\e[0m"
fi
if [ "$CLEANUP" -eq 1 ] &&
[ ${#PULLED_IMAGES[@]} -gt 0 ]; then
echo "----------------------------------------"
echo "Cleaning up downloaded images..."
for img in "${PULLED_IMAGES[@]}"; do
echo "Removing $img..."
docker rmi "$img" >/dev/null 2>&1
done
echo "Cleanup complete."
fi
Save it as:
compare_docker_flags.sh
and make it executable:
chmod +x compare_docker_flags.sh
Prerequisites
You only need Docker, curl, and common UNIX command-line utilities such as grep, sed, sort, and xargs. No YugabyteDB universe needs to be running.
| Requirement | Purpose |
|---|---|
| Docker | Pulls and runs the exact YugabyteDB release binaries. |
| curl | Queries Docker Hub to resolve a base release to a build-specific tag. |
| grep / sed / sort / xargs | Parses the XML output and Docker tag information. |
Example 1: Compare a Flag Across Two Versions
To compare webserver_port between two versions, just provide the base releases:
./compare_docker_flags.sh \
webserver_port \
2.20.10.0 \
2024.1.3.0
Example output:
Querying compiled binaries via Docker (both)...
Flag: webserver_port
----------------------------------------
Resolved 2.20.10.0 ->
Resolved 2024.1.3.0 ->
Result Env 1 (2.20.10.0 ) : 9000 [yb-tserver] | 7000 [yb-master]
Result Env 2 (2024.1.3.0) : 9000 [yb-tserver] | 7000 [yb-master]
----------------------------------------
[β] IDENTICAL
This example also illustrates why checking both processes is useful: the same flag can exist in both yb-master and yb-tserver but have different defaults.
Example 2: Discover When a Flag Appeared
The script can also tell you that a flag did not exist in an older release:
./compare_docker_flags.sh \
master_ysql_operation_lease_ttl_ms \
2.20.10.0 \
2026.1.1.0
Example:
Result Env 1 (2.20.10.0 ) : (Flag not found in binary)
Result Env 2 (2026.1.1.0) : 30000 [yb-tserver] | 30000 [yb-master]
----------------------------------------
[!] DIFFERENCE DETECTED
That answers a slightly different but equally useful upgrade question:
- Did the default change, or was the flag introduced between the two releases?
Example 3: Check Only Master or TServer
Some flags apply only to one process, so you can restrict the comparison to master or tserver. For example, to inspect a RocksDB-related TServer flag:
./compare_docker_flags.sh \
rocksdb_max_background_compactions \
2.20.10.0 \
2024.1.3.0 \
tserver
The script will run only yb-tserver --helpxml, avoiding the unnecessary Master lookup.
Example 4: Automatically Remove Downloaded Images
YugabyteDB Docker images are large enough that you may not want to keep multiple versions around after a quick investigation. Add --cleanup and the script will remove images it downloaded during that run while leaving images that already existed locally untouched.
./compare_docker_flags.sh \
--cleanup \
rocksdb_max_background_compactions \
2.20.10.0 \
2024.1.3.0 \
tserver
Base Version or Exact Build?
The script accepts either a base release such as:
2025.2.5.2
or a full build tag such as:
2025.2.5.2-b5
If you supply only the base release, the script queries Docker Hub and attempts to resolve it to the latest matching build tag. If you already supply -bXX, the exact build is used directly.
Auto-resolution is convenient for quick comparisons, but if you are investigating a specific source or target universe, use its full YugabyteDB build tag. That guarantees you are inspecting the same binaries used by that environment.
Compiled Default vs. Runtime Value
This distinction is important. The script answers what default was compiled into the binary, not necessarily what value your production universe is currently using.
| Question | Where to Look |
|---|---|
| What value was compiled into this release? | yb-master / yb-tserver --helpxml |
| What value is my running process actually using? | Inspect the running process, such as through /varz. |
| Was the value explicitly overridden? | Inspect the universe or YugabyteDB Anywhere configuration. |
| Is an AutoFlag currently promoted? | Inspect the universe’s AutoFlag state. |
That distinction is valuable during troubleshooting because a behavioral change can come from several places: the binary default changed, someone explicitly configured a different value, or the effective runtime state changed.
A Special Note About AutoFlags
AutoFlags require another layer of interpretation. Seeing a new AutoFlag or different metadata in the newer binary does not necessarily mean the corresponding behavior becomes active as soon as the first node is upgraded.
AutoFlags are managed at the cluster level and can be promoted as part of the upgrade lifecycle. Therefore, this script is useful for discovering that an AutoFlag exists or changed between binaries, but actual upgrade troubleshooting should also verify the AutoFlag’s current cluster state.
The script tells you what exists in the binaries. If the flag is an AutoFlag, also verify whether it has actually been promoted in the running universe before attributing a behavior change to it.
A Practical Upgrade Workflow
When someone asks, “Did this flag change as part of our upgrade?”, start by comparing the exact source and target binaries with the script. If the values differ, determine whether the flag applies to Master, TServer, or both, check whether it is an AutoFlag, and then inspect the actual source and target universes for overrides and runtime values.
That sequence helps separate three very different explanations:
- β The compiled default changed.
- β The configured value changed.
- β The runtime or AutoFlag state changed.
Knowing which one occurred can save a lot of time during an upgrade investigation.
Final Takeaway
Documentation and source-code searches are valuable, but sometimes you need a definitive answer for two exact YugabyteDB builds. Docker plus --helpxml gives you a quick way to interrogate the actual yb-master and yb-tserver binaries without provisioning either cluster.
That makes this technique particularly useful for upgrade preparation, patch-to-patch comparisons, regression investigations, newly introduced flags, and cases where Master and TServer may have different defaults.
The question the script answers is very specific… and very useful:
- Did the compiled GFlag default actually change between these two YugabyteDB builds?
Use
--helpxml to determine what was compiled into the two YugabyteDB releases, then compare that with the effective runtime configuration when investigating a real universe. Together, those views help determine whether the behavior changed because of the software or because of configuration.
Have Fun!
Fun night at Bowlmor Lanes in Cupertino with the YugabyteDB engineering team…
Good food, drinks, and a little friendly competition! π³π»
I even managed to break 100 in one game and may have had one of the highest scores on the team. π Thatβs probably a good thingβ¦ Iβd much rather have the YB engineering team be better at coding than bowling! π
