Compare GFlag Defaults Across Versions Without Starting a Cluster

When preparing for a YugabyteDB upgrade or investigating a behavior change, a common question comes up:

  • Did the default value of this GFlag change between my current version and the version I’m upgrading to?

You could search the documentation or GitHub, or even stand up two clusters and compare their /varz output. But if what you really want to know is what default value was compiled into a specific yb-master or yb-tserver binary, you can ask the binaries directly.

YugabyteDB binaries support the gflags option:

				
					--helpxml
				
			

This emits the flags compiled into the binary as structured XML, including their hard-coded default values. Combine that with official YugabyteDB Docker images and you can compare two releases without ever starting a YugabyteDB cluster.

πŸ’‘ What are we really comparing?
This technique compares the compiled default of a GFlag in specific yb-master and/or yb-tserver binaries. It does not tell you what a running YugabyteDB universe is currently using after configuration overrides, AutoFlag promotion, or other runtime changes.

Why Not Just Check the Documentation?

The documentation is still a great place to start, but binary-to-binary comparison becomes especially useful when you need an answer for an exact release build.

Documentation is generally organized around release families rather than every patch and build, so a patch-level change may not be obvious. Some advanced or internal flags may also be difficult to find, and source-code searches introduce another question: Which exact commit or backport made it into the released build?

The released binary removes that ambiguity. If you want to know what was compiled into 2024.1.3.0-bXX, inspect that exact binary.

This approach also avoids the “two cluster” problem. You do not need to provision two environments, start Masters and TServers, query them, and tear everything down. Docker provides the binaries, --helpxml prints their flag metadata, and the process exits.

The --helpxml Trick

For example, inside a YugabyteDB installation you can run:

				
					bin/yb-tserver --helpxml
				
			

The returned XML contains entries such as:

				
					<name>webserver_port</name>
<default>9000</default>
				
			

That means we can programmatically ask each release binary:

  • What is your compiled default for this exact GFlag?

We can do the same against yb-master, which is important because a flag may exist in one process, both processes, or have different defaults between them.

The Script: compare_docker_flags.sh

The following script automates the entire process. It accepts two YugabyteDB versions, resolves base versions to Docker build tags, pulls the required images, runs yb-master and/or yb-tserver with --helpxml, extracts the requested flag, compares the values, and optionally removes images that it downloaded.

				
					#!/bin/bash

# Usage:
# ./compare_docker_flags.sh [--cleanup] \
#   <flag_name> <version1> <version2> [master|tserver|both]
#
# Example:
# ./compare_docker_flags.sh --cleanup \
#   webserver_port 2.20.10.0 2024.1.3.0 master

CLEANUP=0

while [[ "$#" -gt 0 ]]; do
    case $1 in
        --cleanup)
            CLEANUP=1
            shift
            ;;
        -*)
            echo -e "\e[31m[!] Unknown option: $1\e[0m"
            exit 1
            ;;
        *)
            break
            ;;
    esac
done

FLAG=$1
V1=$2
V2=$3
COMP=${4:-both}

if [ "$#" -lt 3 ] || [ "$#" -gt 4 ]; then
    echo -e "\e[31m[!] Error: Invalid number of arguments.\e[0m"
    echo "Usage: $0 [--cleanup] <flag_name> <version1> <version2> [master|tserver|both]"
    exit 1
fi

if [[ ! "$FLAG" =~ ^[a-zA-Z_][a-zA-Z0-9_-]*$ ]]; then
    echo -e "\e[31m[!] Error: '$FLAG' does not look like a valid flag name.\e[0m"
    exit 1
fi

for v in "$V1" "$V2"; do
    if [[ ! "$v" =~ ^[0-9] ]]; then
        echo -e "\e[31m[!] Error: Invalid version string '$v'. Versions must start with a number.\e[0m"
        exit 1
    fi
done

if [[ "$COMP" != "master" &&
      "$COMP" != "tserver" &&
      "$COMP" != "both" ]]; then
    echo -e "\e[31m[!] Error: Invalid component parameter '$COMP'. Use master, tserver, or both.\e[0m"
    exit 1
fi

resolve_tag() {
    local base_version=$1

    if [[ "$base_version" == *-b* ]]; then
        echo "$base_version"
        return
    fi

    local resolved_tag

    resolved_tag=$(
        curl -s \
          "https://hub.docker.com/v2/repositories/yugabytedb/yugabyte/tags?page_size=100&name=${base_version}" |
        grep -o '"name":"[^"]*"' |
        cut -d'"' -f4 |
        grep -E "^${base_version}(\.[0-9]+)*-b[0-9]+$" |
        sort -V |
        tail -n 1
    )

    if [ -n "$resolved_tag" ]; then
        echo "$resolved_tag"
    else
        echo "$base_version"
    fi
}

get_flag_from_binary() {
    local image=$1
    local binary=$2

    docker run --rm "$image" \
        "bin/$binary" --helpxml 2>/dev/null |
        sed 's/<flag>/\n<flag>/g' |
        grep "<name>${FLAG}</name>" |
        grep -o "<default>[^<]*</default>" |
        head -n 1 |
        sed 's/<[^>]*>//g' |
        xargs
}

get_docker_default() {
    local image=$1
    local val_tserver=""
    local val_master=""

    if [[ "$COMP" == "tserver" ||
          "$COMP" == "both" ]]; then
        val_tserver=$(get_flag_from_binary "$image" "yb-tserver")
    fi

    if [[ "$COMP" == "master" ||
          "$COMP" == "both" ]]; then
        val_master=$(get_flag_from_binary "$image" "yb-master")
    fi

    local out=""

    if [ -n "$val_tserver" ]; then
        out="${val_tserver} [yb-tserver]"
    fi

    if [ -n "$val_master" ]; then
        if [ -n "$out" ]; then
            out="${out} | ${val_master} [yb-master]"
        else
            out="${val_master} [yb-master]"
        fi
    fi

    if [ -z "$out" ]; then
        echo "(Flag not found in binary)"
    else
        echo "$out"
    fi
}

echo "Querying compiled binaries via Docker ($COMP)..."
echo "Flag: $FLAG"
echo "----------------------------------------"

TAG1=$(resolve_tag "$V1")
TAG2=$(resolve_tag "$V2")

IMG1="yugabytedb/yugabyte:${TAG1}"
IMG2="yugabytedb/yugabyte:${TAG2}"

echo "Resolved $V1 -> $TAG1"

if [ "$V1" != "$V2" ]; then
    echo "Resolved $V2 -> $TAG2"
fi

PULLED_IMAGES=()

for img in "$IMG1" "$IMG2"; do
    if ! docker image inspect "$img" >/dev/null 2>&1; then
        echo "Pulling image $img (this may take a minute)..."
        docker pull -q "$img" >/dev/null

        if [[ ! " ${PULLED_IMAGES[*]} " =~ " ${img} " ]]; then
            PULLED_IMAGES+=("$img")
        fi
    fi
done

VAL1=$(get_docker_default "$IMG1")
VAL2=$(get_docker_default "$IMG2")

printf "Result Env 1 (%-10s) : %s\n" "$V1" "$VAL1"
printf "Result Env 2 (%-10s) : %s\n" "$V2" "$VAL2"

echo "----------------------------------------"

if [ "$VAL1" != "$VAL2" ]; then
    echo -e "\e[31m[!] DIFFERENCE DETECTED\e[0m"
else
    echo -e "\e[32m[βœ“] IDENTICAL\e[0m"
fi

if [ "$CLEANUP" -eq 1 ] &&
   [ ${#PULLED_IMAGES[@]} -gt 0 ]; then

    echo "----------------------------------------"
    echo "Cleaning up downloaded images..."

    for img in "${PULLED_IMAGES[@]}"; do
        echo "Removing $img..."
        docker rmi "$img" >/dev/null 2>&1
    done

    echo "Cleanup complete."
fi
				
			

Save it as:

				
					compare_docker_flags.sh
				
			

and make it executable:

				
					chmod +x compare_docker_flags.sh
				
			

Prerequisites

You only need Docker, curl, and common UNIX command-line utilities such as grep, sed, sort, and xargs. No YugabyteDB universe needs to be running.

Requirement Purpose
Docker Pulls and runs the exact YugabyteDB release binaries.
curl Queries Docker Hub to resolve a base release to a build-specific tag.
grep / sed / sort / xargs Parses the XML output and Docker tag information.

Example 1: Compare a Flag Across Two Versions

To compare webserver_port between two versions, just provide the base releases:

				
					./compare_docker_flags.sh \
    webserver_port \
    2.20.10.0 \
    2024.1.3.0
				
			

Example output:

				
					Querying compiled binaries via Docker (both)...
Flag: webserver_port
----------------------------------------
Resolved 2.20.10.0 -> <resolved-build-tag>
Resolved 2024.1.3.0 -> <resolved-build-tag>

Result Env 1 (2.20.10.0 ) : 9000 [yb-tserver] | 7000 [yb-master]
Result Env 2 (2024.1.3.0) : 9000 [yb-tserver] | 7000 [yb-master]

----------------------------------------
[βœ“] IDENTICAL
				
			

This example also illustrates why checking both processes is useful: the same flag can exist in both yb-master and yb-tserver but have different defaults.

Example 2: Discover When a Flag Appeared

The script can also tell you that a flag did not exist in an older release:

				
					./compare_docker_flags.sh \
    master_ysql_operation_lease_ttl_ms \
    2.20.10.0 \
    2026.1.1.0
				
			

Example:

				
					Result Env 1 (2.20.10.0 ) : (Flag not found in binary)
Result Env 2 (2026.1.1.0) : 30000 [yb-tserver] | 30000 [yb-master]

----------------------------------------
[!] DIFFERENCE DETECTED
				
			

That answers a slightly different but equally useful upgrade question:

  • Did the default change, or was the flag introduced between the two releases?

Example 3: Check Only Master or TServer

Some flags apply only to one process, so you can restrict the comparison to master or tserver. For example, to inspect a RocksDB-related TServer flag:

				
					./compare_docker_flags.sh \
    rocksdb_max_background_compactions \
    2.20.10.0 \
    2024.1.3.0 \
    tserver
				
			

The script will run only yb-tserver --helpxml, avoiding the unnecessary Master lookup.

Example 4: Automatically Remove Downloaded Images

YugabyteDB Docker images are large enough that you may not want to keep multiple versions around after a quick investigation. Add --cleanup and the script will remove images it downloaded during that run while leaving images that already existed locally untouched.

				
					./compare_docker_flags.sh \
    --cleanup \
    rocksdb_max_background_compactions \
    2.20.10.0 \
    2024.1.3.0 \
    tserver
				
			

Base Version or Exact Build?

The script accepts either a base release such as:

				
					2025.2.5.2
				
			

or a full build tag such as:

				
					2025.2.5.2-b5
				
			

If you supply only the base release, the script queries Docker Hub and attempts to resolve it to the latest matching build tag. If you already supply -bXX, the exact build is used directly.

🎯 For upgrade investigations, use the exact build when possible.
Auto-resolution is convenient for quick comparisons, but if you are investigating a specific source or target universe, use its full YugabyteDB build tag. That guarantees you are inspecting the same binaries used by that environment.

Compiled Default vs. Runtime Value

This distinction is important. The script answers what default was compiled into the binary, not necessarily what value your production universe is currently using.

Question Where to Look
What value was compiled into this release? yb-master / yb-tserver --helpxml
What value is my running process actually using? Inspect the running process, such as through /varz.
Was the value explicitly overridden? Inspect the universe or YugabyteDB Anywhere configuration.
Is an AutoFlag currently promoted? Inspect the universe’s AutoFlag state.

That distinction is valuable during troubleshooting because a behavioral change can come from several places: the binary default changed, someone explicitly configured a different value, or the effective runtime state changed.

A Special Note About AutoFlags

AutoFlags require another layer of interpretation. Seeing a new AutoFlag or different metadata in the newer binary does not necessarily mean the corresponding behavior becomes active as soon as the first node is upgraded.

AutoFlags are managed at the cluster level and can be promoted as part of the upgrade lifecycle. Therefore, this script is useful for discovering that an AutoFlag exists or changed between binaries, but actual upgrade troubleshooting should also verify the AutoFlag’s current cluster state.

⚠️ Don’t confuse binary metadata with AutoFlag state.
The script tells you what exists in the binaries. If the flag is an AutoFlag, also verify whether it has actually been promoted in the running universe before attributing a behavior change to it.

A Practical Upgrade Workflow

When someone asks, “Did this flag change as part of our upgrade?”, start by comparing the exact source and target binaries with the script. If the values differ, determine whether the flag applies to Master, TServer, or both, check whether it is an AutoFlag, and then inspect the actual source and target universes for overrides and runtime values.

That sequence helps separate three very different explanations:

  • ● The compiled default changed.
  • ● The configured value changed.
  • ● The runtime or AutoFlag state changed.

Knowing which one occurred can save a lot of time during an upgrade investigation.

Final Takeaway

Documentation and source-code searches are valuable, but sometimes you need a definitive answer for two exact YugabyteDB builds. Docker plus --helpxml gives you a quick way to interrogate the actual yb-master and yb-tserver binaries without provisioning either cluster.

That makes this technique particularly useful for upgrade preparation, patch-to-patch comparisons, regression investigations, newly introduced flags, and cases where Master and TServer may have different defaults.

The question the script answers is very specific… and very useful:

  • Did the compiled GFlag default actually change between these two YugabyteDB builds?
πŸ’‘ Binary first, runtime second.
Use --helpxml to determine what was compiled into the two YugabyteDB releases, then compare that with the effective runtime configuration when investigating a real universe. Together, those views help determine whether the behavior changed because of the software or because of configuration.

Have Fun!

Fun night at Bowlmor Lanes in Cupertino with the YugabyteDB engineering team…

Good food, drinks, and a little friendly competition! 🎳🍻

I even managed to break 100 in one game and may have had one of the highest scores on the team. πŸ˜‚ That’s probably a good thing… I’d much rather have the YB engineering team be better at coding than bowling! πŸ˜€