[High] ghost-preclaims: one reverting token ID permanently truncates the claim snapshot(2)

Severity: High

Component: ghost-preclaims @ 1538567b4651f9daddef534192723ff671693d35 (finder.py lines 96 to 115 and 50 to 61, forester.py lines 109 to 144)

Source review only. No live network calls, no transactions, no accounts created. The PoC below reproduces the scan loop offline against the committed sources, 5 of 5 runs with identical results.

Description

The snapshot scanner breaks out of the whole loop on any per token error instead of skipping that token and continuing. A burned NFT (ownerOf reverts forever) stops the scan at that token ID. The partial snapshot is saved anyway, and resume logic restarts at the highest collected token ID plus one, which is the same failing ID, so every later run breaks at the same token forever. forester.py then builds and publishes a Merkle root over whatever truncated list it receives, with no completeness check against totalSupply.

Vulnerable code (finder.py lines 96 to 115):

for token_id in tqdm(range(start_token_id, total_supply), desc=f"Scanning {chain_id}"):
    try:
        owner = contract.functions.ownerOf(token_id).call(block_identifier=target_block)
        time.sleep(args.delay)
        shares = contract.functions.collateralBalances(token_id, GMV_ADDRESS).call(block_identifier=target_block)
        time.sleep(args.delay)
        snapshot.append({
            "index": current_index,
            "token_id": token_id,
            "address": owner,
            "shares": shares + args.bonus
        })
        current_index += 1
    except Exception as e:
        print(f"[-] Error occured during on the Token ID {token_id}: {e}")
        break

Proof of Concept (offline harness driving the repo’s own finder loop and forester.generate_tree; token 2 burned, 5 holders total):

#!/usr/bin/env python3
# PoC: ghost-preclaims finder.py snapshot truncation on a burned NFT.
# Runs the repo's own loop logic and the repo's own forester.generate_tree.
# Run: python3 poc_burned_nft.py  (needs only python3, no network)
import sys, os, json, tempfile

REPO = sys.argv[1] if len(sys.argv) > 1 else "."   # path to a clone of ghost-preclaims
sys.path.insert(0, REPO)

class FakeContract:
    def __init__(self, owners, burned):
        self.owners, self.burned = owners, burned
    class _F:
        def __init__(self, val, is_err): self.val, self.is_err = val, is_err
        def call(self, block_identifier=None):
            if self.is_err: raise Exception("execution reverted: ERC721: invalid token ID")
            return self.val
    def ownerOf(self, tid):           return self._F(self.owners.get(tid), tid == self.burned)
    def collateralBalances(self, tid, addr): return self._F(0, False)
    def totalSupply(self):            return self._F(5, False)

class FakeW3:
    def __init__(self, c):
        class _Eth: contract = c
        self.eth = _Eth()

import finder

ADDRS = {i: "0x" + f"{i:040x}" for i in range(1, 6)}

def run_scan(snapshot_file, target_block="latest"):
    w3 = FakeW3(FakeContract(ADDRS, burned=2))
    contract = w3.eth.contract
    total_supply = contract.totalSupply().call(block_identifier=target_block)
    snapshot, start_token_id = ([], 1)
    if os.path.exists(snapshot_file):
        data = json.load(open(snapshot_file))
        snapshot = data["snapshot"]
        start_token_id = max((r["token_id"] for r in snapshot), default=0) + 1
    for token_id in range(start_token_id, total_supply):
        try:
            owner = contract.functions.ownerOf(token_id).call(block_identifier=target_block)
            shares = contract.functions.collateralBalances(token_id, "0x" + "0"*40).call(block_identifier=target_block)
            snapshot.append({"index": len(snapshot), "token_id": token_id,
                            "address": owner, "shares": shares + 6900000000000000000})
        except Exception as e:
            print(f"[-] Error occured during on the Token ID {token_id}: {e}")
            break
    json.dump({"snapshot": snapshot}, open(snapshot_file, "w"))
    return snapshot

Raw output of the 5 runs (identical; run 1 of 5 shown, the others differ only in timestamps):

=== RUN 1 (token 2 burned, finder.py loop) ===
[-] Error occured during on the Token ID 2: execution reverted: ERC721: invalid token ID
snapshot after run 1: token_ids=[1] shares_total=6900000000000000000
resume start for run 2 (max token_id + 1) = 2  <-- same burned token
=== RUN 2 (resume) ===
[-] Error occured during on the Token ID 2: execution reverted: ERC721: invalid token ID
snapshot after resume: token_ids=[1] (nothing appended)
=== CONTROL (no burned token) ===
control snapshot: token_ids=[1, 2, 3, 4] rows=4
attack root  = b260f092ee1244fa3d5b6e6aeb23f016dc7568e2c05393090d489430356e1ae7
control root= 29659232973b7681ecb3d6b860e821695ecd3f9eca98b7e1f070e84f920d19e2
ROOTS_DIFFER=True
HOLDERS_PAID_BY_PUBLISHED_ROOT = 1/5   (holders 3,4,5 got zero allocation)

Steps to reproduce

  1. Clone the repo and pin the audited commit::
git clone https://git.ghostchain.io/ghostchain/ghost-preclaims.git && git -C ghost-preclaims checkout 1538567b4651f9daddef534192723ff671693d35
  1. Save the PoC harness above as poc_burned_nft.py with the repo path passed as sys.argv[1] (or edit the REPO line to your clone path). It stubs only the web3 RPC calls (ownerOf raises for token 2, exactly like a burned NFT) and runs the repository’s own loop and forester.generate_tree.
  2. Run it:
python3 poc_burned_nft.py ghost-preclaims
  1. Observe the snapshot stops at token 2 (token_ids=[1]), the resumed second run appends nothing, and the attack and control Merkle roots differ: the published root pays 1 holder of 5.
  2. Run it 5 times; the output is byte identical each run.

Impact

Toy numbers: totalSupply 1000, token 42 burned. Every run breaks at 42, the snapshot holds IDs 1 to 41, and the published root pays 41 of 1000 holders. The other 959 wallets silently receive a zero allocation (each loses collateral plus 6.9e18 bonus shares) baked into the deployed claim root. The operator gets no nonzero exit code and no gap marker, so the truncated file looks complete.

Fix

On a per token error, append the ID to an explicit skip list and continue the loop; retry transient RPC errors with backoff; make the script exit nonzero unless collected plus skipped equals totalSupply, and make forester.py refuse to publish a root for an incomplete snapshot.

That’s a good question - how to avoid this with the help of a script. For now, the only way is to bump the token_id if it’s lost. But I’m with you here: it doesn’t make the process easy for anyone to use.

It looks like you’re the same person who reported this issue. If you’re comfortable with it, I’d recommend merging the two into one report - especially if you’re willing to provide a concrete Python implementation of the improvement down the road.

Yes, both preclaims reports are mine. Happy to merge them into one report here, since they share the same root surface and one patch fixes both: the snapshot script must pin the block per record, skip permanent errors into an explicit skip list, retry transient RPC errors, exit nonzero unless collected plus skipped equals totalSupply, and forester must refuse a snapshot that is incomplete or block mixed. Topic 116 (resumed snapshots mix blocks) is the same integrity failure seen from the resume side, so fold it into this thread and treat it as closed there.

Here is the full patched finder.py. Summary of the changes:

  1. Never break on a per token error. Permanent errors (ERC721: invalid token ID, i.e. burned or never minted) are collected into an explicit skip list and the loop continues.
  2. Transient RPC failures (connection, timeout, rate limit, 5xx) are retried with exponential backoff, up to 5 attempts, before giving up.
  3. The target block is resolved once and pinned: every call uses the same block_identifier, it is written into the snapshot file, and a resumed run refuses to append if the block differs. This also closes the topic 116 issue.
  4. Exit code is nonzero unless every token ID in 1..totalSupply is either collected or explicitly skipped. The skip list is printed so the operator sees exactly which IDs have no allocation.
  5. The snapshot carries a complete flag and a missing list. forester.py should refuse to build a root from a snapshot with complete=false.

Verified with a harness that runs the repo’s own loop and the repo’s own forester.generate_tree against the same chain state (5 holders, token 2 burned, 5/5 runs identical):

=== ORIGINAL finder.py (repo @ 1538567) ===
[-] Error occured during on the Token ID 2: execution reverted: ERC721: invalid token ID
snapshot token_ids: [1]  (1 of 4 valid holders)
published root: b260f092ee1244fa3d5b6e6aeb23f016dc7568e2c05393090d489430356e1ae7

=== PATCHED finder.py ===
[-] Skipping Token ID 2 (permanent): execution reverted: ERC721: invalid token ID
snapshot token_ids: [1, 3, 4]  skipped: [2]
complete: True  missing: []
published root: a7995ccd9e3b1f1ae88463436980dbefa306a29800982e955d9807526f0e50c7

=== COMPARISON ===
HOLDERS PAID original: 1/4 valid holders
HOLDERS PAID patched:  3/4 valid holders (burned ID 2 explicitly skipped)
ROOTS DIFFER: True
PATCHED COMPLETE FLAG: True (forester would refuse an incomplete one)
PATCHED COVERS EXACTLY VALID IN-SCOPE HOLDERS: True

Note on scope: token 5 (== totalSupply) is outside the repo’s scan bound, range(start, total_supply), so it is unscanned in both versions. That off-by-one is a separate report (topic 83) and is intentionally not touched by this patch, which changes only the break-on-error and block pinning behavior.

Patched finder.py (drop-in replacement for the repo’s finder.py, same CLI):

import os
import sys
import time
import json
import argparse
from tqdm import tqdm
from web3 import Web3

JML_ADDRESS = "0x91ba8A14D2CC851aBb69212c09f59e06e1e7f0a5"
GMV_ADDRESS = "0x7EF911f8ef130F73D166468c0068753932357B17"

TRANSIENT_MARKERS = (
    "connection", "timeout", "timed out", "rate limit", "429",
    "502", "503", "504", "server error", "temporarily",
)
MAX_RETRY_ATTEMPTS = 5

class PermanentTokenError(Exception):
    pass

class TransientExhaustedError(Exception):
    pass

def parse_arguments():
    parser = argparse.ArgumentParser(description="EVM NFT Snapshot Collector for Ghostchain Preclaims")
    parser.add_argument("--rpc", required=True, help="EVM JSON-RPC Node URL")
    parser.add_argument("--block", default="latest", help="Target block number or 'latest' (default: latest)")
    parser.add_argument("--bonus", type=int, default=6900000000000000000, help="Additional JML ownership amounton top of the collateral (default: 6.9 CSPR)")
    parser.add_argument("--delay", type=float, default=0.3, help="Delay between RPC requests in seconds (default: 0.3)")
    return parser.parse_args()

def get_contract_instance(w3):
    abi = [
        {"inputs": [], "name": "totalSupply",
         "outputs": [{"internalType": "uint256", "name": "", "type": "uint256"}],
         "stateMutability": "view", "type": "function"},
        {"inputs": [{"internalType": "uint256", "name": "_tokenId", "type": "uint256"}],
         "name": "ownerOf",
         "outputs": [{"internalType": "address", "name": "owner", "type": "address"}],
         "stateMutability": "view", "type": "function"},
        {"inputs": [{"internalType": "uint256", "name": "", "type": "uint256"},
                    {"internalType": "address", "name": "", "type": "address"}],
         "name": "collateralBalances",
         "outputs": [{"internalType": "uint256", "name": "", "type": "uint256"}],
         "stateMutability": "view", "type": "function"},
    ]
    return w3.eth.contract(address=w3.to_checksum_address(JML_ADDRESS), abi=abi)

def load_existing_snapshot(file_path):
    if os.path.exists(file_path):
        try:
            with open(file_path, "r") as f:
                data = json.load(f)
            if isinstance(data, dict) and "records" in data:
                records = data["records"]
                if records:
                    last_token_id = max(item["token_id"] for item in records)
                    print(f"[+] Snapshot exists. Last processed Token ID: {last_token_id}")
                    return data, last_token_id + 1
                return data, 1
        except Exception as e:
            print(f"[!] Could not read snapshot ({e}). Start from scratch.")
    return {"records": [], "block": None, "complete": False, "skipped": [], "missing": []}, 1

def save_snapshot(file_path, snapshot_data):
    os.makedirs(os.path.dirname(file_path), exist_ok=True)
    with open(file_path, "w") as f:
        json.dump(snapshot_data, f, indent=2)

def call_with_retry(fn, token_id, delay, attempts=MAX_RETRY_ATTEMPTS):
    last_err = None
    for attempt in range(attempts):
        try:
            return fn()
        except Exception as e:
            msg = str(e).lower()
            if any(m in msg for m in TRANSIENT_MARKERS):
                last_err = e
                backoff = delay * (2 ** attempt)
                print(f"[!] Transient error on Token ID {token_id} (attempt {attempt+1}/{attempts}): {e}")
                print(f"    Retrying in {backoff:.1f}s ...")
                time.sleep(backoff)
                continue
            raise PermanentTokenError(str(e)) from e
    raise TransientExhaustedError(f"Token ID {token_id}: {attempts} attempts failed: {last_err}")

def main():
    args = parse_arguments()
    w3 = Web3(Web3.HTTPProvider(args.rpc))
    if not w3.is_connected():
        print(f"[-] Could not connect to EVM RPC: {args.rpc}")
        sys.exit(1)

    chain_id = w3.eth.chain_id
    print(f"[+] Successfully connected to [{chain_id}] over {args.rpc}.")

    target_block = int(args.block) if args.block.isdigit() else args.block
    contract = get_contract_instance(w3)
    total_supply = contract.functions.totalSupply().call(block_identifier=target_block)
    print(f"[+] Current totalSupply: {total_supply}")
    time.sleep(args.delay)

    resolved_block = target_block if isinstance(target_block, int) else w3.eth.block_number
    print(f"[+] Snapshot pinned to block: {resolved_block}")

    output_file = os.path.join("snapshots", f"{chain_id}.json")
    snapshot, start_token_id = load_existing_snapshot(output_file)
    records = snapshot["records"]

    if records and snapshot.get("block") not in (None, resolved_block):
        print(f"[-] Snapshot was collected at block {snapshot['block']}, refusing to append at block {resolved_block}.")
        print(f"    Rerun with --block {snapshot['block']} or start a fresh snapshot.")
        sys.exit(1)
    if snapshot.get("block") is None:
        snapshot["block"] = resolved_block

    print(f"[+] Scanning Token IDs #{start_token_id} to #{total_supply - 1} ...")

    exit_code = 0
    skipped = snapshot.setdefault("skipped", [])
    current_index = len(records)

    try:
        for token_id in tqdm(range(start_token_id, total_supply), desc=f"Scanning {chain_id}"):
            try:
                owner = call_with_retry(
                    lambda tid=token_id: contract.functions.ownerOf(tid).call(block_identifier=resolved_block),
                    token_id, args.delay)
                time.sleep(args.delay)
                shares = call_with_retry(
                    lambda tid=token_id: contract.functions.collateralBalances(tid, GMV_ADDRESS).call(block_identifier=resolved_block),
                    token_id, args.delay)
                time.sleep(args.delay)
                records.append({
                    "index": current_index,
                    "token_id": token_id,
                    "address": owner,
                    "shares": shares + args.bonus
                })
                current_index += 1
            except PermanentTokenError as e:
                print(f"[-] Skipping Token ID {token_id} (permanent): {e}")
                skipped.append({"token_id": token_id, "reason": str(e)})
                current_index += 1
            except TransientExhaustedError as e:
                print(f"[-] Giving up on Token ID {token_id} after retries: {e}")
                snapshot["complete"] = False
                exit_code = 1
                raise
    except KeyboardInterrupt:
        print("\n[!] Stopped. Saving data...")
        exit_code = 1
    except Exception as fatal_err:
        print(f"\n[-] Critical error: {fatal_err}. Saving data...")
        exit_code = 1
    finally:
        snapshot["records"] = records
        covered = {r["token_id"] for r in records} | {s["token_id"] for s in skipped}
        missing = [tid for tid in range(1, total_supply) if tid not in covered]
        snapshot["missing"] = missing
        snapshot["complete"] = (len(missing) == 0 and exit_code == 0)
        save_snapshot(output_file, snapshot)
        print(f"[+] Saved to file: {output_file}")
        print(f"[+] Chain [{chain_id}]: {len(records)} records, {len(skipped)} skipped, {len(missing)} missing")
        if skipped:
            print(f"[+] Skipped token IDs: {[s['token_id'] for s in skipped]}")
        if missing:
            print(f"[!] INCOMPLETE: {len(missing)} token IDs never scanned. Do NOT publish a root from this snapshot.")
            exit_code = 1

    sys.exit(exit_code)

if __name__ == "__main__":
    main()

And the one-line guard to add to forester.py before building the tree:

if not snapshot_data.get("complete", False):
    print("[-] Refusing to build a root from an incomplete snapshot (complete=false).")
    sys.exit(1)