Skip to content

[codex] add evm-only executor load test harness#3658

Draft
codchen wants to merge 4 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest
Draft

[codex] add evm-only executor load test harness#3658
codchen wants to merge 4 commits into
codex/evmonly-staking-dynamic-gasfrom
codex/evm-only-executor-loadtest

Conversation

@codchen

@codchen codchen commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a standalone evmonly-loadtest command that continuously feeds generated EVM transfer blocks into the EVM-only executor with generated in-memory genesis state, discard state/receipt sinks, and Prometheus/stdout throughput metrics.

The harness is intended for optimistic executor saturation testing with one external executor worker and internal OCC workers. It defaults recipients to unique addresses and keeps the workload structure isolated so future ERC20-style transactions can be added beside the transfer generator.

This also adds --prebuild-blocks, which generates all bounded blocks before starting executor workers. That mode separates block-generation throughput from executor throughput for diagnosing whether builders are competing with execution.

The current branch also freezes generated prebuilt genesis state before execution so the synthetic StateReader can serve immutable state without lock/copy overhead during the executor-only measurement.

Adds OCC diagnostics for future ERC20-style load tests: block results now report OCC attempts, sequential fallbacks, fallback reasons, and aggregated conflict keys. The loadtest exports Prometheus counters for attempts/fallbacks/conflicts, fallback reasons, and conflict keys labeled by access/kind/address/slot; stdout totals include occ_attempts, occ_fallbacks, and occ_conflicts.

Validation

  • go test ./giga/evmonly/...
  • Local smoke: go run ./giga/evmonly/cmd/evmonly-loadtest --metrics-addr= --report-interval=0 --prebuild-blocks --blocks=20 --txs-per-block=1000 --builders=16 --workers=1 --executor-workers=12 --gas-price-wei=0 --min-gas-price-wei=0 --queue-size=64

EC2 benchmark

The 199k/202k EC2 benchmark numbers below were run from commit 4ec8da52c on a c8i.48xlarge in us-east-1a with SMT disabled using CoreCount=96,ThreadsPerCore=1, Go 1.25.6, one external executor worker, prebuilt blocks, unique senders/recipients, and zero gas price/min gas price. The current head 71eba7a0f adds metrics only on top of that benchmarked path.

Reproduce the 199.3k TPS run

Launch the instance with CPU options equivalent to:

aws ec2 run-instances \
  --instance-type c8i.48xlarge \
  --cpu-options CoreCount=96,ThreadsPerCore=1 \
  --image-id <al2023-x86_64-ami> \
  --key-name <key> \
  --security-group-ids <sg> \
  --subnet-id <subnet> \
  --associate-public-ip-address

Build the loadtest binary on the instance:

curl -fsSL https://go.dev/dl/go1.25.6.linux-amd64.tar.gz -o /tmp/go1.25.6.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf /tmp/go1.25.6.linux-amd64.tar.gz
git clone --branch codex/evm-only-executor-loadtest https://github.com/sei-protocol/sei-chain.git
cd sei-chain
/usr/local/go/bin/go build -o /tmp/evmonly-loadtest ./giga/evmonly/cmd/evmonly-loadtest

Run the benchmark:

GOMAXPROCS=96 GOGC=200 /tmp/evmonly-loadtest \
  --metrics-addr "" \
  --blocks 1000 \
  --txs-per-block 5000 \
  --prebuild-blocks \
  --builders 96 \
  --workers 1 \
  --executor-workers 96 \
  --gas-price-wei 0 \
  --min-gas-price-wei 0 \
  --report-interval 5s

Observed output:

prebuild complete elapsed=9.366s blocks=1000 txs=5000000 build_blocks/s=106.77 build_tx/s=533873.27
complete elapsed=25.09s input_blocks=1000 finished_blocks=1000 txs=5000000 gas=105000000000 errors=0 avg_input_blocks/s=39.86 avg_finished_blocks/s=39.86 avg_tx/s=199284.61 avg_gas/s=4184976805.58

For comparison, the same shape and command with GOGC=400 crossed the 200k target:

prebuild complete elapsed=8.918s blocks=1000 txs=5000000 build_blocks/s=112.14 build_tx/s=560690.94
complete elapsed=24.643s input_blocks=1000 finished_blocks=1000 txs=5000000 gas=105000000000 errors=0 avg_input_blocks/s=40.58 avg_finished_blocks/s=40.58 avg_tx/s=202893.72 avg_gas/s=4260768150.96

Other comparisons from the same testing round:

  • c8i.48xlarge with default SMT topped out around 171k TPS on the 2M-tx sweep.
  • c8i.48xlarge with SMT disabled and default GC reached about 191k TPS over 5M tx.
  • c7a.48xlarge topped out around 161k TPS on the 2M-tx sweep.
  • Larger 10k tx/block blocks regressed to about 184k TPS over 5M tx.

@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJun 30, 2026, 8:14 AM

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 59.01202% with 307 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.30%. Comparing base (7c6c44c) to head (71eba7a).

Files with missing lines Patch % Lines
giga/evmonly/cmd/evmonly-loadtest/main.go 56.65% 247 Missing and 43 partials ⚠️
giga/evmonly/occ.go 78.75% 16 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@                          Coverage Diff                          @@
##           codex/evmonly-staking-dynamic-gas    #3658      +/-   ##
=====================================================================
+ Coverage                              58.15%   58.30%   +0.15%     
=====================================================================
  Files                                   2185     2179       -6     
  Lines                                 178564   177862     -702     
=====================================================================
- Hits                                  103836   103701     -135     
+ Misses                                 65413    65028     -385     
+ Partials                                9315     9133     -182     
Flag Coverage Δ
sei-chain-pr 68.93% <59.01%> (+12.63%) ⬆️
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
giga/evmonly/occ.go 75.36% <78.75%> (+1.70%) ⬆️
giga/evmonly/cmd/evmonly-loadtest/main.go 56.65% <56.65%> (ø)

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codchen codchen force-pushed the codex/evmonly-staking-dynamic-gas branch from bd57e4c to 7c6c44c Compare June 30, 2026 06:11
@codchen codchen force-pushed the codex/evm-only-executor-loadtest branch from 8e28929 to 4ec8da5 Compare June 30, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant