Advanced Usage & CLI Reference
Skip the TUI and work directly with the SDK CLI, pipeline stages, and container configuration.
Advanced Usage, Configuration & CLI Commands
This section covers the internals of REE for power users, integrators, and anyone who wants to understand what happens under the hood or drive REE directly from the command line.
CLI Reference
While the TUI is the recommended interface, REE can also be driven directly via the gensyn-sdk CLI inside the container, or via the ree.sh shell script included in the repository.
This is useful for scripting, CI pipelines, or when you need fine-grained control over the pipeline.
Global Flags
--verbose is a global flag and must appear before the subcommand:
gensyn-sdk --verbose run-all ...--verbose
Enable debug-level logging.
Location Flags
Every command requires exactly one of the following (they are mutually exclusive):
--tasks-root <path>
Root directory for tasks. The task directory is derived as <tasks-root>/<sanitized-model-name>. This is the recommended default.
--task-dir <path>
Use this exact directory for inputs and outputs. Use when you want to pin artifacts to a specific path.
Run (Subcommand)
Runs the full pipeline: [1] prepare, [2] generate, [3] receipt and [4] decode.
Required Flags:
--model-name / -m
Hugging Face model ID (e.g., Qwen/Qwen3-0.6B).
--prompt-text or --prompt-file
The prompt to run. Mutually exclusive; one is required.
Optional Flags:
--model-revision
main
Specific Hugging Face model revision.
--max-new-tokens
300
Maximum number of tokens to generate.
--cpu-only
false
Force CPU execution even if CUDA is available.
--torch-dtype
bfloat16 on Apple silicon
Torch dtype for the run. In v0.8.0, Apple silicon defaults to bfloat16 instead of float16, which keeps nearly every operation on the GPU.
--force-model-export
false
Re-export ONNX model even if one exists.
--disable-kv-cache
false
Disable KV cache.
--short-circuit-length
—
Generation index at which to inject the short-circuit token.
--short-circuit-token
—
Token ID to inject when short-circuiting.
--n-partitions
1
Number of pipeline partitions to split the model across. Enables running larger models that don't fit on a single GPU. See Pipeline Parallelism.
Validate (Subcommand)
Checks that a receipt is structurally valid by recomputing hashes and comparing them against the stored values. This does not re-run inference.
Required flags:
--receipt-path
Path to the receipt JSON file to validate.
Verify (Subcommand)
Re-runs the full inference pipeline described in a receipt and compares the output against the receipt's claimed results. This is the strongest form of verification, as it is what proves the result is reproducible on your hardware.
Required Flags:
--receipt-path
Path to the receipt JSON file to verify.
--tasks-root or --task-dir
Where to store re-execution artifacts.
Optional Flags:
--cpu-only
false
Force CPU execution during verification.
Sampling Flags
These flags control the sampling behavior during generation.
In the TUI, they are passed via Extra Args. On the CLI, they are passed directly.
--do-sample / --no-do-sample
Enabled
Enable/disable stochastic sampling.
--temperature
1.0
Sampling temperature. Higher = more random.
--top-k
50
Top-k sampling cutoff.
--top-p
1.0
Nucleus sampling threshold.
--min-p
Disabled
Min-p sampling threshold.
--repetition-penalty
1.0
Repetition penalty multiplier.
Prompt Format (JSONL)
For CLI usage, prompts can be provided via --prompt-file using JSONL format. Each line must be either:
A JSON string:
"What is 2 + 2?"A JSON object with a
promptfield:{"prompt": "Explain deterministic inference in one sentence."}
Here's an example prompts.jsonl file:
The Pipeline
Under the hood, REE's run command (whether triggered from the TUI or CLI) executes a four-stage pipeline: [1] prepare, [2] generate, [3] receipt, and [4] decode.
1. Prepare
Downloads the model from Hugging Face, exports it to ONNX through PyTorch's dynamo path with torch-native dynamic shapes, tokenizes the prompt, and writes a task configuration file. All artifacts are written to the task directory.
Artifacts produced:
model/model.onnx: The exported ONNX modelmodel/tensors.binary: Serialized model weightsconfig.json: Task configuration (sampling settings, token limits, etc.)prompt_tokens.parquet: Tokenized promptmetadata/prepare.json: Prepare-stage metadata (model name, commit hash, config hash)
If model/model.onnx already exists in the task directory, prepare skips re-export and reuses it. Use --force-model-export to override this. Re-export cached models when investigating export compatibility after an upgrade.
2. Generate
Loads the prepared ONNX model, compiles it through the Gensyn Compiler (applying RepOp kernels if --operation-set reproducible), and runs the inference loop.
Artifacts produced:
output_tokens.parquet: Generated token IDsmetadata/generate.json: Generate-stage metadata (finish reasons, device info, operation set, seed)compiled-artifacts-*: Compiler output directories
3. Receipt
Assembles a cryptographically hashed receipt from the prepare and generate metadata, config, and output tokens.
Artifacts produced:
metadata/receipt_<timestamp>.json: Hashed receipt for full replication and verification.
4. Decode
Reads output_tokens.parquet, decodes the token IDs back into text using the model's tokenizer, and prints the result.
SDK Session API
REE comes packaged with InferenceSession, a Python SDK abstraction for reusable inference.
InferenceSession loads a prepared task directory and reuses model/session state across completions. It is useful for applications that need multiple completions without repeating setup and model-load work each time.
Tool Call Definitions in SDK Sessions
InferenceSession.complete() accepts tool definitions when using chat-style messages:
Python SDK Sessions
Methods used here are complete(...) and reset_reproducibility_state().
This requires a prepared task_dir (from prepare / run-all / prepare_task.run).
Pipeline Parallelism
Models larger than a single GPU's memory can be split across multiple partitions using the --n-partitions flag. This unlocks models up to 72B parameters on suitable multi-GPU hosts while preserving bitwise-reproducible output.
--n-partitions is compatible with all three operation sets (default, deterministic, reproducible). Reproducibility guarantees are preserved across partition counts: the same model and prompt will produce bitwise-identical output whether run with --n-partitions 1 or --n-partitions 8.
In the TUI, use the dedicated Partitions field. On the CLI, pass --n-partitions <N>. An example CLI command with parallelism enabled would look like:
Persisting Data & Caching
The REE container mounts your host's ~/.cache directory into the container automatically. This persists both Hugging Face model downloads (~/.cache/huggingface) and SDK artifacts like ONNX exports, compiled models, and receipts (~/.cache/gensyn).
This means subsequent runs of the same model will skip the download and export steps automatically. No additional volume mounts or configuration are needed.
SDK Sessions
REE can be used through the CLI for one-shot run, validate, and verify workflows. The SDK also exposes InferenceSession for applications that need more control over the lifecycle of inference.
InferenceSession manages setup, execution, and teardown for a reusable inference workflow. Instead of starting from scratch for each generation, an application can keep a session alive and run multiple inference calls inside the same container instance.
This is particularly useful for application integrations, multi-turn workflows, and tool-call loops.
Tool Calls
REE v0.3.0 adds basic support for tool-call workflows through the SDK.
A tool-call workflow has four parts:
The application defines the tools that are available to the model.
The model requests a tool call during inference.
The application executes the requested tool.
The application passes the tool result back into the inference session.
REE does not execute arbitrary external tools by itself. The host application is responsible for defining tools, enforcing permissions, executing tool calls, and recording tool results.
Tool-Calling Models
REE can execute many Hugging Face models, but tool-call behavior depends on the selected model.
Models that have been instruction-tuned for tool or function calling are more likely to emit valid structured tool-call requests. REE SDK support enables tool-call workflows, but it does not make every supported model equally capable of using tools.
Thinking Mode (enable_thinking)
v0.4.0 adds enable_thinking: Optional[bool] = None to InferenceSession.complete().
InferenceSession.complete()
Yes
gensyn-sdk CLI (run-all, prepare, …)
No
TUI / ree.sh one-shot runs
No
When set to True or False, the value is forwarded to tokenizer.apply_chat_template(..., enable_thinking=...) if the tokenizer supports it. When None (default), the kwarg is omitted. This is recorded in receipts as input.parameters.enable_thinking when explicitly set.
Experimental Quantized Inference
v0.7.0 adds an experimental reproducible quantized inference path to the SDK: quantize a Qwen model and run reproducible inference on it. This path requires an H100 and isn't available from the TUI or ree.sh.
Reproducibility Considerations with Tools
REE can make model inference reproducible, but external tools are only reproducible if their inputs and outputs are captured and replayed.
For example, a calculator tool is naturally easy to replay, but a web-search tool is not, because live search results and page contents can change over time.
Applications that need replayable tool-augmented inference should record:
Tool name.
Tool schema.
Tool-call arguments.
Tool output returned to the model.
Timestamp.
Provider or service metadata.
Hashes of external data.
Any fetched source content used to produce the tool result.
On replay, the application should provide the recorded tool output instead of calling the live external system again.
Limitations
Tool-call support does not make every model equally capable of using tools. The selected model must be able to emit useful structured tool-call requests.
Applications must also capture provenance and replay external data, because REE verifies only the model computation.
EULA
Use of REE and its components (Gensyn SDK, Gensyn Compiler, RepOp kernels) is subject to the Gensyn End User License Agreement. Please review the EULA before use.
Last updated