Advanced Usage & CLI Reference

Skip the TUI and work directly with the SDK CLI, pipeline stages, and container configuration.

Advanced Usage, Configuations & CLI Commands

This section covers the internals of REE for power users, integrators, and anyone who wants to understand what happens under the hood or drive REE directly from the command line.

CLI Reference

While the TUI is the recommended interface, REE can also be driven directly via the gensyn-sdk CLI inside the container, or via the ree.sh shell script included in the repository.

This is useful for scripting, CI pipelines, or when you need fine-grained control over the pipeline.

Global Flags

--verbose is a global flag and must appear before the subcommand:

gensyn-sdk --verbose run ...

Flag	Description
--verbose	Enable debug-level logging.

Location Flags

Every command requires exactly one of the following (they are mutually exclusive):

Flag	Description
--tasks-root <path>	Root directory for tasks. The task directory is derived as <tasks-root>/<sanitized-model-name>. This is the recommended default.
--task-dir <path>	Use this exact directory for inputs and outputs. Use when you want to pin artifacts to a specific path.

When using --tasks-root, the SDK auto-creates a subdirectory named after the model using path-safe characters. For example, --tasks-root /tmp/tasks with model Qwen/Qwen3-0.6B creates /tmp/tasks/Qwen--Qwen3-0.6B/.

Run (Subcommand)

Runs the full pipeline: [1] prepare, [2] generate, [3] receipt and [4] decode.

gensyn-sdk run-all \\
  --tasks-root <path> \\
  --model-name <huggingface-model-id> \\
  --prompt-text "Your prompt here" \\
  --operation-set {default,deterministic,reproducible}

1. Required Flags:

Flag	Description
--model-name / -m	Hugging Face model ID (e.g., Qwen/Qwen3-0.6B).
--prompt-text or --prompt-file	The prompt to run. Mutually exclusive; one is required.

--operation-set is not a required flag. Instead, it defaults to reproducible mode. You would only use this flag if you wanted to switch to deterministic mode.

1. Optional Flags:

Flag	Default	Description
--model-revision	main	Specific Hugging Face model revision.
--max-new-tokens	300	Maximum number of tokens to generate.
--cpu-only	false	Force CPU execution even if CUDA is available.
--force-model-export	false	Re-export ONNX model even if one exists.
--disable-kv-cache	false	Disable KV cache.
--short-circuit-length	—	Generation index at which to inject the short-circuit token.
--short-circuit-token	—	Token ID to inject when short-circuiting.
--n-partitions	1	Number of pipeline partitions to split the model across. Enables running larger models that don't fit on a single GPU. See Pipeline Parallelism.

Validate (Subcommand)

Checks that a receipt is structurally valid by recomputing hashes and comparing them against the stored values. This does not re-run inference.

gensyn-sdk validate --receipt-path <path-to-receipt.json>

1. Required flags:

Flag	Description
--receipt-path	Path to the receipt JSON file to validate.

There aren't any location flags (--tasks-root / --task-dir) needed because validate only inspects the receipt file itself.

Verify (Subcommand)

Re-runs the full inference pipeline described in a receipt and compares the output against the receipt's claimed results. This is the strongest form of verification, as it is what proves the result is reproducible on your hardware.

gensyn-sdk verify \
  --receipt-path <path-to-receipt.json> \
  --tasks-root <path> \
  --cpu-only

1. Required Flags:

Flag	Description
--receipt-path	Path to the receipt JSON file to verify.
--tasks-root or --task-dir	Where to store re-execution artifacts.

1. Optional Flags:

Flag	Default	Description
--cpu-only	false	Force CPU execution during verification.

verify needs both a receipt path (what to verify) and a location (where to put the re-run workspace). When using the TUI, --tasks-root is passed automatically.

Sampling Flags

These flags control the sampling behavior during generation.

In the TUI, they are passed via Extra Args. On the CLI, they are passed directly.

Flag	Default	Description
--do-sample / --no-do-sample	Enabled	Enable/disable stochastic sampling.
--temperature	1.0	Sampling temperature. Higher = more random.
--top-k	50	Top-k sampling cutoff.
--top-p	1.0	Nucleus sampling threshold.
--min-p	Disabled	Min-p sampling threshold.
--repetition-penalty	1.0	Repetition penalty multiplier.

Prompt Format (JSONL)

For CLI usage, prompts can be provided via --prompt-file using JSONL format. Each line must be either:

• A JSON string: "What is 2 + 2?"

• A JSON object with a prompt field: {"prompt": "Explain deterministic inference in one sentence."}

Here's an example prompts.jsonl file:

"What is 2 + 2?"
{"prompt": "Explain deterministic inference in one sentence."}

The Pipeline

Under the hood, REE's run command (whether triggered from the TUI or CLI) executes a four-stage pipeline: [1] prepare, [2] generate, [3] receipt, and [4] decode.

1. Prepare

Downloads the model from Hugging Face, exports it to ONNX format, tokenizes the prompt, and writes a task configuration file. All artifacts are written to the task directory.

Artifacts produced:

• model/model.onnx: The exported ONNX model

• model/tensors.binary: Serialized model weights

• config.json: Task configuration (sampling settings, token limits, etc.)

• prompt_tokens.parquet: Tokenized prompt

• metadata/prepare.json: Prepare-stage metadata (model name, commit hash, config hash)

If model/model.onnx already exists in the task directory, prepare skips re-export and reuses it. Use --force-model-export to override this.

2. Generate

Loads the prepared ONNX model, compiles it through the Gensyn Compiler (applying RepOp kernels if --operation-set reproducible), and runs the inference loop.

Artifacts produced:

• output_tokens.parquet: Generated token IDs

• metadata/generate.json: Generate-stage metadata (finish reasons, device info, operation set, seed)

• compiled-artifacts-*: Compiler output directories

3. Receipt

Assembles a cryptographically hashed receipt from the prepare and generate metadata, config, and output tokens.

Artifacts produced:

• metadata/receipt_<timestamp>.json: Hashed receipt for full replication and verification.

4. Decode

Reads output_tokens.parquet, decodes the token IDs back into text using the model's tokenizer, and prints the result.

Python SDK Sessions

REE comes packaged with InferenceSession, a Python SDK abstraction for reusable inference.

from pathlib import Path
from gensyn_sdk import InferenceSession, OperationSet

session = InferenceSession(
    task_dir=Path("/path/to/prepared/task-dir"),
    operation_set=OperationSet.REPRODUCIBLE,
    cpu_only=False,
    random_seed=12345,
)

result = session.complete(
    prompt="Explain reproducible inference in one sentence.",
    max_new_tokens=128,
)

print(result.text)

InferenceSession loads a prepared task directory and reuses model/session state across completions. It is useful for applications that need multiple completions without repeating setup and model-load work each time.

Use reset_reproducibility_state() before a completion when that completion must be independently reproducible from its receipt.

Tool Call Definitions in SDK Sessions

InferenceSession.complete() accepts tool definitions when using chat-style messages:

result = session.complete(
    messages=[
        {"role": "user", "content": "What is 243 * 17?"}
    ],
    tools=tools,
    max_new_tokens=256,
)

Python SDK Sessions

Methods used here are complete(...) and reset_reproducibility_state().

This requies a prepared task_dir (from prepare / run-all / prepare_task.run).

InferenceSession(
    task_dir: Path,
    operation_set: OperationSet = OperationSet.REPRODUCIBLE,
    cpu_only: bool = False,
    random_seed: int = 12345,
)

Pipeline Parallelism

Models larger than a single GPU's memory can be split across multiple partitions using the --n-partitions flag. This unlocks models up to 72B parameters on suitable multi-GPU hosts while preserving bitwise-reproducible output.

--n-partitions requires multiple GPUs. On a single-GPU host, setting --n-partitions above 1 is expected to fail. You can leave it at its default, or omit the flag entirely.

The largest model you can run on a single GPU is bounded by that GPU's memory regardless of anything else. For a conceptual overview of how partitioning works, see Pipeline Parallelism.

--n-partitions is compatible with all three operation sets (default, deterministic, reproducible). Reproducibility guarantees are preserved across partition counts: the same model and prompt will produce bitwise-identical output whether run with --n-partitions 1 or --n-partitions 8.

In the TUI, pass --n-partitions <N> via the Extra Args field. An example of this flag sequence (with parallelism enabled) would look like:

​gensyn-sdk run-all \ --tasks-root /tmp/tasks \ --model-name Qwen/Qwen2.5-72B-Instruct \ --prompt-text "Explain pipeline parallelism." \ --n-partitions 4 ​

Persisting Data & Caching

The REE container mounts your host's ~/.cache directory into the container automatically. This persists both Hugging Face model downloads (~/.cache/huggingface) and SDK artifacts like ONNX exports, compiled models, and receipts (~/.cache/gensyn).

This means subsequent runs of the same model will skip the download and export steps automatically. No additional volume mounts or configuration are needed.

When using the TUI, this caching is handled for you. The details above apply if you're running the container directly via the CLI.

SDK Sessions

REE can be used through the CLI for one-shot run, validate, and verify workflows. The SDK also exposes InferenceSession for applications that need more control over the lifecycle of inference.

InferenceSession manages setup, execution, and teardown for a reusable inference workflow. Instead of starting from scratch for each generation, an application can keep a session alive and run multiple inference calls inside the same container instance.

This is particularly useful for application integrations, multi-turn workflows, and tool-call loops.

Tool Calls

REE v0.3.0 adds basic support for tool-call workflows through the SDK.

A tool-call workflow has four parts:

1. The application defines the tools that are available to the model.

2. The model requests a tool call during inference.

3. The application executes the requested tool.

4. The application passes the tool result back into the inference session.

REE does not execute arbitrary external tools by itself. The host application is responsible for defining tools, enforcing permissions, executing tool calls, and recording tool results.

Tool-Calling Models

REE can execute many Hugging Face models, but tool-call behavior depends on the selected model.

Models that have been instruction-tuned for tool or function calling are more likely to emit valid structured tool-call requests. REE SDK support enables tool-call workflows, but it does not make every supported model equally capable of using tools.

When building a tool-augmented application, choose a model that is known to follow tool schemas reliably and test that it emits valid tool calls for your use case.

Thinking Mode (enable_thinking)

v0.4.0 adds enable_thinking: Optional[bool] = None to InferenceSession.complete().

Layer	Supported?
InferenceSession.complete()	Yes
gensyn-sdk CLI (run-all, prepare, …)	No
TUI / ree.sh one-shot runs	No

When set to True or False, the value is forwarded to tokenizer.apply_chat_template(..., enable_thinking=...) if the tokenizer supports it. When None (default), the kwarg is omitted. This is recorded in receipts as input.parameters.enable_thinking when explicitly set.

Alternatively (for the CUI and TUI) use --short-circuit-length and --short-circuit-token on run-all / prepare.

Reproducibility Considerations with Tools

REE can make model inference reproducible, but external tools are only reproducible if their inputs and outputs are captured and replayed.

For example, a calculator tool is naturally easy to replay, but a web-search tool is not, because live search results and page contents can change over time.

Applications that need replayable tool-augmented inference should record:

• Tool name.

• Tool schema.

• Tool-call arguments.

• Tool output returned to the model.

• Timestamp.

• Provider or service metadata.

• Hashes of external data.

• Any fetched source content used to produce the tool result.

On replay, the application should provide the recorded tool output instead of calling the live external system again.

Limitations

Tool-call support does not make every model equally capable of using tools. The selected model must be able to emit useful structured tool-call requests.

Applications must also capture provenance and replay external data, because REE verifies only the model computation.

EULA

Use of REE and its components (Gensyn SDK, Gensyn Compiler, RepOp kernels) is subject to the Gensyn End User License Agreement. Please review the EULA before use.