Compare commits

...

22 Commits

Author SHA1 Message Date
Kit Langton c20d070b9a docs(llm): fix stale references in protocols/shared.ts and gemini.ts
- subtractTokens JSDoc said the raw payload lives on Usage.native, but
  that field was renamed to providerMetadata earlier in this PR.
- totalTokens JSDoc still described the abandoned "additive" first-pass
  contract where inputTokens/outputTokens were non-cached / visible only.
  We landed on inclusive totals; the fallback already covers cache and
  reasoning.
- Removed a duplicate inline comment in Gemini's mapUsage — the
  function-level comment already explains the visible/reasoning sum and
  the undefined-when-incomplete rule.
2026-05-10 22:08:12 -04:00
Kit Langton d048bd6f4b test(llm): re-record golden scenarios against live providers
Verifies the new Usage mapper code against live provider responses for
OpenAI Chat, OpenAI Responses, Anthropic, Gemini, DeepSeek, and
TogetherAI — 16 fresh recordings, all assertions pass. No existing
cassettes were modified; these populate test slots that were previously
skipped in replay mode.

Recorded via:
    set -a; source .env.recorded.local; set +a
    RECORD=true bun test test/provider/*.recorded.test.ts

Redactor stripped all auth headers; no secrets in the cassettes.
2026-05-10 22:03:02 -04:00
Kit Langton ab9b79ef88 refactor(llm): rename Usage.native to providerMetadata
Aligns the escape-hatch field name with `LLMEvent.providerMetadata` used
elsewhere in this package (and with AI SDK / pydantic-ai / LangChain
conventions for the same idea). Two parallel escape hatches having
different names was a wart.

The raw payload is now wrapped under the provider key — `{ openai: ... }`,
`{ anthropic: ... }`, `{ google: ... }`, `{ bedrock: ... }` — using the
existing `ProviderMetadata = Record<string, Record<string, unknown>>`
schema rather than a flat record. Same shape as
`LLMEvent.providerMetadata`, so consumers downstream can read both with
the same code.

Anthropic's `mergeUsage` merges the per-provider sub-record across
`message_start` and `message_delta` instead of spreading at the top level.
2026-05-10 21:42:09 -04:00
Kit Langton d4ff331052 refactor(llm): inclusive total + non-overlapping breakdown for Usage
Final shape after considering ecosystem conventions:

  inputTokens             — inclusive total (matches AI SDK / OpenAI / LangChain)
  outputTokens            — inclusive total (includes reasoning)
  nonCachedInputTokens    — breakdown: fresh prompt
  cacheReadInputTokens    — breakdown: cache hit
  cacheWriteInputTokens   — breakdown: cache write
  reasoningTokens         — subset of outputTokens

Invariant:
  nonCached + cacheRead + cacheWrite = inputTokens
  reasoningTokens <= outputTokens

Why this shape:

- `inputTokens` keeps its AI-SDK / OpenAI semantics, so a reader from any
  major ecosystem sees the number they expect.
- The non-overlapping breakdown fields are populated alongside the
  inclusive totals — consumers read whichever they need without
  subtracting. This eliminates the underflow bug class (opencode#26620)
  structurally without diverging on naming.
- Aligns with the AI SDK v3 spec proposal (vercel/ai#9921), which adds
  exactly this kind of non-overlapping breakdown to address the active
  ecosystem bugs around cache token double-counting and underflow
  (pydantic-ai#4364, langfuse#12306/#11979, vercel/ai#8349,
  langchain#32818, langchainjs#10249).

Mappers:

- OpenAI Chat / Responses / Bedrock: provider reports inclusive totals
  natively; mapper derives `nonCachedInputTokens` via
  `ProviderShared.subtractTokens`.
- Gemini: `promptTokenCount` is inclusive; `candidatesTokenCount` is
  *exclusive* of `thoughtsTokenCount`, so mapper sums those to produce
  the inclusive `outputTokens`. Only computes the total when the visible
  component is reported (avoids fabricating an inclusive number from a
  partial breakdown).
- Anthropic: `input_tokens` is *non-cached* natively; mapper sums it with
  cache reads/writes to produce the inclusive `inputTokens`.
  `output_tokens` is inclusive (Anthropic doesn't break thinking out, so
  `reasoningTokens` stays undefined).

Added a `visibleOutputTokens` getter (clamped `outputTokens - reasoningTokens`)
as the one safe escape hatch for consumers wanting the non-reasoning view.

Added `ProviderShared.sumTokens` to derive an inclusive total from a
non-overlapping breakdown, returning `undefined` when every input is
undefined (so we don't fabricate a 0).
2026-05-10 20:39:22 -04:00
Kit Langton f5d199db62 feat(llm): add Usage.totalInputTokens / totalOutputTokens getters
Match the `LLMResponse.text` / `reasoning` / `toolCalls` getter pattern
in the same file — `usage.totalInputTokens` reads naturally and lives
where the Usage data does. Both sums are monotonic under the additive
contract, so callers no longer need to remember which fields are
non-overlapping.

Test fixtures that previously asserted with `usage: { ... }` plain
literals are now wrapped with `new Usage({...})` to match the runtime
shape the mappers actually produce (an instance, not a struct).
2026-05-10 19:29:41 -04:00
Kit Langton 0d4f8d126f refactor(llm): drop Usage.totalInput / totalOutput helpers
The additive contract delivers value at the mapper boundary — every
field is non-overlapping and non-negative, so any caller summing
arbitrary subsets is correct by construction. Two-line helpers that
just sum three or two known fields add API surface without paying for
themselves, and there are no in-tree consumers today. If v2 wants them
at integration time, the right place is a getter on the `Schema.Class`
(matching the `LLMResponse.text` / `reasoning` / `toolCalls` pattern in
the same file), not a static namespace helper.
2026-05-10 19:15:46 -04:00
Kit Langton 478f3ae50c refactor(llm): trim Usage helpers + Bedrock subtraction
Review pass:
- Drop `Pick<>` type aliases on `Usage.totalInput` / `Usage.totalOutput`
  — the helpers can take `Usage` directly since every field is optional.
- Collapse Bedrock's nested `subtractTokens(subtractTokens(...))` into a
  single subtraction against the summed cache subtotals.
- Drop arithmetic-walkthrough comments in test fixtures (the raw
  fixture values are right next to the expected outputs).
- Generalize the comment on `mapUsage` in `openai-chat.ts` so the
  rationale outlives the PR reference.
2026-05-10 13:22:49 -04:00
Kit Langton b9451175a6 refactor(llm): make LLM.Usage a fully-additive contract
Defines a single invariant for `LLM.Usage`: every field is non-negative
and every meaningful aggregate is a *sum*, never a difference. Total
billable input = inputTokens + cacheReadInputTokens + cacheWriteInputTokens.
Total billable output = outputTokens + reasoningTokens. Adding two
non-negatives cannot underflow, so consumers can no longer reproduce the
underflow-then-clamp bug class fixed by #26620.

Each protocol mapper now enforces the contract at the provider boundary
via `ProviderShared.subtractTokens`, which clamps with `Math.max(0, …)`
for defense against provider bugs:

- OpenAI Chat / Responses: pull `cached_tokens` out of `prompt_tokens` /
  `input_tokens`; pull `reasoning_tokens` out of `completion_tokens` /
  `output_tokens`. The provider's `total_tokens` is preserved verbatim.
- Gemini: pull `cachedContentTokenCount` out of `promptTokenCount`.
  Gemini already split visible candidates from thoughts.
- Bedrock: pull `cacheReadInputTokens` and `cacheWriteInputTokens` out of
  `inputTokens`, matching AWS prompt-caching docs.
- Anthropic: already non-overlapping per the Messages API; pass through.

Adds `Usage.totalInput` / `Usage.totalOutput` helpers for callers that
want the merged view, and a regression test covering the clamp behavior.

The reasoning underflow fixed in #26620 was the most visible symptom of
a broader semantic inconsistency in this package: providers also disagreed
on whether `inputTokens` includes cache reads (Anthropic excluded;
OpenAI/Gemini/Bedrock included), which would silently double-subtract
the moment v2 wired LLM.Usage into Session.getUsage. Normalizing now,
pre-integration, closes both holes in one move.
2026-05-10 13:07:58 -04:00
Kit Langton 9c8da69196 Use Effect timeout in compaction test (#26728) 2026-05-10 12:45:54 -04:00
opencode-agent[bot] a78018697c chore: generate 2026-05-10 16:44:40 +00:00
Kit Langton e45b6ef1de refactor(http-recorder): use Schema.TaggedErrorClass for cassette errors (#26729) 2026-05-10 16:43:33 +00:00
opencode-agent[bot] b616543ac2 chore: generate 2026-05-10 16:30:55 +00:00
Kit Langton 2bd3d9a696 refactor(http-recorder): hide cassette format behind Cassette seam (#26725) 2026-05-10 12:29:55 -04:00
Kit Langton fa15dbc5ec Migrate compaction process tests (#26723) 2026-05-10 12:25:44 -04:00
opencode-agent[bot] 312e5c7a7c chore: generate 2026-05-10 16:22:29 +00:00
Kit Langton 049502fac6 fix(server): return diagnosable body for schema rejections (#26631) 2026-05-10 16:21:32 +00:00
opencode-agent[bot] cc2915be16 chore: generate 2026-05-10 16:20:16 +00:00
Kit Langton ce061bf661 Add explicit LLM stream lifecycle events (#26722) 2026-05-10 12:19:13 -04:00
Frank 3b8790e034 zen: fix usage css on mobile 2026-05-10 12:14:11 -04:00
Kit Langton a4f3cedcdf Start effect-style compaction tests 2026-05-10 16:12:00 +00:00
opencode-agent[bot] 1c9a2eb239 chore: generate 2026-05-10 16:06:18 +00:00
Kit Langton 4fb417d3b5 feat(http-recorder): default mode to "auto" (#26719) 2026-05-10 16:05:11 +00:00
59 changed files with 2004 additions and 607 deletions
@@ -67,6 +67,7 @@
display: inline-flex;
align-items: center;
justify-content: center;
flex-shrink: 0;
padding: 0;
background: transparent;
border: none;
@@ -79,6 +80,7 @@
}
svg {
flex-shrink: 0;
width: 16px;
height: 16px;
}
+28 -26
View File
@@ -16,8 +16,9 @@ import { HttpRecorder } from "@opencode-ai/http-recorder"
## Quickstart
Provide `cassetteLayer(name)` in place of (or layered over) your `HttpClient`.
The first run records to `test/fixtures/recordings/<name>.json`; subsequent
runs replay from it.
By default the layer records on first run and replays on subsequent runs —
no env-var ternary at the call site, and `CI=true` forces strict replay so
missing cassettes fail loudly in CI rather than silently re-recording.
```ts
import { Effect } from "effect"
@@ -30,28 +31,22 @@ const program = Effect.gen(function* () {
return yield* response.json
})
// Replay (default). Fails if the cassette is missing.
// Records if the cassette is missing, replays if it exists.
// In CI (CI=true) always replays — fails loudly on missing fixtures.
Effect.runPromise(program.pipe(Effect.provide(HttpRecorder.cassetteLayer("users/get-one"))))
// Record. Hits the upstream and writes the cassette.
// Force a refresh — always hits upstream and overwrites.
Effect.runPromise(program.pipe(Effect.provide(HttpRecorder.cassetteLayer("users/get-one", { mode: "record" }))))
```
Set the mode from the environment in your test setup:
```ts
HttpRecorder.cassetteLayer("users/get-one", {
mode: process.env.RECORD === "true" ? "record" : "replay",
})
```
## Modes
| Mode | Behavior |
| ------------- | -------------------------------------------------------------------- |
| `replay` | Default. Match the request to a recorded interaction; error if none. |
| `record` | Execute upstream, append the interaction, write the cassette. |
| `passthrough` | Bypass the recorder entirely — just call upstream. |
| Mode | Behavior |
| ------------- | ----------------------------------------------------------------------------------- |
| `auto` | Default. Replay if the cassette exists; record if missing. `CI=true` forces replay. |
| `replay` | Strict — match the request to a recorded interaction; error if none. |
| `record` | Execute upstream, append the interaction, write the cassette. |
| `passthrough` | Bypass the recorder entirely — just call upstream. |
## Cassette format
@@ -101,7 +96,6 @@ secrets escape. Redaction is configured by composing a `Redactor`:
import { HttpRecorder, Redactor } from "@opencode-ai/http-recorder"
HttpRecorder.cassetteLayer("anthropic/messages", {
mode: process.env.RECORD === "true" ? "record" : "replay",
redactor: Redactor.defaults({
requestHeaders: { allow: ["content-type", "anthropic-version"] },
url: { transform: (url) => url.replace(/\/accounts\/[^/]+/, "/accounts/{account}") },
@@ -157,7 +151,6 @@ const program = Effect.gen(function* () {
const cassette = yield* HttpRecorder.Cassette.Service
const executor = yield* HttpRecorder.makeWebSocketExecutor({
name: "ws/subscribe",
mode: process.env.RECORD === "true" ? "record" : "replay",
cassette,
live: liveExecutor,
})
@@ -167,9 +160,9 @@ const program = Effect.gen(function* () {
## Inspecting cassettes programmatically
`Cassette.Service` exposes `read`, `write`, `append`, `exists`, `list`, and
`scan` (re-running the secret detector over an existing cassette). Useful
for CI checks:
`Cassette.Service` exposes `read`, `append`, `exists`, and `list`. `read`
returns the recorded interactions for a name; the file format is hidden
behind the seam. Useful for CI checks:
```ts
import { HttpRecorder } from "@opencode-ai/http-recorder"
@@ -177,18 +170,27 @@ import { Effect } from "effect"
const audit = Effect.gen(function* () {
const cassettes = yield* HttpRecorder.Cassette.Service
const findings = yield* Effect.forEach(yield* cassettes.list(), (entry) =>
cassettes.read(entry.name).pipe(Effect.map((c) => ({ entry, findings: cassettes.scan(c) }))),
const entries = yield* cassettes.list()
const issues = yield* Effect.forEach(entries, (entry) =>
cassettes
.read(entry.name)
.pipe(Effect.map((interactions) => ({ name: entry.name, findings: HttpRecorder.secretFindings(interactions) }))),
)
return findings.filter((r) => r.findings.length > 0)
return issues.filter((i) => i.findings.length > 0)
})
```
`cassetteLayer` is the batteries-included entry point — it provides
`Cassette.fileSystem({ directory })` automatically. If you want to provide
your own `Cassette.Service` (e.g. an in-memory adapter for the recorder's
own unit tests), use `recordingLayer` and supply `Cassette.fileSystem` /
`Cassette.memory` yourself.
## Options reference
```ts
type RecordReplayOptions = {
mode?: "record" | "replay" | "passthrough" // default: "replay"
mode?: "auto" | "replay" | "record" | "passthrough" // default: "auto" (CI=true forces "replay")
directory?: string // default: <cwd>/test/fixtures/recordings
metadata?: Record<string, unknown> // merged into cassette.metadata
redactor?: Redactor // default: Redactor.defaults()
+120 -83
View File
@@ -1,62 +1,76 @@
import { Context, Effect, FileSystem, Layer, PlatformError } from "effect"
import { Context, Effect, FileSystem, Layer, Schema } from "effect"
import * as fs from "node:fs"
import * as path from "node:path"
import { cassetteSecretFindings, secretFindings, type SecretFinding } from "./redaction"
import type { Cassette, CassetteMetadata, Interaction } from "./schema"
import { cassetteFor, cassettePath, DEFAULT_RECORDINGS_DIR, formatCassette, parseCassette } from "./storage"
import { secretFindings, type SecretFinding } from "./redaction"
import { decodeCassette, encodeCassette, type Cassette, type CassetteMetadata, type Interaction } from "./schema"
export interface Entry {
readonly name: string
readonly path: string
const DEFAULT_RECORDINGS_DIR = path.resolve(process.cwd(), "test", "fixtures", "recordings")
export class CassetteNotFoundError extends Schema.TaggedErrorClass<CassetteNotFoundError>()("CassetteNotFoundError", {
cassetteName: Schema.String,
}) {
override get message() {
return `Cassette "${this.cassetteName}" not found`
}
}
export interface AppendResult {
readonly findings: ReadonlyArray<SecretFinding>
}
export interface Interface {
readonly path: (name: string) => string
readonly read: (name: string) => Effect.Effect<Cassette, PlatformError.PlatformError>
readonly write: (name: string, cassette: Cassette) => Effect.Effect<void, PlatformError.PlatformError>
readonly append: (
name: string,
interaction: Interaction,
metadata: CassetteMetadata | undefined,
) => Effect.Effect<
{
readonly cassette: Cassette
readonly findings: ReadonlyArray<SecretFinding>
},
PlatformError.PlatformError
>
readonly read: (name: string) => Effect.Effect<ReadonlyArray<Interaction>, CassetteNotFoundError>
readonly append: (name: string, interaction: Interaction, metadata?: CassetteMetadata) => Effect.Effect<AppendResult>
readonly exists: (name: string) => Effect.Effect<boolean>
readonly list: () => Effect.Effect<ReadonlyArray<Entry>, PlatformError.PlatformError>
readonly scan: (cassette: Cassette) => ReadonlyArray<SecretFinding>
readonly list: () => Effect.Effect<ReadonlyArray<string>>
}
export class Service extends Context.Service<Service, Interface>()("@opencode-ai/http-recorder/Cassette") {}
export const layer = (options: { readonly directory?: string } = {}) =>
export const hasCassetteSync = (name: string, options: { readonly directory?: string } = {}) =>
fs.existsSync(path.join(options.directory ?? DEFAULT_RECORDINGS_DIR, `${name}.json`))
const buildCassette = (
name: string,
interactions: ReadonlyArray<Interaction>,
metadata: CassetteMetadata | undefined,
): Cassette => ({
version: 1,
metadata: { name, recordedAt: new Date().toISOString(), ...(metadata ?? {}) },
interactions,
})
const formatCassette = (cassette: Cassette) => `${JSON.stringify(encodeCassette(cassette), null, 2)}\n`
const parseCassette = (raw: string) => decodeCassette(JSON.parse(raw))
export const fileSystem = (
options: { readonly directory?: string } = {},
): Layer.Layer<Service, never, FileSystem.FileSystem> =>
Layer.effect(
Service,
Effect.gen(function* () {
const fileSystem = yield* FileSystem.FileSystem
const fs = yield* FileSystem.FileSystem
const directory = options.directory ?? DEFAULT_RECORDINGS_DIR
const recorded = new Map<string, { interactions: Interaction[]; findings: SecretFinding[] }>()
const directoriesEnsured = new Set<string>()
const pathFor = (name: string) => cassettePath(name, directory)
const cassettePath = (name: string) => path.join(directory, `${name}.json`)
const ensureDirectory = Effect.fn("Cassette.ensureDirectory")(function* (name: string) {
const dir = path.dirname(pathFor(name))
if (directoriesEnsured.has(dir)) return
yield* fileSystem.makeDirectory(dir, { recursive: true })
directoriesEnsured.add(dir)
})
const walk = (directory: string): Effect.Effect<ReadonlyArray<string>, PlatformError.PlatformError> =>
const ensureDirectory = (name: string) =>
Effect.gen(function* () {
const entries = yield* fileSystem
.readDirectory(directory)
.pipe(Effect.catch(() => Effect.succeed([] as string[])))
const dir = path.dirname(cassettePath(name))
if (directoriesEnsured.has(dir)) return
yield* fs.makeDirectory(dir, { recursive: true }).pipe(Effect.orDie)
directoriesEnsured.add(dir)
})
const walk = (current: string): Effect.Effect<ReadonlyArray<string>> =>
Effect.gen(function* () {
const entries = yield* fs.readDirectory(current).pipe(Effect.catch(() => Effect.succeed([] as string[])))
const nested = yield* Effect.forEach(entries, (entry) => {
const full = path.join(directory, entry)
return fileSystem.stat(full).pipe(
const full = path.join(current, entry)
return fs.stat(full).pipe(
Effect.flatMap((stat) => (stat.type === "Directory" ? walk(full) : Effect.succeed([full]))),
Effect.catch(() => Effect.succeed([] as string[])),
)
@@ -64,50 +78,73 @@ export const layer = (options: { readonly directory?: string } = {}) =>
return nested.flat()
})
const read = Effect.fn("Cassette.read")(function* (name: string) {
return parseCassette(yield* fileSystem.readFileString(pathFor(name)))
return Service.of({
read: (name) =>
fs.readFileString(cassettePath(name)).pipe(
Effect.map((raw) => parseCassette(raw).interactions),
Effect.catch(() => Effect.fail(new CassetteNotFoundError({ cassetteName: name }))),
),
append: (name, interaction, metadata) =>
Effect.gen(function* () {
const entry = recorded.get(name) ?? { interactions: [], findings: [] }
if (!recorded.has(name)) recorded.set(name, entry)
entry.interactions.push(interaction)
entry.findings.push(...secretFindings(interaction))
const cassette = buildCassette(name, entry.interactions, metadata)
const findings = [...entry.findings, ...secretFindings(cassette.metadata ?? {})]
if (findings.length === 0) {
yield* ensureDirectory(name)
yield* fs.writeFileString(cassettePath(name), formatCassette(cassette)).pipe(Effect.orDie)
}
return { findings }
}),
exists: (name) =>
fs.access(cassettePath(name)).pipe(
Effect.as(true),
Effect.catch(() => Effect.succeed(false)),
),
list: () =>
walk(directory).pipe(
Effect.map((files) =>
files
.filter((file) => file.endsWith(".json"))
.map((file) =>
path
.relative(directory, file)
.replace(/\\/g, "/")
.replace(/\.json$/, ""),
)
.toSorted((a, b) => a.localeCompare(b)),
),
),
})
const write = Effect.fn("Cassette.write")(function* (name: string, cassette: Cassette) {
yield* ensureDirectory(name)
yield* fileSystem.writeFileString(pathFor(name), formatCassette(cassette))
})
const append = Effect.fn("Cassette.append")(function* (
name: string,
interaction: Interaction,
metadata: CassetteMetadata | undefined,
) {
const entry = recorded.get(name) ?? { interactions: [], findings: [] }
entry.interactions.push(interaction)
entry.findings.push(...secretFindings(interaction))
recorded.set(name, entry)
const cassette = cassetteFor(name, entry.interactions, metadata)
const findings = [...entry.findings, ...secretFindings(cassette.metadata ?? {})]
if (findings.length === 0) yield* write(name, cassette)
return { cassette, findings }
})
const exists = Effect.fn("Cassette.exists")(function* (name: string) {
return yield* fileSystem.access(pathFor(name)).pipe(
Effect.as(true),
Effect.catch(() => Effect.succeed(false)),
)
})
const list = Effect.fn("Cassette.list")(function* () {
return (yield* walk(directory))
.filter((file) => file.endsWith(".json"))
.map((file) => ({
name: path
.relative(directory, file)
.replace(/\\/g, "/")
.replace(/\.json$/, ""),
path: file,
}))
.toSorted((a, b) => a.name.localeCompare(b.name))
})
return Service.of({ path: pathFor, read, write, append, exists, list, scan: cassetteSecretFindings })
}),
)
export const memory = (initial: Record<string, ReadonlyArray<Interaction>> = {}): Layer.Layer<Service> =>
Layer.sync(Service, () => {
const stored = new Map<string, Interaction[]>(
Object.entries(initial).map(([name, interactions]) => [name, [...interactions]]),
)
const accumulatedFindings = new Map<string, SecretFinding[]>()
return Service.of({
read: (name) =>
stored.has(name)
? Effect.succeed(stored.get(name) ?? [])
: Effect.fail(new CassetteNotFoundError({ cassetteName: name })),
append: (name, interaction, metadata) =>
Effect.sync(() => {
const existing = stored.get(name)
if (existing) existing.push(interaction)
else stored.set(name, [interaction])
const findings = accumulatedFindings.get(name)
if (findings) findings.push(...secretFindings(interaction))
else accumulatedFindings.set(name, [...secretFindings(interaction)])
if (metadata) accumulatedFindings.get(name)!.push(...secretFindings({ name, ...metadata }))
return { findings: accumulatedFindings.get(name) ?? [] }
}),
exists: (name) => Effect.sync(() => stored.has(name)),
list: () => Effect.sync(() => Array.from(stored.keys()).toSorted()),
})
})
+8 -5
View File
@@ -12,12 +12,12 @@ import {
} from "effect/unstable/http"
import * as CassetteService from "./cassette"
import { defaultMatcher, selectMatch, selectSequential, type RequestMatcher } from "./matching"
import { appendOrFail, makeReplayState } from "./recorder"
import { appendOrFail, makeReplayState, resolveAutoMode } from "./recorder"
import { defaults, type Redactor } from "./redactor"
import { redactUrl } from "./redaction"
import { httpInteractions, type CassetteMetadata, type HttpInteraction, type ResponseSnapshot } from "./schema"
export type RecordReplayMode = "record" | "replay" | "passthrough"
export type RecordReplayMode = "auto" | "record" | "replay" | "passthrough"
export interface RecordReplayOptions {
readonly mode?: RecordReplayMode
@@ -69,7 +69,8 @@ export const recordingLayer = (
const cassetteService = yield* CassetteService.Service
const redactor = options.redactor ?? defaults()
const match = options.match ?? defaultMatcher
const mode = options.mode ?? "replay"
const requested = options.mode ?? "auto"
const mode = requested === "auto" ? yield* resolveAutoMode(cassetteService, name) : requested
const sequential = options.dispatch === "sequential"
const replay = yield* makeReplayState(cassetteService, name, httpInteractions)
@@ -114,7 +115,9 @@ export const recordingLayer = (
return Effect.gen(function* () {
const incoming = yield* snapshotRequest(request)
const interactions = yield* replay.load.pipe(
Effect.mapError(() => transportError(request, `Fixture "${name}" not found.`)),
Effect.mapError(() =>
transportError(request, `Fixture "${name}" not found. Run locally to record it (CI=true forces replay).`),
),
)
const result = sequential
? selectSequential(interactions, incoming, match, yield* replay.cursor)
@@ -135,7 +138,7 @@ export const recordingLayer = (
export const cassetteLayer = (name: string, options: RecordReplayOptions = {}): Layer.Layer<HttpClient.HttpClient> =>
recordingLayer(name, options).pipe(
Layer.provide(CassetteService.layer({ directory: options.directory })),
Layer.provide(CassetteService.fileSystem({ directory: options.directory })),
Layer.provide(FetchHttpClient.layer),
Layer.provide(NodeFileSystem.layer),
)
+2 -2
View File
@@ -7,9 +7,9 @@ export type {
WebSocketFrame,
WebSocketInteraction,
} from "./schema"
export { hasCassetteSync } from "./storage"
export { CassetteNotFoundError, hasCassetteSync } from "./cassette"
export { defaultMatcher, type RequestMatcher } from "./matching"
export { cassetteSecretFindings, redactHeaders, redactUrl, type SecretFinding } from "./redaction"
export { redactHeaders, redactUrl, secretFindings, type SecretFinding } from "./redaction"
export { UnsafeCassetteError } from "./recorder"
export { cassetteLayer, recordingLayer, type RecordReplayMode, type RecordReplayOptions } from "./effect"
export {
+35 -23
View File
@@ -1,37 +1,49 @@
import { Effect, PlatformError, Ref, Scope } from "effect"
import { Effect, Ref, Schema, Scope } from "effect"
import type * as CassetteService from "./cassette"
import type { SecretFinding } from "./redaction"
import type { Cassette, CassetteMetadata, Interaction } from "./schema"
import type { CassetteNotFoundError } from "./cassette"
import { SecretFindingSchema } from "./redaction"
import type { CassetteMetadata, Interaction } from "./schema"
export class UnsafeCassetteError extends Error {
readonly _tag = "UnsafeCassetteError"
constructor(
readonly cassetteName: string,
readonly findings: ReadonlyArray<SecretFinding>,
) {
super(
`Refusing to write cassette "${cassetteName}" because it contains possible secrets: ${findings
.map((finding) => `${finding.path} (${finding.reason})`)
.join(", ")}`,
)
export class UnsafeCassetteError extends Schema.TaggedErrorClass<UnsafeCassetteError>()("UnsafeCassetteError", {
cassetteName: Schema.String,
findings: Schema.Array(SecretFindingSchema),
}) {
override get message() {
return `Refusing to write cassette "${this.cassetteName}" because it contains possible secrets: ${this.findings
.map((finding) => `${finding.path} (${finding.reason})`)
.join(", ")}`
}
}
export type ResolvedMode = "record" | "replay" | "passthrough"
const isCI = () => {
const value = process.env.CI
return value !== undefined && value !== "" && value !== "false" && value !== "0"
}
export const resolveAutoMode = (cassette: CassetteService.Interface, name: string): Effect.Effect<ResolvedMode> =>
Effect.gen(function* () {
if (isCI()) return "replay"
return (yield* cassette.exists(name)) ? "replay" : "record"
})
export const appendOrFail = (
cassette: CassetteService.Interface,
name: string,
interaction: Interaction,
metadata: CassetteMetadata | undefined,
): Effect.Effect<Cassette, UnsafeCassetteError> =>
cassette.append(name, interaction, metadata).pipe(
Effect.orDie,
Effect.flatMap(({ cassette: result, findings }) =>
findings.length === 0 ? Effect.succeed(result) : Effect.fail(new UnsafeCassetteError(name, findings)),
),
)
): Effect.Effect<void, UnsafeCassetteError> =>
cassette
.append(name, interaction, metadata)
.pipe(
Effect.flatMap(({ findings }) =>
findings.length === 0 ? Effect.void : Effect.fail(new UnsafeCassetteError({ cassetteName: name, findings })),
),
)
export interface ReplayState<T> {
readonly load: Effect.Effect<ReadonlyArray<T>, PlatformError.PlatformError>
readonly load: Effect.Effect<ReadonlyArray<T>, CassetteNotFoundError>
readonly cursor: Effect.Effect<number>
readonly advance: Effect.Effect<void>
}
@@ -39,7 +51,7 @@ export interface ReplayState<T> {
export const makeReplayState = <T>(
cassette: CassetteService.Interface,
name: string,
project: (cassette: Cassette) => ReadonlyArray<T>,
project: (interactions: ReadonlyArray<Interaction>) => ReadonlyArray<T>,
): Effect.Effect<ReplayState<T>, never, Scope.Scope> =>
Effect.gen(function* () {
const load = yield* Effect.cached(cassette.read(name).pipe(Effect.map(project)))
+7 -8
View File
@@ -1,5 +1,3 @@
import type { Cassette } from "./schema"
export const REDACTED = "[REDACTED]"
const DEFAULT_REDACT_HEADERS = [
@@ -97,10 +95,13 @@ export const redactHeaders = (
)
}
export type SecretFinding = {
readonly path: string
readonly reason: string
}
import { Schema } from "effect"
export const SecretFindingSchema = Schema.Struct({
path: Schema.String,
reason: Schema.String,
})
export type SecretFinding = Schema.Schema.Type<typeof SecretFindingSchema>
export const secretFindings = (value: unknown): ReadonlyArray<SecretFinding> =>
stringEntries(value).flatMap((entry) => [
@@ -112,5 +113,3 @@ export const secretFindings = (value: unknown): ReadonlyArray<SecretFinding> =>
.filter((item) => entry.value.includes(item.value))
.map((item) => ({ path: entry.path, reason: `environment secret ${item.name}` })),
])
export const cassetteSecretFindings = (cassette: Cassette) => secretFindings(cassette)
+3 -2
View File
@@ -52,9 +52,10 @@ export const isHttpInteraction = InteractionSchema.guards.http
export const isWebSocketInteraction = InteractionSchema.guards.websocket
export const httpInteractions = (cassette: Cassette) => cassette.interactions.filter(isHttpInteraction)
export const httpInteractions = (interactions: ReadonlyArray<Interaction>) => interactions.filter(isHttpInteraction)
export const webSocketInteractions = (cassette: Cassette) => cassette.interactions.filter(isWebSocketInteraction)
export const webSocketInteractions = (interactions: ReadonlyArray<Interaction>) =>
interactions.filter(isWebSocketInteraction)
export const CassetteSchema = Schema.Struct({
version: Schema.Literal(1),
-28
View File
@@ -1,28 +0,0 @@
import { Option } from "effect"
import * as fs from "node:fs"
import * as path from "node:path"
import { encodeCassette, decodeCassette, type Cassette, type CassetteMetadata, type Interaction } from "./schema"
export const DEFAULT_RECORDINGS_DIR = path.resolve(process.cwd(), "test", "fixtures", "recordings")
export const cassettePath = (name: string, directory = DEFAULT_RECORDINGS_DIR) => path.join(directory, `${name}.json`)
export const cassetteFor = (
name: string,
interactions: ReadonlyArray<Interaction>,
metadata: CassetteMetadata | undefined,
): Cassette => ({
version: 1,
metadata: { name, recordedAt: new Date().toISOString(), ...(metadata ?? {}) },
interactions,
})
export const formatCassette = (cassette: Cassette) => `${JSON.stringify(encodeCassette(cassette), null, 2)}\n`
export const parseCassette = (raw: string) => decodeCassette(JSON.parse(raw))
export const hasCassetteSync = (name: string, options: { readonly directory?: string } = {}) => {
const file = cassettePath(name, options.directory)
if (!fs.existsSync(file)) return false
return Option.isSome(Option.liftThrowable(parseCassette)(fs.readFileSync(file, "utf8")))
}
+5 -3
View File
@@ -2,7 +2,8 @@ import { Effect, Option, Ref, Scope, Stream } from "effect"
import type { Headers } from "effect/unstable/http"
import * as CassetteService from "./cassette"
import { canonicalizeJson, decodeJson } from "./matching"
import { appendOrFail, makeReplayState } from "./recorder"
import { appendOrFail, makeReplayState, resolveAutoMode } from "./recorder"
import type { RecordReplayMode } from "./effect"
import { defaults, type Redactor } from "./redactor"
import { webSocketInteractions, type CassetteMetadata, type WebSocketFrame } from "./schema"
@@ -23,7 +24,7 @@ export interface WebSocketExecutor<E> {
export interface WebSocketRecordReplayOptions<E> {
readonly name: string
readonly mode?: "record" | "replay" | "passthrough"
readonly mode?: RecordReplayMode
readonly metadata?: CassetteMetadata
readonly cassette: CassetteService.Interface
readonly live: WebSocketExecutor<E>
@@ -71,7 +72,8 @@ export const makeWebSocketExecutor = <E>(
options: WebSocketRecordReplayOptions<E>,
): Effect.Effect<WebSocketExecutor<E>, never, Scope.Scope> =>
Effect.gen(function* () {
const mode = options.mode ?? "replay"
const requested = options.mode ?? "auto"
const mode = requested === "auto" ? yield* resolveAutoMode(options.cassette, options.name) : requested
const redactor = options.redactor ?? defaults()
const openSnapshot = (request: WebSocketRequest) => {
const redacted = redactor.request({
@@ -7,7 +7,15 @@ import * as os from "node:os"
import * as path from "node:path"
import { HttpRecorder } from "../src"
import { redactedErrorRequest } from "../src/effect"
import { cassetteFor, formatCassette, parseCassette } from "../src/storage"
import type { Interaction } from "../src/schema"
const seedCassetteDirectory = (directory: string, name: string, interactions: ReadonlyArray<Interaction>) =>
Effect.runPromise(
Effect.gen(function* () {
const cassette = yield* HttpRecorder.Cassette.Service
yield* Effect.forEach(interactions, (interaction) => cassette.append(name, interaction))
}).pipe(Effect.provide(HttpRecorder.Cassette.fileSystem({ directory })), Effect.provide(NodeFileSystem.layer)),
)
const post = (url: string, body: object) =>
Effect.gen(function* () {
@@ -34,7 +42,7 @@ const runRecorder = <A, E>(effect: Effect.Effect<A, E, HttpRecorder.Cassette.Ser
Effect.scoped(
effect.pipe(
Effect.provide(
HttpRecorder.Cassette.layer({ directory: fs.mkdtempSync(path.join(os.tmpdir(), "http-recorder-")) }),
HttpRecorder.Cassette.fileSystem({ directory: fs.mkdtempSync(path.join(os.tmpdir(), "http-recorder-")) }),
),
Effect.provide(NodeFileSystem.layer),
),
@@ -109,7 +117,7 @@ describe("http-recorder", () => {
test("detects secret-looking values without returning the secret", () => {
expect(
HttpRecorder.cassetteSecretFindings({
HttpRecorder.secretFindings({
version: 1,
interactions: [
{
@@ -137,7 +145,7 @@ describe("http-recorder", () => {
test("detects secret-looking values inside metadata", () => {
expect(
HttpRecorder.cassetteSecretFindings({
HttpRecorder.secretFindings({
version: 1,
metadata: { token: "sk-123456789012345678901234" },
interactions: [],
@@ -145,60 +153,42 @@ describe("http-recorder", () => {
).toEqual([{ path: "metadata.token", reason: "API key" }])
})
test("formats websocket cassettes with shared metadata", () => {
const cassette = cassetteFor(
"websocket/basic",
[
{
transport: "websocket",
open: { url: "wss://example.test/realtime", headers: { "content-type": "application/json" } },
client: [{ kind: "text", body: JSON.stringify({ type: "response.create" }) }],
server: [{ kind: "text", body: JSON.stringify({ type: "response.completed" }) }],
},
],
{ provider: "openai" },
)
test("replays websocket interactions seeded into the in-memory cassette adapter", async () => {
await Effect.runPromise(
Effect.scoped(
Effect.gen(function* () {
const cassette = yield* HttpRecorder.Cassette.Service
const executor = yield* HttpRecorder.makeWebSocketExecutor({
name: "websocket/replay",
cassette,
compareClientMessagesAsJson: true,
live: { open: () => Effect.die(new Error("unexpected live WebSocket open")) },
})
const connection = yield* executor.open({
url: "wss://example.test/realtime",
headers: Headers.fromInput({ "content-type": "application/json" }),
})
yield* connection.sendText(JSON.stringify({ type: "response.create" }))
const messages: Array<string | Uint8Array> = []
yield* connection.messages.pipe(Stream.runForEach((message) => Effect.sync(() => messages.push(message))))
yield* connection.close
expect(cassette.metadata).toMatchObject({ name: "websocket/basic", provider: "openai" })
expect(parseCassette(formatCassette(cassette))).toEqual(cassette)
})
test("replays websocket interactions from the shared cassette service", async () => {
await runRecorder(
Effect.gen(function* () {
const cassette = yield* HttpRecorder.Cassette.Service
yield* cassette.write(
"websocket/replay",
cassetteFor(
"websocket/replay",
[
{
transport: "websocket",
open: { url: "wss://example.test/realtime", headers: { "content-type": "application/json" } },
client: [{ kind: "text", body: JSON.stringify({ type: "response.create" }) }],
server: [{ kind: "text", body: JSON.stringify({ type: "response.completed" }) }],
},
],
undefined,
expect(messages).toEqual([JSON.stringify({ type: "response.completed" })])
}).pipe(
Effect.provide(
HttpRecorder.Cassette.memory({
"websocket/replay": [
{
transport: "websocket",
open: { url: "wss://example.test/realtime", headers: { "content-type": "application/json" } },
client: [{ kind: "text", body: JSON.stringify({ type: "response.create" }) }],
server: [{ kind: "text", body: JSON.stringify({ type: "response.completed" }) }],
},
],
}),
),
)
const executor = yield* HttpRecorder.makeWebSocketExecutor({
name: "websocket/replay",
cassette,
compareClientMessagesAsJson: true,
live: { open: () => Effect.die(new Error("unexpected live WebSocket open")) },
})
const connection = yield* executor.open({
url: "wss://example.test/realtime",
headers: Headers.fromInput({ "content-type": "application/json" }),
})
yield* connection.sendText(JSON.stringify({ type: "response.create" }))
const messages: Array<string | Uint8Array> = []
yield* connection.messages.pipe(Stream.runForEach((message) => Effect.sync(() => messages.push(message))))
yield* connection.close
expect(messages).toEqual([JSON.stringify({ type: "response.completed" })])
}),
),
),
)
})
@@ -228,17 +218,14 @@ describe("http-recorder", () => {
yield* connection.messages.pipe(Stream.runDrain)
yield* connection.close
expect(yield* cassette.read("websocket/record")).toMatchObject({
metadata: { name: "websocket/record", provider: "test" },
interactions: [
{
transport: "websocket",
open: { url: "wss://example.test/realtime", headers: { "content-type": "application/json" } },
client: [{ kind: "text", body: JSON.stringify({ type: "response.create" }) }],
server: [{ kind: "text", body: JSON.stringify({ type: "response.completed" }) }],
},
],
})
expect(yield* cassette.read("websocket/record")).toMatchObject([
{
transport: "websocket",
open: { url: "wss://example.test/realtime", headers: { "content-type": "application/json" } },
client: [{ kind: "text", body: JSON.stringify({ type: "response.create" }) }],
server: [{ kind: "text", body: JSON.stringify({ type: "response.completed" }) }],
},
])
}),
)
})
@@ -301,6 +288,49 @@ describe("http-recorder", () => {
)
})
test("auto mode replays when the cassette exists", async () => {
const directory = fs.mkdtempSync(path.join(os.tmpdir(), "http-recorder-auto-"))
await seedCassetteDirectory(directory, "auto-replay", [
{
transport: "http",
request: {
method: "POST",
url: "https://example.test/echo",
headers: { "content-type": "application/json" },
body: JSON.stringify({ step: 1 }),
},
response: { status: 200, headers: { "content-type": "application/json" }, body: '{"reply":"hi"}' },
},
])
const result = await runWith(
"auto-replay",
{ directory, mode: "auto" },
post("https://example.test/echo", { step: 1 }),
)
expect(result).toBe('{"reply":"hi"}')
})
test("auto mode forces replay when CI=true even if cassette is missing", async () => {
const directory = fs.mkdtempSync(path.join(os.tmpdir(), "http-recorder-auto-ci-"))
const previous = process.env.CI
process.env.CI = "true"
try {
const exit = await Effect.runPromise(
Effect.exit(
post("https://example.test/echo", { step: 1 }).pipe(
Effect.provide(HttpRecorder.cassetteLayer("missing-cassette", { directory, mode: "auto" })),
),
),
)
expect(Exit.isFailure(exit)).toBe(true)
expect(failureText(exit)).toContain('Fixture "missing-cassette" not found')
} finally {
if (previous === undefined) delete process.env.CI
else process.env.CI = previous
}
})
test("mismatch diagnostics show closest redacted request differences", async () => {
await run(
Effect.gen(function* () {
+1 -1
View File
@@ -184,7 +184,7 @@ const FakeProtocol = Protocol.make<FakeBody, string, string, void>({
stream: {
event: Schema.String,
initial: () => undefined,
step: (_, frame) => Effect.succeed([undefined, [{ type: "text-delta", text: frame }]] as const),
step: (_, frame) => Effect.succeed([undefined, [{ type: "text-delta", id: "text-0", text: frame }]] as const),
onHalt: () => [{ type: "request-finish", reason: "stop" }],
},
})
@@ -5,10 +5,10 @@ import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type CacheHint,
type FinishReason,
type LLMEvent,
type LLMRequest,
type ProviderMetadata,
type ToolCallPart,
@@ -364,34 +364,56 @@ const mapFinishReason = (reason: string | null | undefined): FinishReason => {
return "unknown"
}
// Anthropic reports the non-overlapping breakdown natively — its
// `input_tokens` is the *non-cached* count per the Messages API docs, with
// cache reads and writes as separate fields. We sum them to derive the
// inclusive `inputTokens` the rest of the contract expects. Extended
// thinking tokens are *not* broken out by Anthropic — they're billed as
// part of `output_tokens`, so `reasoningTokens` stays `undefined` and
// `outputTokens` carries the combined total.
const mapUsage = (usage: AnthropicUsage | undefined): Usage | undefined => {
if (!usage) return undefined
const nonCached = usage.input_tokens
const cacheRead = usage.cache_read_input_tokens ?? undefined
const cacheWrite = usage.cache_creation_input_tokens ?? undefined
const inputTokens = ProviderShared.sumTokens(nonCached, cacheRead, cacheWrite)
return new Usage({
inputTokens: usage.input_tokens,
inputTokens,
outputTokens: usage.output_tokens,
cacheReadInputTokens: usage.cache_read_input_tokens ?? undefined,
cacheWriteInputTokens: usage.cache_creation_input_tokens ?? undefined,
totalTokens: ProviderShared.totalTokens(usage.input_tokens, usage.output_tokens, undefined),
native: usage,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cacheRead,
cacheWriteInputTokens: cacheWrite,
totalTokens: ProviderShared.totalTokens(inputTokens, usage.output_tokens, undefined),
providerMetadata: { anthropic: usage },
})
}
// Anthropic emits usage on `message_start` and again on `message_delta` — the
// final delta carries the authoritative totals. Right-biased merge: each
// field prefers `right` when defined, falls back to `left`. `totalTokens` is
// recomputed from the merged input/output to stay consistent.
// field prefers `right` when defined, falls back to `left`. `inputTokens` is
// recomputed from the merged breakdown so the inclusive total stays
// consistent with `nonCached + cacheRead + cacheWrite`.
const mergeUsage = (left: Usage | undefined, right: Usage | undefined) => {
if (!left) return right
if (!right) return left
const inputTokens = right.inputTokens ?? left.inputTokens
const nonCachedInputTokens = right.nonCachedInputTokens ?? left.nonCachedInputTokens
const cacheReadInputTokens = right.cacheReadInputTokens ?? left.cacheReadInputTokens
const cacheWriteInputTokens = right.cacheWriteInputTokens ?? left.cacheWriteInputTokens
const inputTokens = ProviderShared.sumTokens(nonCachedInputTokens, cacheReadInputTokens, cacheWriteInputTokens)
const outputTokens = right.outputTokens ?? left.outputTokens
return new Usage({
inputTokens,
outputTokens,
cacheReadInputTokens: right.cacheReadInputTokens ?? left.cacheReadInputTokens,
cacheWriteInputTokens: right.cacheWriteInputTokens ?? left.cacheWriteInputTokens,
nonCachedInputTokens,
cacheReadInputTokens,
cacheWriteInputTokens,
totalTokens: ProviderShared.totalTokens(inputTokens, outputTokens, undefined),
native: { ...left.native, ...right.native },
providerMetadata: {
anthropic: {
...(left.providerMetadata?.["anthropic"] ?? {}),
...(right.providerMetadata?.["anthropic"] ?? {}),
},
},
})
}
@@ -415,14 +437,13 @@ const serverToolResultEvent = (block: NonNullable<AnthropicEvent["content_block"
? String((block.content as Record<string, unknown>).type)
: ""
const isError = errorPayload.endsWith("_tool_result_error")
return {
type: "tool-result",
return LLMEvent.toolResult({
id: block.tool_use_id ?? "",
name: SERVER_TOOL_RESULT_NAMES[block.type],
result: isError ? { type: "error", value: block.content } : { type: "json", value: block.content },
providerExecuted: true,
providerMetadata: anthropicMetadata({ blockType: block.type }),
}
})
}
type StepResult = readonly [ParserState, ReadonlyArray<LLMEvent>]
@@ -453,18 +474,17 @@ const onContentBlockStart = (state: ParserState, event: AnthropicEvent): StepRes
}
if (block.type === "text" && block.text) {
return [state, [{ type: "text-delta", text: block.text }]]
return [state, [LLMEvent.textDelta({ id: `text-${event.index ?? 0}`, text: block.text })]]
}
if (block.type === "thinking" && block.thinking) {
return [
state,
[
{
type: "reasoning-delta",
LLMEvent.reasoningDelta({
id: `reasoning-${event.index ?? 0}`,
text: block.thinking,
...(block.signature ? { providerMetadata: anthropicMetadata({ signature: block.signature }) } : {}),
},
}),
],
]
}
@@ -480,17 +500,25 @@ const onContentBlockDelta = Effect.fn("AnthropicMessages.onContentBlockDelta")(f
const delta = event.delta
if (delta?.type === "text_delta" && delta.text) {
return [state, [{ type: "text-delta", text: delta.text }]] satisfies StepResult
return [state, [LLMEvent.textDelta({ id: `text-${event.index ?? 0}`, text: delta.text })]] satisfies StepResult
}
if (delta?.type === "thinking_delta" && delta.thinking) {
return [state, [{ type: "reasoning-delta", text: delta.thinking }]] satisfies StepResult
return [
state,
[LLMEvent.reasoningDelta({ id: `reasoning-${event.index ?? 0}`, text: delta.thinking })],
] satisfies StepResult
}
if (delta?.type === "signature_delta" && delta.signature) {
return [
state,
[{ type: "reasoning-delta", text: "", providerMetadata: anthropicMetadata({ signature: delta.signature }) }],
[
LLMEvent.reasoningEnd({
id: `reasoning-${event.index ?? 0}`,
providerMetadata: anthropicMetadata({ signature: delta.signature }),
}),
],
] satisfies StepResult
}
@@ -524,21 +552,20 @@ const onMessageDelta = (state: ParserState, event: AnthropicEvent): StepResult =
return [
{ ...state, usage },
[
{
type: "request-finish",
LLMEvent.requestFinish({
reason: mapFinishReason(event.delta?.stop_reason),
usage,
...(event.delta?.stop_sequence
? { providerMetadata: anthropicMetadata({ stopSequence: event.delta.stop_sequence }) }
: {}),
},
providerMetadata: event.delta?.stop_sequence
? anthropicMetadata({ stopSequence: event.delta.stop_sequence })
: undefined,
}),
],
]
}
const onError = (state: ParserState, event: AnthropicEvent): StepResult => [
state,
[{ type: "provider-error", message: event.error?.message ?? "Anthropic Messages stream error" }],
[LLMEvent.providerError({ message: event.error?.message ?? "Anthropic Messages stream error" })],
]
const step = (state: ParserState, event: AnthropicEvent) => {
+28 -11
View File
@@ -3,10 +3,10 @@ import { Route, type RouteModelInput } from "../route/client"
import { Endpoint } from "../route/endpoint"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type CacheHint,
type FinishReason,
type LLMEvent,
type LLMRequest,
type ToolCallPart,
type ToolDefinition,
@@ -363,15 +363,22 @@ const mapFinishReason = (reason: string): FinishReason => {
return "unknown"
}
// AWS Bedrock Converse reports `inputTokens` (inclusive total) with
// `cacheReadInputTokens` and `cacheWriteInputTokens` as subsets. Pass
// the total through and derive the non-cached breakdown. Bedrock does
// not break reasoning out of `outputTokens` for any current model.
const mapUsage = (usage: BedrockUsageSchema | undefined): Usage | undefined => {
if (!usage) return undefined
const cacheTotal = (usage.cacheReadInputTokens ?? 0) + (usage.cacheWriteInputTokens ?? 0)
const nonCached = ProviderShared.subtractTokens(usage.inputTokens, cacheTotal)
return new Usage({
inputTokens: usage.inputTokens,
outputTokens: usage.outputTokens,
totalTokens: ProviderShared.totalTokens(usage.inputTokens, usage.outputTokens, usage.totalTokens),
nonCachedInputTokens: nonCached,
cacheReadInputTokens: usage.cacheReadInputTokens,
cacheWriteInputTokens: usage.cacheWriteInputTokens,
native: usage,
totalTokens: ProviderShared.totalTokens(usage.inputTokens, usage.outputTokens, usage.totalTokens),
providerMetadata: { bedrock: usage },
})
}
@@ -400,13 +407,26 @@ const step = (state: ParserState, event: BedrockEvent) =>
}
if (event.contentBlockDelta?.delta?.text) {
return [state, [{ type: "text-delta" as const, text: event.contentBlockDelta.delta.text }]] as const
return [
state,
[
LLMEvent.textDelta({
id: `text-${event.contentBlockDelta.contentBlockIndex}`,
text: event.contentBlockDelta.delta.text,
}),
],
] as const
}
if (event.contentBlockDelta?.delta?.reasoningContent?.text) {
return [
state,
[{ type: "reasoning-delta" as const, text: event.contentBlockDelta.delta.reasoningContent.text }],
[
LLMEvent.reasoningDelta({
id: `reasoning-${event.contentBlockDelta.contentBlockIndex}`,
text: event.contentBlockDelta.delta.reasoningContent.text,
}),
],
] as const
}
@@ -449,16 +469,13 @@ const step = (state: ParserState, event: BedrockEvent) =>
event.modelStreamErrorException?.message ??
event.serviceUnavailableException?.message ??
"Bedrock Converse stream error"
return [state, [{ type: "provider-error" as const, message, retryable: true }]] as const
return [state, [LLMEvent.providerError({ message, retryable: true })]] as const
}
if (event.validationException || event.throttlingException) {
const message =
event.validationException?.message ?? event.throttlingException?.message ?? "Bedrock Converse error"
return [
state,
[{ type: "provider-error" as const, message, retryable: event.throttlingException !== undefined }],
] as const
return [state, [LLMEvent.providerError({ message, retryable: event.throttlingException !== undefined })]] as const
}
return [state, []] as const
@@ -468,7 +485,7 @@ const framing = BedrockEventStream.framing(ADAPTER)
const onHalt = (state: ParserState): ReadonlyArray<LLMEvent> =>
state.pendingFinish
? [{ type: "request-finish", reason: state.pendingFinish.reason, usage: state.pendingFinish.usage }]
? [LLMEvent.requestFinish({ reason: state.pendingFinish.reason, usage: state.pendingFinish.usage })]
: []
// =============================================================================
+25 -8
View File
@@ -5,9 +5,9 @@ import { Endpoint } from "../route/endpoint"
import { Framing } from "../route/framing"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type FinishReason,
type LLMEvent,
type LLMRequest,
type MediaPart,
type TextPart,
@@ -281,15 +281,28 @@ const fromRequest = Effect.fn("Gemini.fromRequest")(function* (request: LLMReque
// =============================================================================
// Stream Parsing
// =============================================================================
// Gemini reports `promptTokenCount` (inclusive total) with a
// `cachedContentTokenCount` subset. `candidatesTokenCount` is *exclusive*
// of `thoughtsTokenCount` — visible-only, not a total — so we sum the two
// to produce the inclusive `outputTokens` the rest of the contract expects.
// Output is left undefined when the visible component is missing, so we
// don't fabricate an inclusive number from a partial breakdown.
const mapUsage = (usage: GeminiUsage | undefined) => {
if (!usage) return undefined
const cached = usage.cachedContentTokenCount
const nonCached = ProviderShared.subtractTokens(usage.promptTokenCount, cached)
const outputTokens =
usage.candidatesTokenCount !== undefined
? usage.candidatesTokenCount + (usage.thoughtsTokenCount ?? 0)
: undefined
return new Usage({
inputTokens: usage.promptTokenCount,
outputTokens: usage.candidatesTokenCount,
outputTokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cached,
reasoningTokens: usage.thoughtsTokenCount,
cacheReadInputTokens: usage.cachedContentTokenCount,
totalTokens: ProviderShared.totalTokens(usage.promptTokenCount, usage.candidatesTokenCount, usage.totalTokenCount),
native: usage,
totalTokens: ProviderShared.totalTokens(usage.promptTokenCount, outputTokens, usage.totalTokenCount),
providerMetadata: { google: usage },
})
}
@@ -311,7 +324,7 @@ const mapFinishReason = (finishReason: string | undefined, hasToolCalls: boolean
const finish = (state: ParserState): ReadonlyArray<LLMEvent> =>
state.finishReason || state.usage
? [{ type: "request-finish", reason: mapFinishReason(state.finishReason, state.hasToolCalls), usage: state.usage }]
? [LLMEvent.requestFinish({ reason: mapFinishReason(state.finishReason, state.hasToolCalls), usage: state.usage })]
: []
const step = (state: ParserState, event: GeminiEvent) => {
@@ -332,14 +345,18 @@ const step = (state: ParserState, event: GeminiEvent) => {
for (const part of candidate.content.parts) {
if ("text" in part && part.text.length > 0) {
events.push({ type: part.thought ? "reasoning-delta" : "text-delta", text: part.text })
events.push(
part.thought
? LLMEvent.reasoningDelta({ id: "reasoning-0", text: part.text })
: LLMEvent.textDelta({ id: "text-0", text: part.text }),
)
continue
}
if ("functionCall" in part) {
const input = part.functionCall.args
const id = `tool_${nextToolCallId++}`
events.push({ type: "tool-call", id, name: part.functionCall.name, input })
events.push(LLMEvent.toolCall({ id, name: part.functionCall.name, input }))
hasToolCalls = true
}
}
+15 -9
View File
@@ -6,9 +6,9 @@ import { Framing } from "../route/framing"
import { HttpTransport } from "../route/transport"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type FinishReason,
type LLMEvent,
type LLMRequest,
type TextPart,
type ToolCallPart,
@@ -290,15 +290,24 @@ const mapFinishReason = (reason: string | null | undefined): FinishReason => {
return "unknown"
}
// OpenAI Chat reports `prompt_tokens` (inclusive total) with a
// `cached_tokens` subset, and `completion_tokens` (inclusive total) with
// a `reasoning_tokens` subset. We pass the inclusive totals through and
// derive the non-cached breakdown so the `LLM.Usage` contract is
// satisfied on both sides.
const mapUsage = (usage: OpenAIChatEvent["usage"]): Usage | undefined => {
if (!usage) return undefined
const cached = usage.prompt_tokens_details?.cached_tokens
const reasoning = usage.completion_tokens_details?.reasoning_tokens
const nonCached = ProviderShared.subtractTokens(usage.prompt_tokens, cached)
return new Usage({
inputTokens: usage.prompt_tokens,
outputTokens: usage.completion_tokens,
reasoningTokens: usage.completion_tokens_details?.reasoning_tokens,
cacheReadInputTokens: usage.prompt_tokens_details?.cached_tokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cached,
reasoningTokens: reasoning,
totalTokens: ProviderShared.totalTokens(usage.prompt_tokens, usage.completion_tokens, usage.total_tokens),
native: usage,
providerMetadata: { openai: usage },
})
}
@@ -312,7 +321,7 @@ const step = (state: ParserState, event: OpenAIChatEvent) =>
const toolDeltas = delta?.tool_calls ?? []
let tools = state.tools
if (delta?.content) events.push({ type: "text-delta", text: delta.content })
if (delta?.content) events.push(LLMEvent.textDelta({ id: "text-0", text: delta.content }))
for (const tool of toolDeltas) {
const result = ToolStream.appendOrStart(
@@ -348,10 +357,7 @@ const step = (state: ParserState, event: OpenAIChatEvent) =>
const finishEvents = (state: ParserState): ReadonlyArray<LLMEvent> => {
const hasToolCalls = state.toolCallEvents.length > 0
const reason = state.finishReason === "stop" && hasToolCalls ? "tool-calls" : state.finishReason
return [
...state.toolCallEvents,
...(reason ? ([{ type: "request-finish", reason, usage: state.usage }] satisfies ReadonlyArray<LLMEvent>) : []),
]
return [...state.toolCallEvents, ...(reason ? [LLMEvent.requestFinish({ reason, usage: state.usage })] : [])]
}
// =============================================================================
+26 -32
View File
@@ -6,9 +6,9 @@ import { Framing } from "../route/framing"
import { HttpTransport, WebSocketTransport } from "../route/transport"
import { Protocol } from "../route/protocol"
import {
LLMEvent,
Usage,
type FinishReason,
type LLMEvent,
type LLMRequest,
type ProviderMetadata,
type TextPart,
@@ -276,15 +276,23 @@ const fromRequest = Effect.fn("OpenAIResponses.fromRequest")(function* (request:
// =============================================================================
// Stream Parsing
// =============================================================================
// OpenAI Responses reports `input_tokens` (inclusive total) with a
// `cached_tokens` subset, and `output_tokens` (inclusive total) with a
// `reasoning_tokens` subset. Pass the totals through and derive the
// non-cached breakdown.
const mapUsage = (usage: OpenAIResponsesUsage | null | undefined) => {
if (!usage) return undefined
const cached = usage.input_tokens_details?.cached_tokens
const reasoning = usage.output_tokens_details?.reasoning_tokens
const nonCached = ProviderShared.subtractTokens(usage.input_tokens, cached)
return new Usage({
inputTokens: usage.input_tokens,
outputTokens: usage.output_tokens,
reasoningTokens: usage.output_tokens_details?.reasoning_tokens,
cacheReadInputTokens: usage.input_tokens_details?.cached_tokens,
nonCachedInputTokens: nonCached,
cacheReadInputTokens: cached,
reasoningTokens: reasoning,
totalTokens: ProviderShared.totalTokens(usage.input_tokens, usage.output_tokens, usage.total_tokens),
native: usage,
providerMetadata: { openai: usage },
})
}
@@ -348,22 +356,20 @@ const hostedToolEvents = (
const tool = HOSTED_TOOLS[item.type]
const providerMetadata = openaiMetadata({ itemId: item.id })
return [
{
type: "tool-call",
LLMEvent.toolCall({
id: item.id,
name: tool.name,
input: tool.input(item),
providerExecuted: true,
providerMetadata,
},
{
type: "tool-result",
}),
LLMEvent.toolResult({
id: item.id,
name: tool.name,
result: hostedToolResult(item),
providerExecuted: true,
providerMetadata,
},
}),
]
}
@@ -379,17 +385,7 @@ const TERMINAL_TYPES = new Set(["response.completed", "response.incomplete", "re
const onOutputTextDelta = (state: ParserState, event: OpenAIResponsesEvent): StepResult => {
if (!event.delta) return [state, NO_EVENTS]
return [
state,
[
{
type: "text-delta",
id: event.item_id,
text: event.delta,
...(event.item_id ? { providerMetadata: openaiMetadata({ itemId: event.item_id }) } : {}),
},
],
]
return [state, [LLMEvent.textDelta({ id: event.item_id ?? "text-0", text: event.delta })]]
}
const onOutputItemAdded = (state: ParserState, event: OpenAIResponsesEvent): StepResult => {
@@ -458,30 +454,28 @@ const onOutputItemDone = Effect.fn("OpenAIResponses.onOutputItemDone")(function*
const onResponseFinish = (state: ParserState, event: OpenAIResponsesEvent): StepResult => [
state,
[
{
type: "request-finish",
LLMEvent.requestFinish({
reason: mapFinishReason(event, state.hasFunctionCall),
usage: mapUsage(event.response?.usage),
...(event.response?.id || event.response?.service_tier
? {
providerMetadata: openaiMetadata({
providerMetadata:
event.response?.id || event.response?.service_tier
? openaiMetadata({
responseId: event.response.id,
serviceTier: event.response.service_tier,
}),
}
: {}),
},
})
: undefined,
}),
],
]
const onResponseFailed = (state: ParserState, event: OpenAIResponsesEvent): StepResult => [
state,
[{ type: "provider-error", message: event.message ?? event.code ?? "OpenAI Responses response failed" }],
[LLMEvent.providerError({ message: event.message ?? event.code ?? "OpenAI Responses response failed" })],
]
const onError = (state: ParserState, event: OpenAIResponsesEvent): StepResult => [
state,
[{ type: "provider-error", message: event.message ?? event.code ?? "OpenAI Responses stream error" }],
[LLMEvent.providerError({ message: event.message ?? event.code ?? "OpenAI Responses stream error" })],
]
const step = (state: ParserState, event: OpenAIResponsesEvent) => {
+38
View File
@@ -42,6 +42,11 @@ export interface ToolAccumulator {
* supplied total; otherwise falls back to `inputTokens + outputTokens` only
* when at least one is defined. Returns `undefined` when neither input nor
* output is known so routes don't publish a misleading `0`.
*
* Under the `LLM.Usage` contract, `inputTokens` and `outputTokens` are
* inclusive totals, so the computed fallback already covers cache reads /
* writes and reasoning — used mainly for Anthropic-style providers that
* don't surface a top-level total.
*/
export const totalTokens = (
inputTokens: number | undefined,
@@ -53,6 +58,39 @@ export const totalTokens = (
return (inputTokens ?? 0) + (outputTokens ?? 0)
}
/**
* Subtract `subtrahend` from `total`, clamping to zero if the provider
* reports a non-sensical breakdown (e.g. `cached_tokens > prompt_tokens`).
* Used by protocol mappers when deriving a non-overlapping breakdown field
* from a provider's inclusive total — `nonCachedInputTokens` from
* `inputTokens - cacheReadInputTokens - cacheWriteInputTokens`.
*
* If `total` is `undefined`, returns `undefined` (we don't fabricate
* counts). If `subtrahend` is `undefined`, returns `total` unchanged. The
* provider-native breakdown stays available on `Usage.providerMetadata`
* for debugging.
*/
export const subtractTokens = (
total: number | undefined,
subtrahend: number | undefined,
): number | undefined => {
if (total === undefined) return undefined
if (subtrahend === undefined) return total
return Math.max(0, total - subtrahend)
}
/**
* Sum a list of optional token counts, returning `undefined` only when
* every value is `undefined` (so we don't fabricate a `0`). Used by
* protocol mappers to derive the inclusive `inputTokens` total from a
* provider that natively reports a non-overlapping breakdown
* (e.g. Anthropic, whose `input_tokens` is already non-cached only).
*/
export const sumTokens = (...values: ReadonlyArray<number | undefined>): number | undefined => {
if (values.every((value) => value === undefined)) return undefined
return values.reduce<number>((acc, value) => acc + (value ?? 0), 0)
}
export const eventError = (route: string, message: string, raw?: string) =>
new LLMError({
module: "ProviderShared",
+14 -24
View File
@@ -1,5 +1,5 @@
import { Effect } from "effect"
import { LLMError, type ProviderMetadata, type ToolCall, type ToolInputDelta } from "../../schema"
import { LLMError, LLMEvent, type ProviderMetadata, type ToolCall, type ToolInputDelta } from "../../schema"
import { eventError, parseToolInput, type ToolAccumulator } from "../shared"
type StreamKey = string | number
@@ -49,34 +49,24 @@ const withoutTool = <K extends StreamKey>(tools: State<K>, key: K): State<K> =>
return next
}
const inputDelta = (tool: PendingTool, text: string): ToolInputDelta => ({
type: "tool-input-delta",
id: tool.id,
name: tool.name,
text,
...(tool.providerMetadata ? { providerMetadata: tool.providerMetadata } : {}),
})
const inputDelta = (tool: PendingTool, text: string): ToolInputDelta =>
LLMEvent.toolInputDelta({
id: tool.id,
name: tool.name,
text,
})
const toolCall = (route: string, tool: PendingTool, inputOverride?: string) =>
parseToolInput(route, tool.name, inputOverride ?? tool.input).pipe(
Effect.map(
(input): ToolCall =>
tool.providerExecuted
? {
type: "tool-call",
id: tool.id,
name: tool.name,
input,
providerExecuted: true,
...(tool.providerMetadata ? { providerMetadata: tool.providerMetadata } : {}),
}
: {
type: "tool-call",
id: tool.id,
name: tool.name,
input,
...(tool.providerMetadata ? { providerMetadata: tool.providerMetadata } : {}),
},
LLMEvent.toolCall({
id: tool.id,
name: tool.name,
input,
providerExecuted: tool.providerExecuted ? true : undefined,
providerMetadata: tool.providerMetadata,
}),
),
)
+148 -30
View File
@@ -1,73 +1,155 @@
import { Schema } from "effect"
import { FinishReason, ProtocolID, ProviderMetadata, RouteID } from "./ids"
import { ContentBlockID, FinishReason, ProtocolID, ProviderMetadata, ResponseID, RouteID, ToolCallID } from "./ids"
import { ModelRef } from "./options"
import { ToolResultValue } from "./messages"
/**
* Token usage reported by an LLM provider.
*
* **Inclusive totals** (match AI SDK / OpenAI / LangChain convention — a
* reader from any of those ecosystems sees the number they expect):
*
* - `inputTokens` — total prompt tokens, *including* cached reads/writes.
* - `outputTokens` — total output tokens, *including* reasoning.
* - `totalTokens` — provider-supplied total, or `inputTokens + outputTokens`.
*
* **Non-overlapping breakdown** (every field is independently meaningful;
* consumers never have to subtract):
*
* - `nonCachedInputTokens` — the "fresh" portion of the prompt.
* - `cacheReadInputTokens` — input tokens served from cache.
* - `cacheWriteInputTokens` — input tokens written to cache.
* - `reasoningTokens` — subset of `outputTokens` spent on hidden reasoning.
*
* **Invariant**: `nonCachedInputTokens + cacheReadInputTokens +
* cacheWriteInputTokens = inputTokens`, and `reasoningTokens ≤ outputTokens`.
* Each protocol mapper computes whichever side it doesn't get natively,
* with `Math.max(0, …)` clamping for defense against provider bugs. Because
* every breakdown field is stored independently, downstream consumers can
* read whatever they need (cost-by-category, context-pressure, AI-SDK-style
* inclusive total) without ever subtracting — eliminating the underflow
* class of bug where a clamped difference would silently store the wrong
* value.
*
* **Semantics by provider**:
*
* - OpenAI Chat / Responses / Gemini / Bedrock: provider reports inclusive
* `inputTokens` and an inclusive `outputTokens`; mapper subtracts to
* derive the breakdown.
* - Anthropic: provider reports the breakdown natively (`input_tokens` is
* non-cached only); mapper sums to derive the inclusive `inputTokens`.
* Anthropic does *not* break extended-thinking out of `output_tokens`, so
* `reasoningTokens` is `undefined` and `outputTokens` carries the
* combined total — a documented limitation of the Anthropic API.
*
* `providerMetadata` always carries the provider's raw usage payload —
* keyed by provider name (`{ openai: ... }`, `{ anthropic: ... }`, etc.)
* — for fields we don't normalize and for billing-level audit trails.
* Matches the same escape-hatch field on `LLMEvent`.
*/
export class Usage extends Schema.Class<Usage>("LLM.Usage")({
inputTokens: Schema.optional(Schema.Number),
outputTokens: Schema.optional(Schema.Number),
reasoningTokens: Schema.optional(Schema.Number),
nonCachedInputTokens: Schema.optional(Schema.Number),
cacheReadInputTokens: Schema.optional(Schema.Number),
cacheWriteInputTokens: Schema.optional(Schema.Number),
reasoningTokens: Schema.optional(Schema.Number),
totalTokens: Schema.optional(Schema.Number),
native: Schema.optional(Schema.Record(Schema.String, Schema.Unknown)),
}) {}
providerMetadata: Schema.optional(ProviderMetadata),
}) {
/**
* Visible output tokens — `outputTokens` minus `reasoningTokens`, clamped
* to zero. The one place subtraction happens in this contract; the clamp
* means a provider reporting `reasoningTokens > outputTokens` produces a
* harmless zero rather than a negative that crashes downstream schemas.
*/
get visibleOutputTokens() {
return Math.max(0, (this.outputTokens ?? 0) - (this.reasoningTokens ?? 0))
}
}
export const RequestStart = Schema.Struct({
type: Schema.Literal("request-start"),
id: Schema.String,
type: Schema.tag("request-start"),
id: ResponseID,
model: ModelRef,
}).annotate({ identifier: "LLM.Event.RequestStart" })
export type RequestStart = Schema.Schema.Type<typeof RequestStart>
export const StepStart = Schema.Struct({
type: Schema.Literal("step-start"),
type: Schema.tag("step-start"),
index: Schema.Number,
}).annotate({ identifier: "LLM.Event.StepStart" })
export type StepStart = Schema.Schema.Type<typeof StepStart>
export const TextStart = Schema.Struct({
type: Schema.Literal("text-start"),
id: Schema.String,
type: Schema.tag("text-start"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextStart" })
export type TextStart = Schema.Schema.Type<typeof TextStart>
export const TextDelta = Schema.Struct({
type: Schema.Literal("text-delta"),
id: Schema.optional(Schema.String),
type: Schema.tag("text-delta"),
id: ContentBlockID,
text: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextDelta" })
export type TextDelta = Schema.Schema.Type<typeof TextDelta>
export const TextEnd = Schema.Struct({
type: Schema.Literal("text-end"),
id: Schema.String,
type: Schema.tag("text-end"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.TextEnd" })
export type TextEnd = Schema.Schema.Type<typeof TextEnd>
export const ReasoningDelta = Schema.Struct({
type: Schema.Literal("reasoning-delta"),
id: Schema.optional(Schema.String),
text: Schema.String,
export const ReasoningStart = Schema.Struct({
type: Schema.tag("reasoning-start"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ReasoningStart" })
export type ReasoningStart = Schema.Schema.Type<typeof ReasoningStart>
export const ReasoningDelta = Schema.Struct({
type: Schema.tag("reasoning-delta"),
id: ContentBlockID,
text: Schema.String,
}).annotate({ identifier: "LLM.Event.ReasoningDelta" })
export type ReasoningDelta = Schema.Schema.Type<typeof ReasoningDelta>
export const ReasoningEnd = Schema.Struct({
type: Schema.tag("reasoning-end"),
id: ContentBlockID,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ReasoningEnd" })
export type ReasoningEnd = Schema.Schema.Type<typeof ReasoningEnd>
export const ToolInputStart = Schema.Struct({
type: Schema.tag("tool-input-start"),
id: ToolCallID,
name: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolInputStart" })
export type ToolInputStart = Schema.Schema.Type<typeof ToolInputStart>
export const ToolInputDelta = Schema.Struct({
type: Schema.Literal("tool-input-delta"),
id: Schema.String,
type: Schema.tag("tool-input-delta"),
id: ToolCallID,
name: Schema.String,
text: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolInputDelta" })
export type ToolInputDelta = Schema.Schema.Type<typeof ToolInputDelta>
export const ToolInputEnd = Schema.Struct({
type: Schema.tag("tool-input-end"),
id: ToolCallID,
name: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
}).annotate({ identifier: "LLM.Event.ToolInputEnd" })
export type ToolInputEnd = Schema.Schema.Type<typeof ToolInputEnd>
export const ToolCall = Schema.Struct({
type: Schema.Literal("tool-call"),
id: Schema.String,
type: Schema.tag("tool-call"),
id: ToolCallID,
name: Schema.String,
input: Schema.Unknown,
providerExecuted: Schema.optional(Schema.Boolean),
@@ -76,8 +158,8 @@ export const ToolCall = Schema.Struct({
export type ToolCall = Schema.Schema.Type<typeof ToolCall>
export const ToolResult = Schema.Struct({
type: Schema.Literal("tool-result"),
id: Schema.String,
type: Schema.tag("tool-result"),
id: ToolCallID,
name: Schema.String,
result: ToolResultValue,
providerExecuted: Schema.optional(Schema.Boolean),
@@ -86,8 +168,8 @@ export const ToolResult = Schema.Struct({
export type ToolResult = Schema.Schema.Type<typeof ToolResult>
export const ToolError = Schema.Struct({
type: Schema.Literal("tool-error"),
id: Schema.String,
type: Schema.tag("tool-error"),
id: ToolCallID,
name: Schema.String,
message: Schema.String,
providerMetadata: Schema.optional(ProviderMetadata),
@@ -95,7 +177,7 @@ export const ToolError = Schema.Struct({
export type ToolError = Schema.Schema.Type<typeof ToolError>
export const StepFinish = Schema.Struct({
type: Schema.Literal("step-finish"),
type: Schema.tag("step-finish"),
index: Schema.Number,
reason: FinishReason,
usage: Schema.optional(Usage),
@@ -104,7 +186,7 @@ export const StepFinish = Schema.Struct({
export type StepFinish = Schema.Schema.Type<typeof StepFinish>
export const RequestFinish = Schema.Struct({
type: Schema.Literal("request-finish"),
type: Schema.tag("request-finish"),
reason: FinishReason,
usage: Schema.optional(Usage),
providerMetadata: Schema.optional(ProviderMetadata),
@@ -112,7 +194,7 @@ export const RequestFinish = Schema.Struct({
export type RequestFinish = Schema.Schema.Type<typeof RequestFinish>
export const ProviderErrorEvent = Schema.Struct({
type: Schema.Literal("provider-error"),
type: Schema.tag("provider-error"),
message: Schema.String,
retryable: Schema.optional(Schema.Boolean),
providerMetadata: Schema.optional(ProviderMetadata),
@@ -125,8 +207,12 @@ const llmEventTagged = Schema.Union([
TextStart,
TextDelta,
TextEnd,
ReasoningStart,
ReasoningDelta,
ReasoningEnd,
ToolInputStart,
ToolInputDelta,
ToolInputEnd,
ToolCall,
ToolResult,
ToolError,
@@ -135,20 +221,52 @@ const llmEventTagged = Schema.Union([
ProviderErrorEvent,
]).pipe(Schema.toTaggedUnion("type"))
type WithID<Event extends { readonly id: unknown }, ID> = Omit<Event, "type" | "id"> & { readonly id: ID | string }
const responseID = (value: ResponseID | string) => ResponseID.make(value)
const contentBlockID = (value: ContentBlockID | string) => ContentBlockID.make(value)
const toolCallID = (value: ToolCallID | string) => ToolCallID.make(value)
/**
* camelCase aliases for `LLMEvent.guards` (provided by `Schema.toTaggedUnion`).
* Lets consumers write `events.filter(LLMEvent.is.toolCall)` instead of
* `events.filter(LLMEvent.guards["tool-call"])`.
*/
export const LLMEvent = Object.assign(llmEventTagged, {
requestStart: (input: WithID<RequestStart, ResponseID>) => RequestStart.make({ ...input, id: responseID(input.id) }),
stepStart: StepStart.make,
textStart: (input: WithID<TextStart, ContentBlockID>) => TextStart.make({ ...input, id: contentBlockID(input.id) }),
textDelta: (input: WithID<TextDelta, ContentBlockID>) => TextDelta.make({ ...input, id: contentBlockID(input.id) }),
textEnd: (input: WithID<TextEnd, ContentBlockID>) => TextEnd.make({ ...input, id: contentBlockID(input.id) }),
reasoningStart: (input: WithID<ReasoningStart, ContentBlockID>) =>
ReasoningStart.make({ ...input, id: contentBlockID(input.id) }),
reasoningDelta: (input: WithID<ReasoningDelta, ContentBlockID>) =>
ReasoningDelta.make({ ...input, id: contentBlockID(input.id) }),
reasoningEnd: (input: WithID<ReasoningEnd, ContentBlockID>) =>
ReasoningEnd.make({ ...input, id: contentBlockID(input.id) }),
toolInputStart: (input: WithID<ToolInputStart, ToolCallID>) =>
ToolInputStart.make({ ...input, id: toolCallID(input.id) }),
toolInputDelta: (input: WithID<ToolInputDelta, ToolCallID>) =>
ToolInputDelta.make({ ...input, id: toolCallID(input.id) }),
toolInputEnd: (input: WithID<ToolInputEnd, ToolCallID>) => ToolInputEnd.make({ ...input, id: toolCallID(input.id) }),
toolCall: (input: WithID<ToolCall, ToolCallID>) => ToolCall.make({ ...input, id: toolCallID(input.id) }),
toolResult: (input: WithID<ToolResult, ToolCallID>) => ToolResult.make({ ...input, id: toolCallID(input.id) }),
toolError: (input: WithID<ToolError, ToolCallID>) => ToolError.make({ ...input, id: toolCallID(input.id) }),
stepFinish: StepFinish.make,
requestFinish: RequestFinish.make,
providerError: ProviderErrorEvent.make,
is: {
requestStart: llmEventTagged.guards["request-start"],
stepStart: llmEventTagged.guards["step-start"],
textStart: llmEventTagged.guards["text-start"],
textDelta: llmEventTagged.guards["text-delta"],
textEnd: llmEventTagged.guards["text-end"],
reasoningStart: llmEventTagged.guards["reasoning-start"],
reasoningDelta: llmEventTagged.guards["reasoning-delta"],
reasoningEnd: llmEventTagged.guards["reasoning-end"],
toolInputStart: llmEventTagged.guards["tool-input-start"],
toolInputDelta: llmEventTagged.guards["tool-input-delta"],
toolInputEnd: llmEventTagged.guards["tool-input-end"],
toolCall: llmEventTagged.guards["tool-call"],
toolResult: llmEventTagged.guards["tool-result"],
toolError: llmEventTagged.guards["tool-error"],
+9
View File
@@ -14,6 +14,15 @@ export type ModelID = typeof ModelID.Type
export const ProviderID = Schema.String.pipe(Schema.brand("LLM.ProviderID"))
export type ProviderID = typeof ProviderID.Type
export const ResponseID = Schema.String
export type ResponseID = Schema.Schema.Type<typeof ResponseID>
export const ContentBlockID = Schema.String
export type ContentBlockID = Schema.Schema.Type<typeof ContentBlockID>
export const ToolCallID = Schema.String
export type ToolCallID = Schema.Schema.Type<typeof ToolCallID>
export const ReasoningEfforts = ["none", "minimal", "low", "medium", "high", "xhigh", "max"] as const
export const ReasoningEffort = Schema.Literals(ReasoningEfforts)
export type ReasoningEffort = Schema.Schema.Type<typeof ReasoningEffort>
+14 -6
View File
@@ -4,7 +4,7 @@ import {
type ContentPart,
type FinishReason,
type LLMError,
type LLMEvent,
LLMEvent,
LLMRequest,
Message,
type ProviderMetadata,
@@ -115,11 +115,19 @@ interface StepState {
const accumulate = (state: StepState, event: LLMEvent) => {
if (event.type === "text-delta") {
appendStreamingText(state, "text", event.text, event.providerMetadata)
appendStreamingText(state, "text", event.text, undefined)
return
}
if (event.type === "reasoning-delta") {
appendStreamingText(state, "reasoning", event.text, event.providerMetadata)
appendStreamingText(state, "reasoning", event.text, undefined)
return
}
if (event.type === "reasoning-end") {
appendStreamingText(state, "reasoning", "", event.providerMetadata)
return
}
if (event.type === "text-end") {
appendStreamingText(state, "text", "", event.providerMetadata)
return
}
if (event.type === "tool-call") {
@@ -219,10 +227,10 @@ const decodeAndExecute = (tool: AnyTool, input: unknown): Effect.Effect<ToolResu
const emitEvents = (call: ToolCallPart, result: ToolResultValue): ReadonlyArray<LLMEvent> =>
result.type === "error"
? [
{ type: "tool-error", id: call.id, name: call.name, message: String(result.value) },
{ type: "tool-result", id: call.id, name: call.name, result },
LLMEvent.toolError({ id: call.id, name: call.name, message: String(result.value) }),
LLMEvent.toolResult({ id: call.id, name: call.name, result }),
]
: [{ type: "tool-result", id: call.id, name: call.name, result }]
: [LLMEvent.toolResult({ id: call.id, name: call.name, result })]
const followUpRequest = (
request: LLMRequest,
+3 -1
View File
@@ -50,7 +50,9 @@ const request = LLM.request({
})
const raiseEvent = (event: FakeEvent): import("../src/schema").LLMEvent =>
event.type === "finish" ? { type: "request-finish", reason: event.reason } : { type: "text-delta", text: event.text }
event.type === "finish"
? { type: "request-finish", reason: event.reason }
: { type: "text-delta", id: "text-0", text: event.text }
const fakeProtocol = Protocol.make<FakeBody, FakeEvent, FakeEvent, void>({
id: "fake",
@@ -0,0 +1,38 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/anthropic-haiku-4-5-text",
"recordedAt": "2026-05-11T02:02:03.804Z",
"provider": "anthropic",
"route": "anthropic-messages",
"transport": "http",
"model": "claude-haiku-4-5-20251001",
"tags": [
"prefix:anthropic-messages",
"provider:anthropic",
"text",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"system\":[{\"type\":\"text\",\"text\":\"You are concise.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Reply exactly with: Hello!\"}]}],\"stream\":true,\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01SvRWwb75gDuhBpVMHjnFaf\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":18,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":2,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Hello!\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":18,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":5} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}
@@ -0,0 +1,39 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/anthropic-haiku-4-5-tool-call",
"recordedAt": "2026-05-11T02:02:04.363Z",
"provider": "anthropic",
"route": "anthropic-messages",
"transport": "http",
"model": "claude-haiku-4-5-20251001",
"tags": [
"prefix:anthropic-messages",
"provider:anthropic",
"tool",
"tool-call",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"system\":[{\"type\":\"text\",\"text\":\"Call tools exactly as requested.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Call get_weather with city exactly Paris.\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"tool_choice\":{\"type\":\"tool\",\"name\":\"get_weather\"},\"stream\":true,\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_01Lu38yDM3WD8QBQTcg3dBaF\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":677,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":16,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"}}}\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_017Dqk9SAAsHHfiLKsUyitaQ\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"cit\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"y\\\": \\\"Paris\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":677,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":33} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}
@@ -0,0 +1,59 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/anthropic-opus-4-7-tool-loop",
"recordedAt": "2026-05-11T02:02:07.788Z",
"provider": "anthropic",
"route": "anthropic-messages",
"transport": "http",
"model": "claude-opus-4-7",
"tags": [
"prefix:anthropic-messages",
"provider:anthropic",
"flagship",
"tool",
"tool-loop",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-opus-4-7\",\"system\":[{\"type\":\"text\",\"text\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What is the weather in Paris?\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"stream\":true,\"max_tokens\":80}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-opus-4-7\",\"id\":\"msg_01GK5kgi8AuVfRCnQFcXEfV8\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":812,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":0,\"service_tier\":\"standard\",\"inference_geo\":\"global\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01BnVDAp13NU8ZdJ9JeJ7byF\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}}}\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"c\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"ity\\\": \\\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"Paris\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":812,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":66} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-opus-4-7\",\"system\":[{\"type\":\"text\",\"text\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"}],\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What is the weather in Paris?\"}]},{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_01BnVDAp13NU8ZdJ9JeJ7byF\",\"name\":\"get_weather\",\"input\":{\"city\":\"Paris\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"toolu_01BnVDAp13NU8ZdJ9JeJ7byF\",\"content\":\"{\\\"temperature\\\":22,\\\"condition\\\":\\\"sunny\\\"}\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"stream\":true,\"max_tokens\":80}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-opus-4-7\",\"id\":\"msg_01237VTnjPeYSRh31UjXWaEa\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":909,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":8,\"service_tier\":\"standard\",\"inference_geo\":\"global\"}} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"Paris is sunny.\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null},\"usage\":{\"input_tokens\":909,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":12} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n"
}
}
]
}
@@ -0,0 +1,35 @@
{
"version": 1,
"metadata": {
"name": "anthropic-messages/rejects-malformed-assistant-tool-order",
"recordedAt": "2026-05-11T02:01:44.544Z",
"tags": [
"prefix:anthropic-messages",
"provider:anthropic",
"protocol:anthropic-messages",
"tool",
"sad-path"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.anthropic.com/v1/messages",
"headers": {
"anthropic-version": "2023-06-01",
"content-type": "application/json"
},
"body": "{\"model\":\"claude-haiku-4-5-20251001\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_1\",\"name\":\"get_weather\",\"input\":{\"city\":\"Paris\"}},{\"type\":\"text\",\"text\":\"I will check the weather.\"}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_1\",\"content\":\"{\\\"temperature\\\":\\\"72F\\\"}\"}]},{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"Use that result to answer briefly.\"}]}],\"tools\":[{\"name\":\"get_weather\",\"description\":\"Get weather\",\"input_schema\":{\"type\":\"object\",\"properties\":{}}}],\"stream\":true,\"max_tokens\":4096}"
},
"response": {
"status": 400,
"headers": {
"content-type": "application/json"
},
"body": "{\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"messages.1: `tool_use` ids were found without `tool_result` blocks immediately after: call_1. Each `tool_use` block must have a corresponding `tool_result` block in the next message.\"},\"request_id\":\"req_011CauxVdQf3N2PPFJ5aH8Bh\"}"
}
}
]
}
@@ -0,0 +1,37 @@
{
"version": 1,
"metadata": {
"name": "gemini/gemini-2-5-flash-text",
"recordedAt": "2026-05-11T02:02:08.410Z",
"provider": "google",
"route": "gemini",
"transport": "http",
"model": "gemini-2.5-flash",
"tags": [
"prefix:gemini",
"provider:google",
"text",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:streamGenerateContent?alt=sse",
"headers": {
"content-type": "application/json"
},
"body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"text\":\"Reply exactly with: Hello!\"}]}],\"systemInstruction\":{\"parts\":[{\"text\":\"You are concise.\"}]},\"generationConfig\":{\"maxOutputTokens\":80,\"temperature\":0}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"Hello!\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 11,\"candidatesTokenCount\": 2,\"totalTokenCount\": 13,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 11}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"nzgBatP3OZmW-8YP567bqQs\"}\r\n\r\n"
}
}
]
}
@@ -0,0 +1,38 @@
{
"version": 1,
"metadata": {
"name": "gemini/gemini-2-5-flash-tool-call",
"recordedAt": "2026-05-11T02:02:09.308Z",
"provider": "google",
"route": "gemini",
"transport": "http",
"model": "gemini-2.5-flash",
"tags": [
"prefix:gemini",
"provider:google",
"tool",
"tool-call",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:streamGenerateContent?alt=sse",
"headers": {
"content-type": "application/json"
},
"body": "{\"contents\":[{\"role\":\"user\",\"parts\":[{\"text\":\"Call get_weather with city exactly Paris.\"}]}],\"systemInstruction\":{\"parts\":[{\"text\":\"Call tools exactly as requested.\"}]},\"tools\":[{\"functionDeclarations\":[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"required\":[\"city\"],\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}}}}]}],\"toolConfig\":{\"functionCallingConfig\":{\"mode\":\"ANY\",\"allowedFunctionNames\":[\"get_weather\"]}},\"generationConfig\":{\"maxOutputTokens\":80,\"temperature\":0}}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream"
},
"body": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"functionCall\": {\"name\": \"get_weather\",\"args\": {\"city\": \"Paris\"}},\"thoughtSignature\": \"CiQBDDnWx/X6sWeX2joSugyWO3L/lt0AgIPCvhpqf3845fj+H70KXwEMOdbHB/cnaqYCro0pU+yLWoA55jhuwoLmTcnYm4Qzcm5DuW/v0NUyz8RDx6DFh61juENveUztly6yc6/XiWJHtsgncd9YgcZhuQKqtp5KZTkGYpT3g6v3yP9GK4AoCoUBAQw51sePh3WuWovHnwIotKLVZiU9pwh34k4FY7ugPOxyDAG9j69cy7BYYzSchI10LEjLoLlCMZuNIPootBgI02QWY/4h2PIv33BAADrFPM2T3aE4cAuMoa3GCu2nztJ/95junDhIhuXZSQ/Mh9EVxpx7ml99Z7Hxb7OtDsSZZLeCBuGmSw==\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0,\"finishMessage\": \"Model generated function call(s).\"}],\"usageMetadata\": {\"promptTokenCount\": 55,\"candidatesTokenCount\": 15,\"totalTokenCount\": 115,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 55}],\"thoughtsTokenCount\": 45,\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"oDgBauj6HZ3B-8YPpaOc6QU\"}\r\n\r\n"
}
}
]
}
@@ -0,0 +1,37 @@
{
"version": 1,
"metadata": {
"name": "openai-chat/openai-chat-gpt-4o-mini-text",
"recordedAt": "2026-05-11T02:01:46.536Z",
"provider": "openai",
"route": "openai-chat",
"transport": "http",
"model": "gpt-4o-mini",
"tags": [
"prefix:openai-chat",
"provider:openai",
"text",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply exactly with: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DeAFmC9UiBnVu6dAjBvWVBVypm2gc\",\"object\":\"chat.completion.chunk\",\"created\":1778464906,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_3ef558a83f\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"wHajUz1js\"}\n\ndata: {\"id\":\"chatcmpl-DeAFmC9UiBnVu6dAjBvWVBVypm2gc\",\"object\":\"chat.completion.chunk\",\"created\":1778464906,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_3ef558a83f\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"lVHzGq\"}\n\ndata: {\"id\":\"chatcmpl-DeAFmC9UiBnVu6dAjBvWVBVypm2gc\",\"object\":\"chat.completion.chunk\",\"created\":1778464906,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_3ef558a83f\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"3gaQRWEjE7\"}\n\ndata: {\"id\":\"chatcmpl-DeAFmC9UiBnVu6dAjBvWVBVypm2gc\",\"object\":\"chat.completion.chunk\",\"created\":1778464906,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_3ef558a83f\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"vuKPZ\"}\n\ndata: {\"id\":\"chatcmpl-DeAFmC9UiBnVu6dAjBvWVBVypm2gc\",\"object\":\"chat.completion.chunk\",\"created\":1778464906,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_3ef558a83f\",\"choices\":[],\"usage\":{\"prompt_tokens\":21,\"completion_tokens\":2,\"total_tokens\":23,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"kfFsssdujmM\"}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,38 @@
{
"version": 1,
"metadata": {
"name": "openai-chat/openai-chat-gpt-4o-mini-tool-call",
"recordedAt": "2026-05-11T02:01:47.484Z",
"provider": "openai",
"route": "openai-chat",
"transport": "http",
"model": "gpt-4o-mini",
"tags": [
"prefix:openai-chat",
"provider:openai",
"tool",
"tool-call",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"Call tools exactly as requested.\"},{\"role\":\"user\",\"content\":\"Call get_weather with city exactly Paris.\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"tool_choice\":{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}},\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"index\":0,\"id\":\"call_sH8T7MPdJXS5KJginKrzexL5\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}],\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"d0rJ\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"hL82erd6bBoy91\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"city\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"e3aoKH4tvJlXW\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\":\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"T8EzFVNaUCLE\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"Paris\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"lelRBxF08Zes\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"F1xO8sVMyZt4BU\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"YL3pl\"}\n\ndata: {\"id\":\"chatcmpl-DeAFnVMdELyuymMRlTqN5dBdM7eGd\",\"object\":\"chat.completion.chunk\",\"created\":1778464907,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_d88f6e55bb\",\"choices\":[],\"usage\":{\"prompt_tokens\":67,\"completion_tokens\":5,\"total_tokens\":72,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"7cov9qkofwo\"}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,56 @@
{
"version": 1,
"metadata": {
"name": "openai-chat/openai-chat-gpt-4o-mini-tool-loop",
"recordedAt": "2026-05-11T02:01:50.433Z",
"provider": "openai",
"route": "openai-chat",
"transport": "http",
"model": "gpt-4o-mini",
"tags": [
"prefix:openai-chat",
"provider:openai",
"tool",
"tool-loop",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris?\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"index\":0,\"id\":\"call_HrDZhrMUauvVddKWzVQFJ69Q\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}],\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"aAqC\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"haZ3pKk14oDay0\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"city\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"ZUJHQytFyDezp\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\":\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"3buqnStPteyG\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"Paris\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"7epPqHEU3OeA\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"gSU5gyl9K7sUCv\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"usage\":null,\"obfuscation\":\"P3oSvByfy60JCsz\"}\n\ndata: {\"id\":\"chatcmpl-DeAFo2rbsuuMn0ftzxwfDpqnYANrT\",\"object\":\"chat.completion.chunk\",\"created\":1778464908,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[],\"usage\":{\"prompt_tokens\":71,\"completion_tokens\":14,\"total_tokens\":85,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"5AhQ5ToYra\"}\n\ndata: [DONE]\n\n"
}
},
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"},{\"role\":\"user\",\"content\":\"What is the weather in Paris?\"},{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"id\":\"call_HrDZhrMUauvVddKWzVQFJ69Q\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"}}]},{\"role\":\"tool\",\"tool_call_id\":\"call_HrDZhrMUauvVddKWzVQFJ69Q\",\"content\":\"{\\\"temperature\\\":22,\\\"condition\\\":\\\"sunny\\\"}\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"2QWjdWPSe\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Paris\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"sm5IpQ\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" is\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"cWsGxqHw\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" sunny\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"8hNCi\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"4yKCNDRcwq\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"NvRgg\"}\n\ndata: {\"id\":\"chatcmpl-DeAFpaPsUwxt5hjwGgC4BBmlALxwR\",\"object\":\"chat.completion.chunk\",\"created\":1778464909,\"model\":\"gpt-4o-mini-2024-07-18\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_c83829e916\",\"choices\":[],\"usage\":{\"prompt_tokens\":103,\"completion_tokens\":5,\"total_tokens\":108,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"4s7mLtNpZ\"}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,37 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/deepseek-chat-text",
"recordedAt": "2026-05-11T02:02:10.220Z",
"provider": "deepseek",
"route": "openai-compatible-chat",
"transport": "http",
"model": "deepseek-chat",
"tags": [
"prefix:openai-compatible-chat",
"provider:deepseek",
"text",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.deepseek.com/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"deepseek-chat\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply exactly with: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream; charset=utf-8"
},
"body": "data: {\"id\":\"159019a5-f981-4103-8b3c-2be37cefc505\",\"object\":\"chat.completion.chunk\",\"created\":1778464929,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"159019a5-f981-4103-8b3c-2be37cefc505\",\"object\":\"chat.completion.chunk\",\"created\":1778464929,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"159019a5-f981-4103-8b3c-2be37cefc505\",\"object\":\"chat.completion.chunk\",\"created\":1778464929,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"159019a5-f981-4103-8b3c-2be37cefc505\",\"object\":\"chat.completion.chunk\",\"created\":1778464929,\"model\":\"deepseek-v4-flash\",\"system_fingerprint\":\"fp_058df29938_prod0820_fp8_kvcache_20260402\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\"},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":14,\"completion_tokens\":2,\"total_tokens\":16,\"prompt_tokens_details\":{\"cached_tokens\":0},\"prompt_cache_hit_tokens\":0,\"prompt_cache_miss_tokens\":14}}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,37 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/togetherai-llama-3-3-70b-text",
"recordedAt": "2026-05-11T02:02:13.341Z",
"provider": "togetherai",
"route": "openai-compatible-chat",
"transport": "http",
"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo",
"tags": [
"prefix:openai-compatible-chat",
"provider:togetherai",
"text",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.together.xyz/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"messages\":[{\"role\":\"system\",\"content\":\"You are concise.\"},{\"role\":\"user\",\"content\":\"Reply exactly with: Hello!\"}],\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":40,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream;charset=utf-8"
},
"body": "data: {\"id\":\"oibreET-3pDw3Z-9f9d9996ce37066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464931,\"choices\":[{\"index\":0,\"text\":\"Hello\",\"logprobs\":null,\"finish_reason\":null,\"seed\":null,\"delta\":{\"token_id\":9906,\"role\":\"assistant\",\"content\":\"Hello\"}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"usage\":null}\n\ndata: {\"id\":\"oibreET-3pDw3Z-9f9d9996ce37066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464931,\"choices\":[{\"index\":0,\"text\":\"!\",\"logprobs\":null,\"finish_reason\":null,\"seed\":null,\"delta\":{\"token_id\":null,\"role\":\"assistant\",\"content\":\"!\"}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"usage\":null}\n\ndata: {\"id\":\"oibreET-3pDw3Z-9f9d9996ce37066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464931,\"choices\":[{\"index\":0,\"text\":\"\",\"logprobs\":null,\"finish_reason\":\"stop\",\"seed\":12144769634208630000,\"delta\":{\"token_id\":128009,\"role\":\"assistant\",\"content\":\"\"}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"usage\":{\"prompt_tokens\":45,\"completion_tokens\":3,\"total_tokens\":48,\"cached_tokens\":0}}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,38 @@
{
"version": 1,
"metadata": {
"name": "openai-compatible-chat/togetherai-llama-3-3-70b-tool-call",
"recordedAt": "2026-05-11T02:02:14.453Z",
"provider": "togetherai",
"route": "openai-compatible-chat",
"transport": "http",
"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo",
"tags": [
"prefix:openai-compatible-chat",
"provider:togetherai",
"tool",
"tool-call",
"golden"
]
},
"interactions": [
{
"transport": "http",
"request": {
"method": "POST",
"url": "https://api.together.xyz/v1/chat/completions",
"headers": {
"content-type": "application/json"
},
"body": "{\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"messages\":[{\"role\":\"system\",\"content\":\"Call tools exactly as requested.\"},{\"role\":\"user\",\"content\":\"Call get_weather with city exactly Paris.\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}}],\"tool_choice\":{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}},\"stream\":true,\"stream_options\":{\"include_usage\":true},\"max_tokens\":80,\"temperature\":0}"
},
"response": {
"status": 200,
"headers": {
"content-type": "text/event-stream;charset=utf-8"
},
"body": "data: {\"id\":\"oibreu3-6Ng1vN-9f9d99a9ba73066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464933,\"choices\":[{\"index\":0,\"role\":\"assistant\",\"text\":\"\",\"logprobs\":null,\"finish_reason\":null,\"delta\":{\"token_id\":null,\"role\":\"assistant\",\"content\":\"\"}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\"}\n\ndata: {\"id\":\"oibreu3-6Ng1vN-9f9d99a9ba73066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464933,\"choices\":[{\"index\":0,\"text\":\"\",\"logprobs\":null,\"finish_reason\":null,\"delta\":{\"token_id\":null,\"role\":\"assistant\",\"content\":\"\",\"tool_calls\":[{\"index\":0,\"id\":\"call_jue52dtu6iozr0ny9rq357u5\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}]}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\"}\n\ndata: {\"id\":\"oibreu3-6Ng1vN-9f9d99a9ba73066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464933,\"choices\":[{\"index\":0,\"text\":\"\",\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"delta\":{\"token_id\":null,\"role\":\"assistant\",\"content\":\"\",\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"}}]}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\"}\n\ndata: {\"id\":\"oibreu3-6Ng1vN-9f9d99a9ba73066b\",\"object\":\"chat.completion.chunk\",\"created\":1778464933,\"choices\":[{\"index\":0,\"text\":\"\",\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"seed\":17440360718047570000,\"delta\":{\"token_id\":128009,\"role\":\"assistant\",\"content\":\"\"}}],\"model\":\"meta-llama/Llama-3.3-70B-Instruct-Turbo\",\"usage\":{\"prompt_tokens\":194,\"completion_tokens\":19,\"total_tokens\":213,\"cached_tokens\":0}}\n\ndata: [DONE]\n\n"
}
}
]
}
@@ -0,0 +1,143 @@
{
"version": 1,
"metadata": {
"name": "openai-responses-websocket/openai-responses-websocket-gpt-4-1-mini-tool-loop",
"recordedAt": "2026-05-11T02:02:03.284Z",
"provider": "openai",
"route": "openai-responses-websocket",
"transport": "websocket",
"model": "gpt-4.1-mini",
"tags": [
"prefix:openai-responses-websocket",
"provider:openai",
"transport:websocket",
"tool",
"tool-loop",
"golden"
]
},
"interactions": [
{
"transport": "websocket",
"open": {
"url": "wss://api.openai.com/v1/responses",
"headers": {}
},
"client": [
{
"kind": "text",
"body": "{\"type\":\"response.create\",\"model\":\"gpt-4.1-mini\",\"input\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"},{\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"What is the weather in Paris?\"}]}],\"tools\":[{\"type\":\"function\",\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"store\":false,\"max_output_tokens\":80,\"temperature\":0}"
}
],
"server": [
{
"kind": "text",
"body": "{\"type\":\"response.created\",\"response\":{\"id\":\"resp_0f8d81b5d3287513016a013897415481a28585571d6710366f\",\"object\":\"response\",\"created_at\":1778464919,\"status\":\"in_progress\",\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":0}"
},
{
"kind": "text",
"body": "{\"type\":\"response.in_progress\",\"response\":{\"id\":\"resp_0f8d81b5d3287513016a013897415481a28585571d6710366f\",\"object\":\"response\",\"created_at\":1778464919,\"status\":\"in_progress\",\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":1}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_item.added\",\"item\":{\"id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"type\":\"function_call\",\"status\":\"in_progress\",\"arguments\":\"\",\"call_id\":\"call_wRJoUdcb6w5SIpieBFOKxF6z\",\"name\":\"get_weather\"},\"output_index\":0,\"sequence_number\":2}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.delta\",\"delta\":\"{\\\"\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"obfuscation\":\"EAdvUfYaysnNd3\",\"output_index\":0,\"sequence_number\":3}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.delta\",\"delta\":\"city\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"obfuscation\":\"b0QWJJlscLyl\",\"output_index\":0,\"sequence_number\":4}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.delta\",\"delta\":\"\\\":\\\"\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"obfuscation\":\"m5EV1wzceCkjb\",\"output_index\":0,\"sequence_number\":5}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.delta\",\"delta\":\"Paris\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"obfuscation\":\"QM3FjPoN9Gi\",\"output_index\":0,\"sequence_number\":6}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.delta\",\"delta\":\"\\\"}\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"obfuscation\":\"lug74orAYXtg0e\",\"output_index\":0,\"sequence_number\":7}"
},
{
"kind": "text",
"body": "{\"type\":\"response.function_call_arguments.done\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\",\"item_id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"output_index\":0,\"sequence_number\":8}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_item.done\",\"item\":{\"id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"type\":\"function_call\",\"status\":\"completed\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\",\"call_id\":\"call_wRJoUdcb6w5SIpieBFOKxF6z\",\"name\":\"get_weather\"},\"output_index\":0,\"sequence_number\":9}"
},
{
"kind": "text",
"body": "{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_0f8d81b5d3287513016a013897415481a28585571d6710366f\",\"object\":\"response\",\"created_at\":1778464919,\"status\":\"completed\",\"background\":false,\"completed_at\":1778464920,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[{\"id\":\"fc_0f8d81b5d3287513016a01389815f081a2a68583ed9b19c405\",\"type\":\"function_call\",\"status\":\"completed\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\",\"call_id\":\"call_wRJoUdcb6w5SIpieBFOKxF6z\",\"name\":\"get_weather\"}],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"default\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":{\"input_tokens\":69,\"input_tokens_details\":{\"cached_tokens\":0},\"output_tokens\":15,\"output_tokens_details\":{\"reasoning_tokens\":0},\"total_tokens\":84},\"user\":null,\"metadata\":{}},\"sequence_number\":10}"
}
]
},
{
"transport": "websocket",
"open": {
"url": "wss://api.openai.com/v1/responses",
"headers": {}
},
"client": [
{
"kind": "text",
"body": "{\"type\":\"response.create\",\"model\":\"gpt-4.1-mini\",\"input\":[{\"role\":\"system\",\"content\":\"Use the get_weather tool exactly once. After the tool result, reply exactly: Paris is sunny.\"},{\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"What is the weather in Paris?\"}]},{\"type\":\"function_call\",\"call_id\":\"call_wRJoUdcb6w5SIpieBFOKxF6z\",\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\"},{\"type\":\"function_call_output\",\"call_id\":\"call_wRJoUdcb6w5SIpieBFOKxF6z\",\"output\":\"{\\\"temperature\\\":22,\\\"condition\\\":\\\"sunny\\\"}\"}],\"tools\":[{\"type\":\"function\",\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false}}],\"store\":false,\"max_output_tokens\":80,\"temperature\":0}"
}
],
"server": [
{
"kind": "text",
"body": "{\"type\":\"response.created\",\"response\":{\"id\":\"resp_0767cfe3f5d98b2a016a0138994258819485d082e5c78849a4\",\"object\":\"response\",\"created_at\":1778464921,\"status\":\"in_progress\",\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":0}"
},
{
"kind": "text",
"body": "{\"type\":\"response.in_progress\",\"response\":{\"id\":\"resp_0767cfe3f5d98b2a016a0138994258819485d082e5c78849a4\",\"object\":\"response\",\"created_at\":1778464921,\"status\":\"in_progress\",\"background\":false,\"completed_at\":null,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"auto\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":null,\"user\":null,\"metadata\":{}},\"sequence_number\":1}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_item.added\",\"item\":{\"id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"type\":\"message\",\"status\":\"in_progress\",\"content\":[],\"role\":\"assistant\"},\"output_index\":0,\"sequence_number\":2}"
},
{
"kind": "text",
"body": "{\"type\":\"response.content_part.added\",\"content_index\":0,\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"output_index\":0,\"part\":{\"type\":\"output_text\",\"annotations\":[],\"logprobs\":[],\"text\":\"\"},\"sequence_number\":3}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\"Paris\",\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"logprobs\":[],\"obfuscation\":\"fWbkBKTZ5oG\",\"output_index\":0,\"sequence_number\":4}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\" is\",\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"logprobs\":[],\"obfuscation\":\"BGuPlDrPchXwG\",\"output_index\":0,\"sequence_number\":5}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\" sunny\",\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"logprobs\":[],\"obfuscation\":\"7THzde2pni\",\"output_index\":0,\"sequence_number\":6}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_text.delta\",\"content_index\":0,\"delta\":\".\",\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"logprobs\":[],\"obfuscation\":\"Eo38bSElfyNmljj\",\"output_index\":0,\"sequence_number\":7}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_text.done\",\"content_index\":0,\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"logprobs\":[],\"output_index\":0,\"sequence_number\":8,\"text\":\"Paris is sunny.\"}"
},
{
"kind": "text",
"body": "{\"type\":\"response.content_part.done\",\"content_index\":0,\"item_id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"output_index\":0,\"part\":{\"type\":\"output_text\",\"annotations\":[],\"logprobs\":[],\"text\":\"Paris is sunny.\"},\"sequence_number\":9}"
},
{
"kind": "text",
"body": "{\"type\":\"response.output_item.done\",\"item\":{\"id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"type\":\"message\",\"status\":\"completed\",\"content\":[{\"type\":\"output_text\",\"annotations\":[],\"logprobs\":[],\"text\":\"Paris is sunny.\"}],\"role\":\"assistant\"},\"output_index\":0,\"sequence_number\":10}"
},
{
"kind": "text",
"body": "{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_0767cfe3f5d98b2a016a0138994258819485d082e5c78849a4\",\"object\":\"response\",\"created_at\":1778464921,\"status\":\"completed\",\"background\":false,\"completed_at\":1778464923,\"error\":null,\"frequency_penalty\":0.0,\"incomplete_details\":null,\"instructions\":null,\"max_output_tokens\":80,\"max_tool_calls\":null,\"model\":\"gpt-4.1-mini-2025-04-14\",\"moderation\":null,\"output\":[{\"id\":\"msg_0767cfe3f5d98b2a016a01389b1434819493f3d33e0ec6973c\",\"type\":\"message\",\"status\":\"completed\",\"content\":[{\"type\":\"output_text\",\"annotations\":[],\"logprobs\":[],\"text\":\"Paris is sunny.\"}],\"role\":\"assistant\"}],\"parallel_tool_calls\":true,\"presence_penalty\":0.0,\"previous_response_id\":null,\"prompt_cache_key\":null,\"prompt_cache_retention\":\"in_memory\",\"reasoning\":{\"effort\":null,\"summary\":null},\"safety_identifier\":null,\"service_tier\":\"default\",\"store\":false,\"temperature\":0.0,\"text\":{\"format\":{\"type\":\"text\"},\"verbosity\":\"medium\"},\"tool_choice\":\"auto\",\"tools\":[{\"type\":\"function\",\"description\":\"Get current weather for a city.\",\"name\":\"get_weather\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"],\"additionalProperties\":false},\"strict\":true}],\"top_logprobs\":0,\"top_p\":1.0,\"truncation\":\"disabled\",\"usage\":{\"input_tokens\":99,\"input_tokens_details\":{\"cached_tokens\":0},\"output_tokens\":6,\"output_tokens_details\":{\"reasoning_tokens\":0},\"total_tokens\":105},\"user\":null,\"metadata\":{}},\"sequence_number\":11}"
}
]
}
]
}
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -126,7 +126,7 @@ describe("llm constructors", () => {
expect(
LLMResponse.text({
events: [
{ type: "text-delta", text: "hi" },
{ type: "text-delta", id: "text-0", text: "hi" },
{ type: "request-finish", reason: "stop" },
],
}),
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { CacheHint, LLM, LLMError } from "../../src"
import { CacheHint, LLM, LLMError, Usage } from "../../src"
import { LLMClient } from "../../src/route"
import * as AnthropicMessages from "../../src/protocols/anthropic-messages"
import { it } from "../lib/effect"
@@ -110,12 +110,13 @@ describe("Anthropic Messages route", () => {
expect(response.text).toBe("Hello!")
expect(response.reasoning).toBe("thinking")
expect(response.usage).toMatchObject({
inputTokens: 5,
inputTokens: 6,
outputTokens: 2,
nonCachedInputTokens: 5,
cacheReadInputTokens: 1,
totalTokens: 7,
totalTokens: 8,
})
expect(response.events.find((event) => event.type === "reasoning-delta" && event.text === "")).toMatchObject({
expect(response.events.find((event) => event.type === "reasoning-end")).toMatchObject({
providerMetadata: { anthropic: { signature: "sig_1" } },
})
expect(response.events.at(-1)).toMatchObject({
@@ -152,7 +153,13 @@ describe("Anthropic Messages route", () => {
{
type: "request-finish",
reason: "tool-calls",
usage: { inputTokens: 5, outputTokens: 1, totalTokens: 6, native: { input_tokens: 5, output_tokens: 1 } },
usage: new Usage({
inputTokens: 5,
outputTokens: 1,
nonCachedInputTokens: 5,
totalTokens: 6,
providerMetadata: { anthropic: { input_tokens: 5, output_tokens: 1 } },
}),
},
])
}),
+24 -19
View File
@@ -1,6 +1,6 @@
import { describe, expect } from "bun:test"
import { Effect } from "effect"
import { LLM, LLMError } from "../../src"
import { LLM, LLMError, Usage } from "../../src"
import { LLMClient } from "../../src/route"
import * as Gemini from "../../src/protocols/gemini"
import { it } from "../lib/effect"
@@ -198,32 +198,36 @@ describe("Gemini route", () => {
expect(response.reasoning).toBe("thinking")
expect(response.usage).toMatchObject({
inputTokens: 5,
outputTokens: 2,
reasoningTokens: 1,
outputTokens: 3,
nonCachedInputTokens: 4,
cacheReadInputTokens: 1,
reasoningTokens: 1,
totalTokens: 7,
})
expect(response.events).toEqual([
{ type: "reasoning-delta", text: "thinking" },
{ type: "text-delta", text: "Hello" },
{ type: "text-delta", text: "!" },
{ type: "reasoning-delta", id: "reasoning-0", text: "thinking" },
{ type: "text-delta", id: "text-0", text: "Hello" },
{ type: "text-delta", id: "text-0", text: "!" },
{
type: "request-finish",
reason: "stop",
usage: {
usage: new Usage({
inputTokens: 5,
outputTokens: 2,
reasoningTokens: 1,
outputTokens: 3,
nonCachedInputTokens: 4,
cacheReadInputTokens: 1,
reasoningTokens: 1,
totalTokens: 7,
native: {
promptTokenCount: 5,
candidatesTokenCount: 2,
totalTokenCount: 7,
thoughtsTokenCount: 1,
cachedContentTokenCount: 1,
providerMetadata: {
google: {
promptTokenCount: 5,
candidatesTokenCount: 2,
totalTokenCount: 7,
thoughtsTokenCount: 1,
cachedContentTokenCount: 1,
},
},
},
}),
},
])
}),
@@ -257,12 +261,13 @@ describe("Gemini route", () => {
{
type: "request-finish",
reason: "tool-calls",
usage: {
usage: new Usage({
inputTokens: 5,
outputTokens: 1,
nonCachedInputTokens: 5,
totalTokens: 6,
native: { promptTokenCount: 5, candidatesTokenCount: 1 },
},
providerMetadata: { google: { promptTokenCount: 5, candidatesTokenCount: 1 } },
}),
},
])
}),
+15 -12
View File
@@ -1,7 +1,7 @@
import { describe, expect } from "bun:test"
import { Effect, Schema, Stream } from "effect"
import { HttpClientRequest } from "effect/unstable/http"
import { LLM, LLMError } from "../../src"
import { LLM, LLMError, Usage } from "../../src"
import * as Azure from "../../src/providers/azure"
import * as OpenAI from "../../src/providers/openai"
import * as OpenAIChat from "../../src/protocols/openai-chat"
@@ -225,25 +225,28 @@ describe("OpenAI Chat route", () => {
expect(response.text).toBe("Hello!")
expect(response.events).toEqual([
{ type: "text-delta", text: "Hello" },
{ type: "text-delta", text: "!" },
{ type: "text-delta", id: "text-0", text: "Hello" },
{ type: "text-delta", id: "text-0", text: "!" },
{
type: "request-finish",
reason: "stop",
usage: {
usage: new Usage({
inputTokens: 5,
outputTokens: 2,
reasoningTokens: 0,
nonCachedInputTokens: 4,
cacheReadInputTokens: 1,
reasoningTokens: 0,
totalTokens: 7,
native: {
prompt_tokens: 5,
completion_tokens: 2,
total_tokens: 7,
prompt_tokens_details: { cached_tokens: 1 },
completion_tokens_details: { reasoning_tokens: 0 },
providerMetadata: {
openai: {
prompt_tokens: 5,
completion_tokens: 2,
total_tokens: 7,
prompt_tokens_details: { cached_tokens: 1 },
completion_tokens_details: { reasoning_tokens: 0 },
},
},
},
}),
},
])
}),
@@ -1,7 +1,7 @@
import { describe, expect } from "bun:test"
import { ConfigProvider, Effect, Layer, Stream } from "effect"
import { Headers, HttpClientRequest } from "effect/unstable/http"
import { LLM, LLMError } from "../../src"
import { LLM, LLMError, Usage } from "../../src"
import { Auth, LLMClient, RequestExecutor, WebSocketExecutor } from "../../src/route"
import * as Azure from "../../src/providers/azure"
import * as OpenAI from "../../src/providers/openai"
@@ -336,26 +336,29 @@ describe("OpenAI Responses route", () => {
expect(response.text).toBe("Hello!")
expect(response.events).toEqual([
{ type: "text-delta", id: "msg_1", text: "Hello", providerMetadata: { openai: { itemId: "msg_1" } } },
{ type: "text-delta", id: "msg_1", text: "!", providerMetadata: { openai: { itemId: "msg_1" } } },
{ type: "text-delta", id: "msg_1", text: "Hello" },
{ type: "text-delta", id: "msg_1", text: "!" },
{
type: "request-finish",
reason: "stop",
providerMetadata: { openai: { responseId: "resp_1", serviceTier: "default" } },
usage: {
usage: new Usage({
inputTokens: 5,
outputTokens: 2,
reasoningTokens: 0,
nonCachedInputTokens: 4,
cacheReadInputTokens: 1,
reasoningTokens: 0,
totalTokens: 7,
native: {
input_tokens: 5,
output_tokens: 2,
total_tokens: 7,
input_tokens_details: { cached_tokens: 1 },
output_tokens_details: { reasoning_tokens: 0 },
providerMetadata: {
openai: {
input_tokens: 5,
output_tokens: 2,
total_tokens: 7,
input_tokens_details: { cached_tokens: 1 },
output_tokens_details: { reasoning_tokens: 0 },
},
},
},
}),
},
])
}),
@@ -394,14 +397,12 @@ describe("OpenAI Responses route", () => {
id: "call_1",
name: "lookup",
text: '{"query"',
providerMetadata: { openai: { itemId: "item_1" } },
},
{
type: "tool-input-delta",
id: "call_1",
name: "lookup",
text: ':"weather"}',
providerMetadata: { openai: { itemId: "item_1" } },
},
{
type: "tool-call",
@@ -413,7 +414,13 @@ describe("OpenAI Responses route", () => {
{
type: "request-finish",
reason: "tool-calls",
usage: { inputTokens: 5, outputTokens: 1, totalTokens: 6, native: { input_tokens: 5, output_tokens: 1 } },
usage: new Usage({
inputTokens: 5,
outputTokens: 1,
nonCachedInputTokens: 5,
totalTokens: 6,
providerMetadata: { openai: { input_tokens: 5, output_tokens: 1 } },
}),
},
])
}),
+1 -1
View File
@@ -53,7 +53,7 @@ export const recordedTests = (options: RecordedTestsOptions) =>
...metadata,
}
const mode = recorderOptions?.mode ?? (recording ? "record" : "replay")
const cassetteService = HttpRecorder.Cassette.layer({ directory: FIXTURES_DIR }).pipe(
const cassetteService = HttpRecorder.Cassette.fileSystem({ directory: FIXTURES_DIR }).pipe(
Layer.provide(NodeFileSystem.layer),
)
const requestExecutor = RequestExecutor.layer.pipe(
+2 -3
View File
@@ -1,14 +1,13 @@
import { Cassette, makeWebSocketExecutor } from "@opencode-ai/http-recorder"
import { Cassette, makeWebSocketExecutor, type RecordReplayMode } from "@opencode-ai/http-recorder"
import { Effect, Layer } from "effect"
import { WebSocketExecutor } from "../src/route"
import type { Service as WebSocketExecutorService } from "../src/route/transport/websocket"
const liveWebSocket = WebSocketExecutor.open
type Mode = "record" | "replay" | "passthrough"
export const webSocketCassetteLayer = (
cassette: string,
input: { readonly metadata?: Record<string, unknown>; readonly mode: Mode },
input: { readonly metadata?: Record<string, unknown>; readonly mode: RecordReplayMode },
): Layer.Layer<WebSocketExecutorService, never, Cassette.Service> =>
Layer.effect(
WebSocketExecutor.Service,
+29 -1
View File
@@ -1,6 +1,7 @@
import { describe, expect, test } from "bun:test"
import { Schema } from "effect"
import { ContentPart, LLMEvent, LLMRequest, ModelID, ModelLimits, ModelRef, ProviderID } from "../src/schema"
import { ContentPart, LLMEvent, LLMRequest, ModelID, ModelLimits, ModelRef, ProviderID, Usage } from "../src/schema"
import { ProviderShared } from "../src/protocols/shared"
const model = new ModelRef({
id: ModelID.make("fake-model"),
@@ -48,3 +49,30 @@ describe("llm schema", () => {
expect(ContentPart.guards.media({ type: "text", text: "hi" })).toBe(false)
})
})
describe("LLM.Usage", () => {
test("subtractTokens clamps non-sensical breakdowns to zero", () => {
// Defense against a provider reporting cached_tokens > prompt_tokens or
// reasoning_tokens > completion_tokens — the negative would otherwise
// round-trip through the pipeline and crash strict downstream schemas.
expect(ProviderShared.subtractTokens(5, 3)).toBe(2)
expect(ProviderShared.subtractTokens(5, 10)).toBe(0)
expect(ProviderShared.subtractTokens(5, undefined)).toBe(5)
expect(ProviderShared.subtractTokens(undefined, 3)).toBeUndefined()
expect(ProviderShared.subtractTokens(undefined, undefined)).toBeUndefined()
})
test("sumTokens returns undefined only when every input is undefined", () => {
expect(ProviderShared.sumTokens(1, 2, 3)).toBe(6)
expect(ProviderShared.sumTokens(1, undefined, 3)).toBe(4)
expect(ProviderShared.sumTokens(undefined, undefined, undefined)).toBeUndefined()
expect(ProviderShared.sumTokens()).toBeUndefined()
})
test("visibleOutputTokens clamps reasoning > output to zero", () => {
expect(new Usage({ outputTokens: 10, reasoningTokens: 4 }).visibleOutputTokens).toBe(6)
expect(new Usage({ outputTokens: 10 }).visibleOutputTokens).toBe(10)
expect(new Usage({ outputTokens: 4, reasoningTokens: 10 }).visibleOutputTokens).toBe(0)
expect(new Usage({}).visibleOutputTokens).toBe(0)
})
})
@@ -21,6 +21,7 @@ import { TuiApi } from "./groups/tui"
import { WorkspaceApi } from "./groups/workspace"
import { V2Api } from "./groups/v2"
import { Authorization } from "./middleware/authorization"
import { SchemaErrorMiddleware } from "./middleware/schema-error"
// SSE event schemas built from the BusEvent/SyncEvent registries.
const EventSchema = Schema.Union(BusEvent.effectPayloads()).annotate({ identifier: "Event" })
@@ -29,6 +30,7 @@ const SyncEventSchemas = SyncEvent.effectPayloads()
export const RootHttpApi = HttpApi.make("opencode-root")
.addHttpApi(ControlApi)
.addHttpApi(GlobalApi)
.middleware(SchemaErrorMiddleware)
.middleware(Authorization)
export const InstanceHttpApi = HttpApi.make("opencode-instance")
@@ -47,6 +49,7 @@ export const InstanceHttpApi = HttpApi.make("opencode-instance")
.addHttpApi(V2Api)
.addHttpApi(TuiApi)
.addHttpApi(WorkspaceApi)
.middleware(SchemaErrorMiddleware)
export const OpenCodeHttpApi = HttpApi.make("opencode")
.addHttpApi(RootHttpApi)
@@ -0,0 +1,30 @@
import { Effect } from "effect"
import { HttpServerResponse } from "effect/unstable/http"
import { HttpApiMiddleware } from "effect/unstable/httpapi"
import * as Log from "@opencode-ai/core/util/log"
const log = Log.create({ service: "server" })
// Effect's Issue formatter recursively dumps the rejected `actual` value with
// no truncation, so a 5KB invalid array produces a ~360KB string. Cap to keep
// 4xx responses small and avoid mirroring entire request payloads (which may
// contain secrets) into the response body and log file.
const REASON_LIMIT = 1024
function truncateReason(reason: string) {
if (reason.length <= REASON_LIMIT) return reason
return reason.slice(0, REASON_LIMIT) + `… (${reason.length - REASON_LIMIT} more chars)`
}
// Default Respondable returns an empty 400 body. Match the NamedError shape
// used by other 4xx/5xx so the SDK's `wrapClientError` extracts `.data.message`.
export class SchemaErrorMiddleware extends HttpApiMiddleware.Service<SchemaErrorMiddleware>()(
"@opencode/HttpApiSchemaError",
) {}
export const schemaErrorLayer = HttpApiMiddleware.layerSchemaErrorTransform(SchemaErrorMiddleware, (error) => {
const reason = truncateReason(error.cause.message)
log.warn("schema rejection", { kind: error.kind, reason })
return Effect.succeed(
HttpServerResponse.jsonUnsafe({ name: "BadRequest", data: { message: reason, kind: error.kind } }, { status: 400 }),
)
})
@@ -178,17 +178,20 @@ function addLegacyErrorSchemas(spec: OpenApiSpec) {
if (!spec.components?.schemas) return
spec.components.schemas.BadRequestError = {
type: "object",
required: ["data", "errors", "success"],
required: ["name", "data"],
properties: {
data: {},
errors: {
type: "array",
items: {
type: "object",
additionalProperties: {},
name: { type: "string", enum: ["BadRequest"] },
data: {
type: "object",
required: ["message"],
properties: {
message: { type: "string" },
kind: {
type: "string",
enum: ["Params", "Headers", "Query", "Body", "Payload"],
},
},
},
success: { type: "boolean", enum: [false] },
},
}
spec.components.schemas.NotFoundError = {
@@ -84,6 +84,7 @@ import { compressionLayer } from "./middleware/compression"
import { corsVaryFix } from "./middleware/cors-vary"
import { errorLayer } from "./middleware/error"
import { fenceLayer } from "./middleware/fence"
import { schemaErrorLayer } from "./middleware/schema-error"
export const context = Context.makeUnsafe<unknown>(new Map())
@@ -114,6 +115,7 @@ const authOnlyRouterLayer = authorizationRouterMiddleware.layer.pipe(Layer.provi
const httpApiAuthLayer = authorizationLayer.pipe(Layer.provide(ServerAuth.Config.defaultLayer))
const rootApiRoutes = HttpApiBuilder.layer(RootHttpApi).pipe(
Layer.provide([controlHandlers, globalHandlers]),
Layer.provide(schemaErrorLayer),
Layer.provide(httpApiAuthLayer),
)
const instanceRouterLayer = authorizationRouterMiddleware
@@ -150,6 +152,7 @@ const instanceRoutes = Layer.mergeAll(rawInstanceRoutes, instanceApiRoutes).pipe
httpApiAuthLayer,
workspaceRoutingLayer.pipe(Layer.provide(Socket.layerWebSocketConstructorGlobal)),
instanceContextLayer,
schemaErrorLayer,
]),
)
@@ -18,6 +18,7 @@ import * as DateTime from "effect/DateTime"
import { InstanceState } from "@/effect/instance-state"
import { isOverflow as overflow, usable } from "./overflow"
import { makeRuntime } from "@/effect/run-service"
import { serviceUse } from "@/effect/service-use"
import { fn } from "@/util/fn"
import { EventV2 } from "@/v2/event"
import { SessionEvent } from "@/v2/session-event"
@@ -208,6 +209,8 @@ export interface Interface {
export class Service extends Context.Service<Service, Interface>()("@opencode/SessionCompaction") {}
export const use = serviceUse(Service)
export const layer: Layer.Layer<
Service,
never,
@@ -0,0 +1,162 @@
import { afterEach, describe, expect } from "bun:test"
import { Effect } from "effect"
import { eq } from "drizzle-orm"
import * as Database from "@/storage/db"
import { ModelID, ProviderID } from "../../src/provider/schema"
import { WithInstance } from "../../src/project/with-instance"
import { Server } from "../../src/server/server"
import { Session } from "@/session/session"
import { SessionPaths } from "../../src/server/routes/instance/httpapi/groups/session"
import { SyncPaths } from "../../src/server/routes/instance/httpapi/groups/sync"
import { MessageID, PartID } from "../../src/session/schema"
import { PartTable } from "@/session/session.sql"
import { resetDatabase } from "../fixture/db"
import { disposeAllInstances, tmpdir } from "../fixture/fixture"
import { it } from "../lib/effect"
afterEach(async () => {
await disposeAllInstances()
await resetDatabase()
})
const withTmp = <A, E, R>(
options: Parameters<typeof tmpdir>[0],
fn: (tmp: Awaited<ReturnType<typeof tmpdir>>) => Effect.Effect<A, E, R>,
) =>
Effect.acquireRelease(
Effect.promise(() => tmpdir(options)),
(tmp) => Effect.promise(() => tmp[Symbol.asyncDispose]()),
).pipe(Effect.flatMap(fn))
async function seedCorruptStepFinishPart(directory: string) {
return WithInstance.provide({
directory,
fn: () =>
Effect.runPromise(
Effect.gen(function* () {
const session = yield* Session.Service
const info = yield* session.create({})
const message = yield* session.updateMessage({
id: MessageID.ascending(),
role: "user",
sessionID: info.id,
agent: "build",
model: { providerID: ProviderID.make("test"), modelID: ModelID.make("test") },
time: { created: Date.now() },
})
const partID = PartID.ascending()
yield* session.updatePart({
id: partID,
sessionID: info.id,
messageID: message.id,
type: "step-finish",
reason: "stop",
cost: 0,
tokens: { input: 0, output: 0, reasoning: 0, cache: { read: 0, write: 0 } },
})
// Schema.Finite still rejects NaN at encode — exact mirror of the
// corrupt row that broke the user's session in the OMO/Windows bug.
Database.use((db) =>
db
.update(PartTable)
.set({
data: {
type: "step-finish",
reason: "stop",
cost: 0,
tokens: { input: 0, output: NaN, reasoning: 0, cache: { read: 0, write: 0 } },
} as never, // drizzle's .set() can't narrow the discriminated union
})
.where(eq(PartTable.id, partID))
.run(),
)
return info.id
}).pipe(Effect.provide(Session.defaultLayer)),
),
})
}
describe("schema-rejection wire shape", () => {
it.live(
"Payload schema rejection returns NamedError-shaped JSON, not empty",
withTmp({ git: true, config: { formatter: false, lsp: false } }, (tmp) =>
Effect.gen(function* () {
const res = yield* Effect.promise(async () =>
Server.Default().app.request(SyncPaths.history, {
method: "POST",
headers: { "x-opencode-directory": tmp.path, "content-type": "application/json" },
body: JSON.stringify({ aggregate: -1 }),
}),
)
const body = yield* Effect.promise(async () => res.text())
expect(res.status).toBe(400)
expect(res.headers.get("content-type") ?? "").toContain("application/json")
const parsed = JSON.parse(body)
expect(parsed).toMatchObject({
name: "BadRequest",
data: { kind: expect.stringMatching(/^(Body|Payload)$/) },
})
expect(parsed.data.message).toEqual(expect.any(String))
expect(parsed.data.message.length).toBeGreaterThan(0)
}),
),
)
it.live(
"Query schema rejection returns NamedError-shaped JSON",
withTmp({ git: true, config: { formatter: false, lsp: false } }, (tmp) =>
Effect.gen(function* () {
// /find/file?limit=999999 violates the limit constraint check.
const url = `/find/file?query=foo&limit=999999&directory=${encodeURIComponent(tmp.path)}`
const res = yield* Effect.promise(async () => Server.Default().app.request(url))
const body = yield* Effect.promise(async () => res.text())
expect(res.status).toBe(400)
const parsed = JSON.parse(body)
expect(parsed).toMatchObject({ name: "BadRequest", data: { kind: "Query" } })
}),
),
)
it.live(
"rejected request body never echoes back unbounded — message is capped",
// Defense against DoS-amplification + secret-echo: Effect's Issue formatter
// dumps the rejected `actual` verbatim. A multi-MB invalid array would
// become a multi-MB 400 response and log line. Cap kicks in around 1KB.
withTmp({ git: true, config: { formatter: false, lsp: false } }, (tmp) =>
Effect.gen(function* () {
const huge = "X".repeat(50_000)
const res = yield* Effect.promise(async () =>
Server.Default().app.request(SyncPaths.history, {
method: "POST",
headers: { "x-opencode-directory": tmp.path, "content-type": "application/json" },
body: JSON.stringify({ aggregate: huge }),
}),
)
const body = yield* Effect.promise(async () => res.text())
expect(res.status).toBe(400)
// 1 KB cap + small JSON envelope ≈ <2 KB — never tens of KB.
expect(body.length).toBeLessThan(2 * 1024)
const parsed = JSON.parse(body)
expect(parsed.data.message).not.toContain(huge)
}),
),
)
it.live(
"response-encode failure: corrupted stored row returns NamedError-shaped JSON with field path",
withTmp({ config: { formatter: false, lsp: false } }, (tmp) =>
Effect.gen(function* () {
const sessionID = yield* Effect.promise(() => seedCorruptStepFinishPart(tmp.path))
const url = `${SessionPaths.messages.replace(":sessionID", sessionID)}?limit=80&directory=${encodeURIComponent(tmp.path)}`
const res = yield* Effect.promise(async () => Server.Default().app.request(url))
const body = yield* Effect.promise(async () => res.text())
expect(res.status).toBe(400)
expect(res.headers.get("content-type") ?? "").toContain("application/json")
const parsed = JSON.parse(body)
expect(parsed).toMatchObject({ name: "BadRequest", data: { kind: "Body" } })
// Field path in data.message — what made this PR worth shipping.
expect(parsed.data.message).toMatch(/output/)
}),
),
)
})
@@ -52,23 +52,33 @@ describe("v2 SDK error shape", () => {
})
})
test("400 with empty body throws a real Error naming the status", async () => {
test("400 schema rejection: SDK extracts the field-level reason from the NamedError body", async () => {
// Canary for the #26631 wire shape. Asserts the contract end-to-end:
// server emits {name:"BadRequest", data:{message, kind}}, SDK's
// wrapClientError extracts .data.message into Error.message. If either
// side regresses (#26457 reverted because both layers were missing),
// this test fails before users see (empty response body).
await using tmp = await tmpdir({ config: { formatter: false, lsp: false } })
const sdk = client(tmp.path)
let caught: unknown
try {
// POST /sync/history with `aggregate: -1` triggers schema validation
// that returns an empty 400 body (verified via plan-mode probe).
await sdk.sync.history.list({ aggregate: -1 } as any, { throwOnError: true })
await sdk.sync.history.list({ body: { aggregate: -1 } as any }, { throwOnError: true })
} catch (e) {
caught = e
}
expect(caught).toBeInstanceOf(Error)
const err = caught as Error
const cause = err.cause as { status?: number }
expect(err.message.length).toBeGreaterThan(0)
const cause = err.cause as { body?: any; status?: number }
expect(cause.status).toBe(400)
expect(cause.body).toMatchObject({
name: "BadRequest",
data: { kind: expect.stringMatching(/^(Body|Payload)$/) },
})
expect(typeof cause.body.data.message).toBe("string")
expect(cause.body.data.message.length).toBeGreaterThan(0)
// Whatever the server put in data.message must be what the user sees.
expect(err.message).toBe(cause.body.data.message)
})
})
@@ -0,0 +1,60 @@
// Smoke test: v1 SDK (the plugin contract) can actually reach core endpoints
// against the current server. v1 generation has been frozen since #5216
// (2025-12-07) so types may be stale, but runtime calls should still work
// for endpoints the v1 SDK was generated against.
import { afterEach, describe, expect, test } from "bun:test"
import { createOpencodeClient } from "@opencode-ai/sdk"
import { Server } from "../../src/server/server"
import { tmpdir, disposeAllInstances } from "../fixture/fixture"
import { resetDatabase } from "../fixture/db"
import * as Log from "@opencode-ai/core/util/log"
void Log.init({ print: false })
afterEach(async () => {
await disposeAllInstances()
await resetDatabase()
})
function client(directory: string) {
return createOpencodeClient({
baseUrl: "http://test",
directory,
fetch: ((req: Request) => Server.Default().app.fetch(req)) as unknown as typeof fetch,
})
}
describe("v1 SDK runtime smoke", () => {
test("session.list reaches the server and returns 200", async () => {
await using tmp = await tmpdir({ git: true, config: { formatter: false, lsp: false } })
const sdk = client(tmp.path)
const result = await sdk.session.list()
expect(result.error).toBeUndefined()
expect(Array.isArray(result.data)).toBe(true)
})
test("path.get reaches the server and returns 200", async () => {
await using tmp = await tmpdir({ git: true, config: { formatter: false, lsp: false } })
const sdk = client(tmp.path)
const result = await sdk.path.get()
expect(result.error).toBeUndefined()
expect(result.data).toBeDefined()
})
test("config.get reaches the server and returns 200", async () => {
await using tmp = await tmpdir({ git: true, config: { formatter: false, lsp: false } })
const sdk = client(tmp.path)
const result = await sdk.config.get()
expect(result.error).toBeUndefined()
expect(result.data).toBeDefined()
})
test("session 404: result-tuple path returns the error body", async () => {
await using tmp = await tmpdir({ git: true, config: { formatter: false, lsp: false } })
const sdk = client(tmp.path)
const result = await sdk.session.get({ path: { id: "ses_no_such" } as never })
expect(result.error).toBeDefined()
// wire body for 404 is NamedError-shaped
expect(result.error).toMatchObject({ name: "NotFoundError" })
})
})
+133 -116
View File
@@ -1,6 +1,6 @@
import { afterEach, describe, expect, mock, test } from "bun:test"
import { APICallError } from "ai"
import { Cause, Effect, Exit, Layer, ManagedRuntime } from "effect"
import { Cause, Deferred, Effect, Exit, Layer, ManagedRuntime } from "effect"
import * as Stream from "effect/Stream"
import z from "zod"
import { Bus } from "../../src/bus"
@@ -15,7 +15,7 @@ import { WithInstance } from "../../src/project/with-instance"
import * as Log from "@opencode-ai/core/util/log"
import { Permission } from "../../src/permission"
import { Plugin } from "../../src/plugin"
import { provideTmpdirInstance, tmpdir } from "../fixture/fixture"
import { provideTmpdirInstance, TestInstance, tmpdir } from "../fixture/fixture"
import { Session as SessionNs } from "@/session/session"
import { MessageV2 } from "../../src/session/message-v2"
import { MessageID, PartID, SessionID } from "../../src/session/schema"
@@ -147,6 +147,53 @@ async function assistant(sessionID: SessionID, parentID: MessageID, root: string
return msg
}
function createUserMessage(sessionID: SessionID, text: string) {
return Effect.gen(function* () {
const ssn = yield* SessionNs.Service
const msg = yield* ssn.updateMessage({
id: MessageID.ascending(),
role: "user",
sessionID,
agent: "build",
model: ref,
time: { created: Date.now() },
})
yield* ssn.updatePart({
id: PartID.ascending(),
messageID: msg.id,
sessionID,
type: "text",
text,
})
return msg
})
}
function createAssistantMessage(sessionID: SessionID, parentID: MessageID, root: string) {
return SessionNs.Service.use((ssn) =>
ssn.updateMessage({
id: MessageID.ascending(),
role: "assistant",
sessionID,
mode: "build",
agent: "build",
path: { cwd: root, root },
cost: 0,
tokens: {
output: 0,
input: 0,
reasoning: 0,
cache: { read: 0, write: 0 },
},
modelID: ref.modelID,
providerID: ref.providerID,
parentID,
time: { created: Date.now() },
finish: "end_turn",
}),
)
}
async function summaryAssistant(sessionID: SessionID, parentID: MessageID, root: string, text: string) {
const msg: MessageV2.Assistant = {
id: MessageID.ascending(),
@@ -238,7 +285,7 @@ function runtime(
}
const deps = Layer.mergeAll(
ProviderTest.fake().layer,
wide().layer,
layer("continue"),
Agent.defaultLayer,
Plugin.defaultLayer,
@@ -805,85 +852,65 @@ describe("session.compaction.prune", () => {
})
describe("session.compaction.process", () => {
test("throws when parent is not a user message", async () => {
await using tmp = await tmpdir()
await WithInstance.provide({
directory: tmp.path,
fn: async () => {
const session = await svc.create({})
const msg = await user(session.id, "hello")
const reply = await assistant(session.id, msg.id, tmp.path)
const rt = runtime("continue")
try {
const msgs = await svc.messages({ sessionID: session.id })
await expect(
rt.runPromise(
SessionCompaction.Service.use((svc) =>
svc.process({
parentID: reply.id,
messages: msgs,
sessionID: session.id,
auto: false,
}),
),
),
).rejects.toThrow(`Compaction parent must be a user message: ${reply.id}`)
} finally {
await rt.dispose()
it.instance(
"throws when parent is not a user message",
Effect.gen(function* () {
const test = yield* TestInstance
const ssn = yield* SessionNs.Service
const session = yield* ssn.create({})
const msg = yield* createUserMessage(session.id, "hello")
const reply = yield* createAssistantMessage(session.id, msg.id, test.directory)
const msgs = yield* ssn.messages({ sessionID: session.id })
const exit = yield* Effect.exit(
SessionCompaction.use.process({
parentID: reply.id,
messages: msgs,
sessionID: session.id,
auto: false,
}),
)
expect(Exit.isFailure(exit)).toBe(true)
if (Exit.isFailure(exit)) {
const error = Cause.squash(exit.cause)
expect(error).toBeInstanceOf(Error)
if (error instanceof Error) {
expect(error.message).toContain(`Compaction parent must be a user message: ${reply.id}`)
}
},
})
})
}
}),
)
test("publishes compacted event on continue", async () => {
await using tmp = await tmpdir()
await WithInstance.provide({
directory: tmp.path,
fn: async () => {
const session = await svc.create({})
const msg = await user(session.id, "hello")
const msgs = await svc.messages({ sessionID: session.id })
const done = defer()
let seen = false
const rt = runtime("continue", Plugin.defaultLayer, wide())
let unsub: (() => void) | undefined
try {
unsub = await rt.runPromise(
Bus.Service.use((svc) =>
svc.subscribeCallback(SessionCompaction.Event.Compacted, (evt) => {
if (evt.properties.sessionID !== session.id) return
seen = true
done.resolve()
}),
),
)
it.instance(
"publishes compacted event on continue",
Effect.gen(function* () {
const bus = yield* Bus.Service
const ssn = yield* SessionNs.Service
const session = yield* ssn.create({})
const msg = yield* createUserMessage(session.id, "hello")
const msgs = yield* ssn.messages({ sessionID: session.id })
const done = yield* Deferred.make<void, Error>()
let seen = false
const unsub = yield* bus.subscribeCallback(SessionCompaction.Event.Compacted, (evt) => {
if (evt.properties.sessionID !== session.id) return
seen = true
Deferred.doneUnsafe(done, Effect.void)
})
yield* Effect.addFinalizer(() => Effect.sync(unsub))
const result = await rt.runPromise(
SessionCompaction.Service.use((svc) =>
svc.process({
parentID: msg.id,
messages: msgs,
sessionID: session.id,
auto: false,
}),
),
)
const result = yield* SessionCompaction.use.process({
parentID: msg.id,
messages: msgs,
sessionID: session.id,
auto: false,
})
await Promise.race([
done.promise,
wait(500).then(() => {
throw new Error("timed out waiting for compacted event")
}),
])
expect(result).toBe("continue")
expect(seen).toBe(true)
} finally {
unsub?.()
await rt.dispose()
}
},
})
})
yield* Deferred.await(done).pipe(Effect.timeout("500 millis"))
expect(result).toBe("continue")
expect(seen).toBe(true)
}),
)
test("marks summary message as errored on compact result", async () => {
await using tmp = await tmpdir()
@@ -923,46 +950,36 @@ describe("session.compaction.process", () => {
})
})
test("adds synthetic continue prompt when auto is enabled", async () => {
await using tmp = await tmpdir()
await WithInstance.provide({
directory: tmp.path,
fn: async () => {
const session = await svc.create({})
const msg = await user(session.id, "hello")
const rt = runtime("continue", Plugin.defaultLayer, wide())
try {
const msgs = await svc.messages({ sessionID: session.id })
const result = await rt.runPromise(
SessionCompaction.Service.use((svc) =>
svc.process({
parentID: msg.id,
messages: msgs,
sessionID: session.id,
auto: true,
}),
),
)
it.instance(
"adds synthetic continue prompt when auto is enabled",
Effect.gen(function* () {
const ssn = yield* SessionNs.Service
const session = yield* ssn.create({})
const msg = yield* createUserMessage(session.id, "hello")
const msgs = yield* ssn.messages({ sessionID: session.id })
const all = await svc.messages({ sessionID: session.id })
const last = all.at(-1)
const result = yield* SessionCompaction.use.process({
parentID: msg.id,
messages: msgs,
sessionID: session.id,
auto: true,
})
expect(result).toBe("continue")
expect(last?.info.role).toBe("user")
expect(last?.parts[0]).toMatchObject({
type: "text",
synthetic: true,
metadata: { compaction_continue: true },
})
if (last?.parts[0]?.type === "text") {
expect(last.parts[0].text).toContain("Continue if you have next steps")
}
} finally {
await rt.dispose()
}
},
})
})
const all = yield* ssn.messages({ sessionID: session.id })
const last = all.at(-1)
expect(result).toBe("continue")
expect(last?.info.role).toBe("user")
expect(last?.parts[0]).toMatchObject({
type: "text",
synthetic: true,
metadata: { compaction_continue: true },
})
if (last?.parts[0]?.type === "text") {
expect(last.parts[0].text).toContain("Continue if you have next steps")
}
}),
)
test("persists tail_start_id for retained recent turns", async () => {
await using tmp = await tmpdir()
+5 -5
View File
@@ -752,11 +752,11 @@ export type Project = {
}
export type BadRequestError = {
data: unknown
errors: Array<{
[key: string]: unknown
}>
success: false
name: "BadRequest"
data: {
message: string
kind?: "Params" | "Headers" | "Query" | "Body" | "Payload"
}
}
export type NotFoundError = {
+5 -5
View File
@@ -3296,11 +3296,11 @@ export type EventTuiToastShow1 = {
}
export type BadRequestError = {
data: unknown
errors: Array<{
[key: string]: unknown
}>
success: false
name: "BadRequest"
data: {
message: string
kind?: "Params" | "Headers" | "Query" | "Body" | "Payload"
}
}
export type AuthRemoveData = {
+16 -11
View File
@@ -18725,19 +18725,24 @@
},
"BadRequestError": {
"type": "object",
"required": ["data", "errors", "success"],
"required": ["name", "data"],
"properties": {
"data": {},
"errors": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": {}
}
"name": {
"type": "string",
"enum": ["BadRequest"]
},
"success": {
"type": "boolean",
"enum": [false]
"data": {
"type": "object",
"required": ["message"],
"properties": {
"message": {
"type": "string"
},
"kind": {
"type": "string",
"enum": ["Params", "Headers", "Query", "Body", "Payload"]
}
}
}
}
}