llm provider and test renderer

This commit is contained in:
James Long
2026-05-17 14:34:50 -04:00
parent 012fc184bc
commit eab9a4f564
10 changed files with 409 additions and 42 deletions
@@ -0,0 +1,27 @@
# Property-Based TUI Testing Working Notes
## Mock LLM Provider
- `SessionPrompt` gets model metadata through `Provider.Service.getModel(...)`.
- Actual generation is routed through `LLM.Service` / `streamText(...)`, so the first mock should return an AI SDK `LanguageModelV3` from the provider service.
- The normal `Provider.layer` builds providers from config/models.dev/plugin state. For first pass, the simulated graph can replace `Provider.Service` with a smaller simulation provider service instead of trying to flow through provider config.
- Control state should own an ordered LLM script queue. The provider/model should consume from that queue when the AI SDK calls the language model.
- First version can support text-only output. Tool calls and stream chunk fidelity can come next.
- Missing script should fail loudly with a typed simulation error, not silently return an empty assistant message.
- Implemented `SimulationProvider.layer`, replacing `Provider.Service` in `createSimulatedRoutes`.
- The provider exposes provider `simulation` and model `mock`.
- `doGenerate` and `doStream` both consume one queued script through `Simulation.Service.nextLLM()`.
- Current script support: `text`, `thinking` (treated as text for now), and `error`.
- Snapshot currently records `llmQueued` and `llmConsumed`, not per-step details yet.
## OpenTUI Fake Renderer
- OpenTUI Solid exposes `testRender(...)` from `@opentui/solid`.
- The lower-level core API is `createTestRenderer(...)` from `@opentui/core/testing`.
- `createTestRenderer(...)` returns `renderer`, `mockInput`, `mockMouse`, `renderOnce`, `captureCharFrame`, `captureSpans`, and `resize`.
- `captureCharFrame()` is the simple screen-buffer string API used heavily in OpenTUI snapshots.
- `captureSpans()` returns structured lines/spans plus cursor position, which is a better starting point for visible element discovery than parsing raw characters.
- `mockInput` supports interactions like `typeText`, `pressEnter`, and `pressArrow`.
- Implemented `TuiSimulation.createSimulationRenderer(...)` beside `thread.ts`. It creates a test renderer and exposes `renderOnce`, `screen`, `spans`, and `destroy`.
- `thread.ts` checks `OPENCODE_SIMULATION`, creates the fake renderer there, starts the normal worker/backend, and passes the renderer into `tui(...)`.
- `tui(...)` now accepts an injected `CliRenderer`, test mode, and an `onReady` callback. Production still creates the real renderer.
@@ -199,7 +199,9 @@ Implementation shape:
- First pass uses a raw route wrapper at `packages/opencode/src/server/routes/instance/httpapi/simulation.ts` to avoid SDK regeneration while the API shape is still moving.
- Current control service can reset state, seed filesystem files, register static network responses, and return a snapshot.
- Register/configure a local mock provider/model through the normal provider path.
- The mock model reads scripts from simulation control state.
- The simulated route graph replaces `Provider.Service` with `SimulationProvider.layer`.
- The mock model reads queued scripts from simulation control state.
- Current mock provider supports text/thinking/error actions for the first step only. Tool calls and multi-round step selection are still pending.
- No JSON-in-prompt fallback.
- Missing script means typed simulation error.
@@ -236,10 +238,11 @@ Todos:
- [x] Add simulation control state and reset semantics.
- [x] Add gated simulation endpoints for reset, filesystem seed, network register, and snapshot.
- [x] Decide raw route vs typed HttpApi route. Raw route for first pass; no SDK regeneration yet.
- [ ] Implement mock provider/model on the normal provider path.
- [ ] Port the useful stream chunk behavior from the old branch to the current AI SDK interface.
- [ ] Make missing scripts fail with a typed simulation error.
- [ ] Record consumed script step in simulation snapshot.
- [x] Implement mock provider/model on the normal provider path.
- [x] Make missing scripts fail with a typed simulation error.
- [x] Record consumed script count in simulation snapshot.
- [ ] Support tool-call script actions.
- [ ] Support multi-step script selection after tool result rounds.
- [ ] Verify `session.prompt_async` exercises real `SessionPrompt` and `SessionProcessor`.
## OpenTUI Fake Renderer And Interactable Elements
@@ -255,6 +258,8 @@ Known starting points:
Implementation shape:
- Add a renderer factory/testing hook to `tui(...)` so tests can pass a fake renderer.
- Current first pass checks `OPENCODE_SIMULATION` in `cli/cmd/tui/thread.ts`, starts the normal worker/backend, and injects an OpenTUI test renderer into `tui(...)`.
- Fake renderer setup lives in `cli/cmd/tui/simulation.ts` and returns `renderOnce`, `screen`, and `spans` helpers for the thread-side simulation runner.
- Do not render to a real terminal in simulation mode.
- Investigate OpenTUI APIs for walking the render tree and extracting focusable/clickable/editable elements.
- Investigate OpenTUI APIs for reading the screen buffer from the fake renderer.
@@ -262,11 +267,11 @@ Implementation shape:
Todos:
- [ ] Inspect `@opentui/core/testing` `createTestRenderer` capabilities.
- [ ] Inspect `@opentui/solid` `testRender` capabilities.
- [ ] Determine how to get a screen buffer string/snapshot from the fake renderer.
- [ ] Determine how to iterate renderables and identify interactable elements.
- [ ] Add a minimal renderer factory override to `tui(...)` or app startup.
- [x] Inspect `@opentui/core/testing` `createTestRenderer` capabilities.
- [x] Inspect `@opentui/solid` `testRender` capabilities.
- [x] Determine how to get a screen buffer string/snapshot from the fake renderer.
- [x] Determine first structured capture API for interactable discovery: `captureSpans()`.
- [x] Add a minimal renderer factory override to `tui(...)` or app startup.
- [ ] Expose prompt ref, route, sync state, keymap, and renderer to the simulation harness.
- [ ] Verify TUI starts in fake renderer with no real terminal output.
- [ ] Verify screen buffer can be captured after a render.
+7 -3
View File
@@ -3,7 +3,7 @@ import { createDefaultOpenTuiKeymap } from "@opentui/keymap/opentui"
import * as Clipboard from "@tui/util/clipboard"
import * as Selection from "@tui/util/selection"
import * as TuiAudio from "@tui/util/audio"
import { createCliRenderer, MouseButton, type CliRendererConfig } from "@opentui/core"
import { createCliRenderer, MouseButton, type CliRenderer, type CliRendererConfig } from "@opentui/core"
import { RouteProvider, useRoute } from "@tui/context/route"
import {
Switch,
@@ -165,6 +165,9 @@ export function tui(input: {
fetch?: typeof fetch
headers?: RequestInit["headers"]
events?: EventSource
renderer?: CliRenderer
mode?: "dark" | "light"
onReady?: (ctx: { renderer: CliRenderer }) => void
}) {
// promise to prevent immediate exit
// oxlint-disable-next-line no-async-promise-executor -- intentional: async executor used for sequential setup before resolve
@@ -182,10 +185,10 @@ export function tui(input: {
TuiAudio.dispose()
}
const renderer = await createCliRenderer(rendererConfig(input.config))
const renderer = input.renderer ?? (await createCliRenderer(rendererConfig(input.config)))
// Prewarm palette before ThemeProvider mounts so `system` theme avoids a first-paint fallback flash.
void renderer.getPalette({ size: 16 }).catch(() => undefined)
const mode = (await renderer.waitForThemeMode(1000)) ?? "dark"
const mode = input.mode ?? (await renderer.waitForThemeMode(1000)) ?? "dark"
const keymap = createDefaultOpenTuiKeymap(renderer)
const offKeymap = registerOpencodeKeymap(keymap, renderer, input.config)
@@ -256,6 +259,7 @@ export function tui(input: {
</ErrorBoundary>
)
}, renderer)
input.onReady?.({ renderer })
})
}
@@ -0,0 +1,30 @@
import type { CliRenderer } from "@opentui/core"
import type { CapturedFrame } from "@opentui/core"
export interface SimulationRenderer {
readonly renderer: CliRenderer
readonly renderOnce: () => Promise<void>
readonly screen: () => string
readonly spans: () => CapturedFrame
readonly destroy: () => void
}
export async function createSimulationRenderer(): Promise<SimulationRenderer> {
const { createTestRenderer } = await import("@opentui/core/testing")
const setup = await createTestRenderer({
width: Number(process.env.OPENCODE_SIMULATION_TUI_WIDTH) || 100,
height: Number(process.env.OPENCODE_SIMULATION_TUI_HEIGHT) || 40,
screenMode: "main-screen",
consoleMode: "disabled",
})
return {
renderer: setup.renderer,
renderOnce: setup.renderOnce,
screen: setup.captureCharFrame,
spans: setup.captureSpans,
destroy: () => setup.renderer.destroy(),
}
}
export * as TuiSimulation from "./simulation"
+34 -20
View File
@@ -21,6 +21,8 @@ import {
sanitizedProcessEnv,
} from "@opencode-ai/core/util/opencode-process"
import { validateSession } from "./validate-session"
import { Flag } from "@opencode-ai/core/flag/flag"
import { TuiSimulation } from "./simulation"
declare global {
const OPENCODE_WORKER_PATH: string
@@ -229,26 +231,38 @@ export const TuiThreadCommand = cmd({
try {
const { tui } = await import("./app")
await tui({
url: transport.url,
async onSnapshot() {
const tui = writeHeapSnapshot("tui.heapsnapshot")
const server = await client.call("snapshot", undefined)
return [tui, server]
},
config,
directory: cwd,
fetch: transport.fetch,
events: transport.events,
args: {
continue: args.continue,
sessionID: args.session,
agent: args.agent,
model: args.model,
prompt,
fork: args.fork,
},
})
const simulationRenderer = Flag.OPENCODE_SIMULATION ? await TuiSimulation.createSimulationRenderer() : undefined
try {
await tui({
url: transport.url,
async onSnapshot() {
const tui = writeHeapSnapshot("tui.heapsnapshot")
const server = await client.call("snapshot", undefined)
return [tui, server]
},
config,
directory: cwd,
fetch: transport.fetch,
events: transport.events,
renderer: simulationRenderer?.renderer,
mode: simulationRenderer ? "dark" : undefined,
onReady: simulationRenderer
? async () => {
await simulationRenderer.renderOnce()
}
: undefined,
args: {
continue: args.continue,
sessionID: args.session,
agent: args.agent,
model: args.model,
prompt,
fork: args.fork,
},
})
} finally {
simulationRenderer?.destroy()
}
} finally {
await stop()
}
@@ -64,6 +64,7 @@ import { Workspace } from "@/control-plane/workspace"
import { SimulationFileSystem } from "@/testing/simulation/filesystem"
import { SimulationNetwork } from "@/testing/simulation/network"
import { SimulationNetworkRoutes } from "@/testing/simulation/network-routes"
import { SimulationProvider } from "@/testing/simulation/provider"
import { Simulation } from "@/testing/simulation/service"
import { CorsConfig, isAllowedCorsOrigin, type CorsOptions } from "@/server/cors"
import { serveUIEffect } from "@/server/shared/ui"
@@ -284,7 +285,7 @@ export function createSimulatedRoutes(corsOptions?: CorsOptions): ReturnType<typ
Plugin.layer,
Project.layer,
ProviderAuth.layer,
Provider.layer,
SimulationProvider.layer,
Pty.layer,
PtyTicket.layer,
Question.layer,
@@ -38,7 +38,12 @@ export const simulationRoute = HttpRouter.use((router) =>
)
yield* router.add("POST", "/experimental/simulation/llm/enqueue", () =>
json(Effect.succeed({ ok: false, skipped: "mock provider scripts are not implemented yet" })),
json(
Effect.gen(function* () {
const input = yield* HttpServerRequest.schemaBodyJson(Simulation.LLMEnqueueInput)
return yield* simulation.enqueueLLM(input)
}),
),
)
yield* router.add("GET", "/experimental/simulation/snapshot", () => json(simulation.snapshot()))
@@ -0,0 +1,142 @@
import type { LanguageModelV3, LanguageModelV3CallOptions, LanguageModelV3FinishReason } from "@ai-sdk/provider"
import { Effect, Layer } from "effect"
import { Provider } from "@/provider/provider"
import { ModelID, ProviderID } from "@/provider/schema"
import { Simulation, type LLMScript } from "./service"
const providerID = ProviderID.make("simulation")
const modelID = ModelID.make("mock")
const model: Provider.Model = {
id: modelID,
providerID,
api: { id: modelID, url: "simulation://mock", npm: "simulation" },
name: "Simulation Mock",
capabilities: {
temperature: true,
reasoning: true,
attachment: false,
toolcall: false,
input: { text: true, audio: false, image: false, video: false, pdf: false },
output: { text: true, audio: false, image: false, video: false, pdf: false },
interleaved: false,
},
cost: { input: 0, output: 0, cache: { read: 0, write: 0 } },
limit: { context: 128_000, output: 32_000 },
status: "active",
options: {},
headers: {},
release_date: "2026-01-01",
variants: {},
}
const provider: Provider.Info = {
id: providerID,
name: "Simulation",
source: "custom",
env: [],
options: {},
models: { [modelID]: model },
}
function text(script: LLMScript) {
return script.steps[0]?.flatMap((item) => (item.type === "text" || item.type === "thinking" ? [item.content] : []))
.join("") ?? ""
}
function error(script: LLMScript) {
return script.steps[0]?.find((item) => item.type === "error")
}
function stream(script: LLMScript) {
return new ReadableStream({
start(controller) {
controller.enqueue({ type: "stream-start", warnings: [] })
let index = 0
for (const item of script.steps[0] ?? []) {
index++
if (item.type === "error") {
controller.enqueue({ type: "error", error: new Error(item.message) })
controller.close()
return
}
const id = `simulation-${item.type}-${index}`
if (item.type === "thinking") {
controller.enqueue({ type: "reasoning-start", id })
controller.enqueue({ type: "reasoning-delta", id, delta: item.content })
controller.enqueue({ type: "reasoning-end", id })
continue
}
controller.enqueue({ type: "text-start", id })
controller.enqueue({ type: "text-delta", id, delta: item.content })
controller.enqueue({ type: "text-end", id })
}
controller.enqueue({ type: "finish", finishReason: finishReason(script), usage: usage(script) })
controller.close()
},
})
}
function usage(script: LLMScript) {
return {
inputTokens: {
total: script.usage?.inputTokens ?? 0,
noCache: script.usage?.inputTokens ?? 0,
cacheRead: undefined,
cacheWrite: undefined,
},
outputTokens: {
total: script.usage?.outputTokens ?? text(script).length,
text: script.usage?.outputTokens ?? text(script).length,
reasoning: undefined,
},
raw: script.usage,
}
}
function finishReason(script: LLMScript): LanguageModelV3FinishReason {
return { unified: script.finish === "unknown" ? "other" : (script.finish ?? "stop"), raw: script.finish }
}
function language(simulation: Simulation.Interface): LanguageModelV3 {
return {
specificationVersion: "v3",
provider: "simulation",
modelId: modelID,
supportedUrls: {},
async doGenerate(_options: LanguageModelV3CallOptions) {
const script = await Effect.runPromise(simulation.nextLLM())
const err = error(script)
if (err?.type === "error") throw new Error(err.message)
return {
content: [{ type: "text", text: text(script) }],
finishReason: finishReason(script),
usage: usage(script),
warnings: [],
}
},
async doStream(_options: LanguageModelV3CallOptions) {
const script = await Effect.runPromise(simulation.nextLLM())
return { stream: stream(script) }
},
}
}
export const layer = Layer.effect(
Provider.Service,
Effect.gen(function* () {
const simulation = yield* Simulation.Service
const lang = language(simulation)
return Provider.Service.of({
list: () => Effect.succeed({ [providerID]: provider }),
getProvider: () => Effect.succeed(provider),
getModel: () => Effect.succeed(model),
getLanguage: () => Effect.succeed(lang),
closest: () => Effect.succeed({ providerID, modelID }),
getSmallModel: () => Effect.succeed(model),
defaultModel: () => Effect.succeed({ providerID, modelID }),
})
}),
)
export * as SimulationProvider from "./provider"
@@ -38,20 +38,56 @@ export const NetworkRegisterInput = Schema.Union([
}),
])
export const LLMScriptAction = Schema.Union([
Schema.Struct({ type: Schema.Literal("text"), content: Schema.String }),
Schema.Struct({ type: Schema.Literal("thinking"), content: Schema.String }),
Schema.Struct({ type: Schema.Literal("error"), message: Schema.String }),
])
export type LLMScriptAction = typeof LLMScriptAction.Type
export const LLMScript = Schema.Struct({
steps: Schema.Array(Schema.Array(LLMScriptAction)),
usage: Schema.optional(
Schema.Struct({
inputTokens: Schema.Number,
outputTokens: Schema.Number,
totalTokens: Schema.Number,
}),
),
finish: Schema.optional(Schema.Literals(["stop", "tool-calls", "error", "length", "unknown"])),
})
export type LLMScript = typeof LLMScript.Type
export const LLMEnqueueInput = Schema.Struct({
scripts: Schema.Array(LLMScript),
})
type FilePath = string
interface State {
readonly files: readonly FilePath[]
readonly networkRegistrations: readonly string[]
readonly llmScripts: readonly LLMScript[]
readonly consumedLLMScripts: number
}
export class SimulationLLMError extends Schema.TaggedErrorClass<SimulationLLMError>()("SimulationLLMError", {
message: Schema.String,
}) {}
export interface Interface {
readonly reset: () => Effect.Effect<void>
readonly seedFilesystem: (input: typeof FilesystemSeedInput.Type) => Effect.Effect<{ files: string[] }, unknown>
readonly registerNetwork: (input: typeof NetworkRegisterInput.Type) => Effect.Effect<{ registered: string }, unknown>
readonly enqueueLLM: (input: typeof LLMEnqueueInput.Type) => Effect.Effect<{ queued: number }>
readonly nextLLM: () => Effect.Effect<LLMScript, SimulationLLMError>
readonly snapshot: () => Effect.Effect<{
files: readonly string[]
networkRegistrations: readonly string[]
llmQueued: number
llmConsumed: number
network: SimulationNetwork.Snapshot
}>
}
@@ -72,11 +108,11 @@ export const layer = Layer.effect(
Effect.gen(function* () {
const fs = yield* AppFileSystem.Service
const network = yield* SimulationNetwork.Service
const state = yield* Ref.make<State>({ files: [], networkRegistrations: [] })
const state = yield* Ref.make<State>({ files: [], networkRegistrations: [], llmScripts: [], consumedLLMScripts: 0 })
const reset = Effect.fn("Simulation.reset")(function* () {
yield* network.reset()
yield* Ref.set(state, { files: [], networkRegistrations: [] })
yield* Ref.set(state, { files: [], networkRegistrations: [], llmScripts: [], consumedLLMScripts: 0 })
})
const seedFilesystem = Effect.fn("Simulation.seedFilesystem")(function* (input: typeof FilesystemSeedInput.Type) {
@@ -112,12 +148,31 @@ export const layer = Layer.effect(
return { registered: input.url }
})
const snapshot = Effect.fn("Simulation.snapshot")(function* () {
const current = yield* Ref.get(state)
return { ...current, network: yield* network.snapshot() }
const enqueueLLM = Effect.fn("Simulation.enqueueLLM")(function* (input: typeof LLMEnqueueInput.Type) {
yield* Ref.update(state, (current) => ({ ...current, llmScripts: [...current.llmScripts, ...input.scripts] }))
return { queued: input.scripts.length }
})
return Service.of({ reset, seedFilesystem, registerNetwork, snapshot })
const nextLLM = Effect.fn("Simulation.nextLLM")(function* () {
const current = yield* Ref.get(state)
const [script, ...rest] = current.llmScripts
if (!script) return yield* new SimulationLLMError({ message: "No LLM script queued" })
yield* Ref.set(state, { ...current, llmScripts: rest, consumedLLMScripts: current.consumedLLMScripts + 1 })
return script
})
const snapshot = Effect.fn("Simulation.snapshot")(function* () {
const current = yield* Ref.get(state)
return {
files: current.files,
networkRegistrations: current.networkRegistrations,
llmQueued: current.llmScripts.length,
llmConsumed: current.consumedLLMScripts,
network: yield* network.snapshot(),
}
})
return Service.of({ reset, seedFilesystem, registerNetwork, enqueueLLM, nextLLM, snapshot })
}),
)
@@ -2,15 +2,18 @@ import { describe, expect } from "bun:test"
import { AppFileSystem } from "@opencode-ai/core/filesystem"
import { Effect, Layer } from "effect"
import { HttpClient, HttpClientRequest } from "effect/unstable/http"
import { Provider } from "../../../src/provider/provider"
import { SimulationFileSystem } from "../../../src/testing/simulation/filesystem"
import { SimulationNetwork } from "../../../src/testing/simulation/network"
import { SimulationProvider } from "../../../src/testing/simulation/provider"
import { Simulation } from "../../../src/testing/simulation/service"
import { testEffect } from "../../lib/effect"
const fsLayer = SimulationFileSystem.layer({ root: "/opencode" })
const networkLayer = SimulationNetwork.layer({ allowLoopback: false })
const simulationLayer = Simulation.layer.pipe(Layer.provide(fsLayer), Layer.provide(networkLayer))
const it = testEffect(Layer.mergeAll(fsLayer, networkLayer, simulationLayer))
const providerLayer = SimulationProvider.layer.pipe(Layer.provide(simulationLayer))
const it = testEffect(Layer.mergeAll(fsLayer, networkLayer, simulationLayer, providerLayer))
describe("Simulation", () => {
it.effect("seeds files into the simulated filesystem", () =>
@@ -64,4 +67,85 @@ describe("Simulation", () => {
expect(exit._tag).toBe("Failure")
}),
)
it.effect("queues and consumes LLM scripts", () =>
Effect.gen(function* () {
const simulation = yield* Simulation.Service
expect(
yield* simulation.enqueueLLM({
scripts: [{ steps: [[{ type: "text", content: "hello" }]], finish: "stop" }],
}),
).toEqual({ queued: 1 })
expect((yield* simulation.snapshot()).llmQueued).toBe(1)
expect(yield* simulation.nextLLM()).toEqual({ steps: [[{ type: "text", content: "hello" }]], finish: "stop" })
expect((yield* simulation.snapshot()).llmConsumed).toBe(1)
}),
)
it.effect("simulation provider consumes queued text scripts", () =>
Effect.gen(function* () {
const simulation = yield* Simulation.Service
const provider = yield* Provider.Service
const model = yield* provider.defaultModel().pipe(Effect.flatMap((item) => provider.getModel(item.providerID, item.modelID)))
const language = yield* provider.getLanguage(model)
yield* simulation.enqueueLLM({ scripts: [{ steps: [[{ type: "text", content: "assistant text" }]] }] })
const result = yield* Effect.promise(() => language.doGenerate({ prompt: [], abortSignal: undefined }))
expect(result.content).toEqual([{ type: "text", text: "assistant text" }])
expect((yield* simulation.snapshot()).llmConsumed).toBe(1)
}),
)
it.effect("simulation provider streams queued script actions", () =>
Effect.gen(function* () {
const simulation = yield* Simulation.Service
const provider = yield* Provider.Service
const model = yield* provider.defaultModel().pipe(Effect.flatMap((item) => provider.getModel(item.providerID, item.modelID)))
const language = yield* provider.getLanguage(model)
yield* simulation.enqueueLLM({
scripts: [
{
steps: [
[
{ type: "thinking", content: "thinking" },
{ type: "text", content: "answer" },
],
],
},
],
})
const result = yield* Effect.promise(() => language.doStream({ prompt: [], abortSignal: undefined }))
const reader = result.stream.getReader()
const parts: unknown[] = []
while (true) {
const next = yield* Effect.promise(() => reader.read())
if (next.done) break
parts.push(next.value)
}
expect(parts).toEqual([
{ type: "stream-start", warnings: [] },
{ type: "reasoning-start", id: "simulation-thinking-1" },
{ type: "reasoning-delta", id: "simulation-thinking-1", delta: "thinking" },
{ type: "reasoning-end", id: "simulation-thinking-1" },
{ type: "text-start", id: "simulation-text-2" },
{ type: "text-delta", id: "simulation-text-2", delta: "answer" },
{ type: "text-end", id: "simulation-text-2" },
{
type: "finish",
finishReason: { unified: "stop", raw: undefined },
usage: {
inputTokens: { total: 0, noCache: 0, cacheRead: undefined, cacheWrite: undefined },
outputTokens: { total: "thinkinganswer".length, text: "thinkinganswer".length, reasoning: undefined },
raw: undefined,
},
},
])
expect((yield* simulation.snapshot()).llmConsumed).toBe(1)
}),
)
})