Skip to content
Looms
Documentation

Evaluations

defineEvaluator is an evaluator kind: defineKind('evaluator', …) with typed questions and a route function. An evaluator is not a chat model. It answers boolean (with a probability), choice, or score questions, writes evaluator.evaluated on the run log, and lets a durable workflow branch on that result. TypeSafe Jev is one model that speaks this API. Later models plug in through the same definition and a different adapter.

Score, then branch#

Pass your own questions. Use skipModel when the input should not call the model, and route to turn answers into an application string such as auto or review. The reason is model, uncertain, skipped, or unavailable: confidence as control flow. The example scores a refund, skips large amounts, auto-pays a routine case, and spawns a stub agent when review is needed. It uses the built-in stub, so it needs no API key.

ts
import { z } from 'zod'import { agent, defineAgent } from '@swirls/looms/agent'import { defineEvaluator, evaluations, evaluator } from '@swirls/looms/evaluator'import { createLooms } from '@swirls/looms/runtime'import { defineWorkflow, workflow } from '@swirls/looms/workflow'// Score with defineEvaluator, branch on route/reason, then maybe spawn an agent.// The stub stands in for any evaluation model, including TypeSafe Jev.// bun run --filter @looms/examples evaluatorconst RefundInput = z.object({  amount: z.number(),  reason: z.string(),})const scoreRefund = defineEvaluator({  name: 'score-refund',  description: 'Cheap typed questions before an expensive review.',  input: RefundInput,  questions: {    needsReview: {      type: 'boolean',      instructions: 'Should a reasoning model or a human review this refund before it ships?',    },    urgency: {      type: 'choice',      instructions: 'How urgent is this refund?',      criteria: {        high: 'Customer-visible failure or dispute',        normal: 'Needs a look but is not blocking',        low: 'Routine, already resolved, or duplicate',      },    },    risk: {      type: 'score',      instructions: 'Score the risk of paying this refund incorrectly.',      criteria: ['None', 'Low', 'Moderate', 'High', 'Severe'],    },  },  skipModel: (input) => {    if (input.amount >= 500) {      return { route: 'review', reason: 'skipped' }    }    return undefined  },  route: ({ answers }) => {    const needs = answers.needsReview    const risk = answers.risk    const probability = needs?.type === 'boolean' ? needs.probability : null    const score = risk?.type === 'score' ? risk.value : null    if (probability === null || score === null || probability < 0 || probability > 1) {      return { route: 'review', reason: 'unavailable' }    }    const auto = probability < 0.1 && score <= 1    const confident = auto || probability >= 0.9    return {      route: auto ? 'auto' : 'review',      reason: confident ? 'model' : 'uncertain',    }  },})const reviewer = defineAgent({  name: 'refund-reviewer',  instructions: 'Draft a short refund review.',  input: RefundInput,  runTurn: ({ input }) => ({    message: { role: 'assistant', content: `Review refund of ${input.amount}: ${input.reason}` },    done: true,    output: { draft: `Review refund of ${input.amount}` },  }),})const refund = defineWorkflow({  name: 'refund',  input: RefundInput,  nodes: [    {      id: 'score',      run: (ctx) => ctx.spawn(scoreRefund, ctx.input),    },    {      id: 'next',      deps: ['score'],      run: (ctx) => {        const scored = z          .object({            route: z.string(),            reason: z.enum(['model', 'uncertain', 'skipped', 'unavailable']),          })          .safeParse(ctx.results.score)        const route = scored.success ? scored.data.route : 'review'        const reason = scored.success ? scored.data.reason : 'unavailable'        if (route === 'review' || reason === 'uncertain' || reason === 'unavailable') {          return ctx.spawn(reviewer, ctx.input)        }        return { path: 'auto', paid: true }      },    },  ],  output: ({ results }) => results.next ?? null,})const looms = createLooms({  modules: [    evaluator({ definitions: [scoreRefund] }),    agent({ definitions: [reviewer] }),    workflow({ definitions: [refund] }),  ],})const routine = await looms.start(refund, {  amount: 12,  reason: 'duplicate charge, already resolved',})console.log('routine:', {  status: routine.state.status,  output: routine.state.rootThreadId    ? routine.state.threads[routine.state.rootThreadId]?.output    : null,})const dispute = await looms.start(refund, {  amount: 40,  reason: 'failed capture, customer dispute',})console.log('dispute:', {  status: dispute.state.status,  output: dispute.state.rootThreadId    ? dispute.state.threads[dispute.state.rootThreadId]?.output    : null,})const large = await looms.start(refund, {  amount: 900,  reason: 'goodwill credit',})console.log('large:', {  status: large.state.status,  output: large.state.rootThreadId ? large.state.threads[large.state.rootThreadId]?.output : null,})console.log('evaluations:', {  routine: (await looms.project(routine.runId, evaluations)).items.map(    (item) => `${item.reason}:${item.route}`,  ),  dispute: (await looms.project(dispute.runId, evaluations)).items.map(    (item) => `${item.reason}:${item.route}`,  ),  large: (await looms.project(large.runId, evaluations)).items.map(    (item) => `${item.reason}:${item.route}`,  ),})await looms.stop()

Compose with workflows and agents#

Spawn the definition from a workflow node with ctx.spawn(scoreRefund, input), pass it to an agent as asEvaluatorTool, or call evaluate(scoreRefund, input) like gate() when you want an inline wait instead of a child thread. Register the definition on evaluator({ definitions: [scoreRefund] }).

Branch on route in deterministic workflow code. Treat uncertain and unavailable the same way you treat a review path unless you have a cheaper fallback. After a crash, the answers and decision are still on evaluator.evaluated.

Connect a model#

ts
import { vercelEvaluator } from '@swirls/looms/ai-vercel'import { defineEvaluator, evaluator } from '@swirls/looms/evaluator'const scoreRefund = defineEvaluator({  name: 'score-refund',  model: 'typesafe-ai/jev',  questions: {    needsReview: {      type: 'boolean',      instructions: 'Should this refund be reviewed before it ships?',    },  },  route: ({ answers }) => {    const probability =      answers.needsReview?.type === 'boolean' ? answers.needsReview.probability : null    return {      route: probability !== null && probability < 0.1 ? 'auto' : 'review',      reason:        probability !== null && (probability < 0.1 || probability >= 0.9)          ? 'model'          : 'uncertain',    }  },})evaluator({  definitions: [scoreRefund],  evaluator: vercelEvaluator({ model: 'typesafe-ai/jev' }),})

vercelEvaluator uses the Vercel AI SDK experimental_evaluate API. Pass typesafe-ai/jev today. A later model that answers the same boolean, choice, and score questions uses the same definition with a different model id. Keep AI_GATEWAY_API_KEY on the server. Without an adapter, the evaluator module uses a deterministic stub. See agents and model providers for chat adapters, and examples for the stubbed script.

Use your own TypeSafe key#

To call TypeSafe directly instead of through the AI Gateway, pass a provider. It resolves the adapter's model and any model set on a definition, so omit model from the definition or use a TypeSafe id such as jev-latest.

ts
import { createTypeSafeAi } from '@ai-sdk/typesafe-ai'import { vercelEvaluator } from '@swirls/looms/ai-vercel'import { evaluator } from '@swirls/looms/evaluator'evaluator({  definitions: [scoreRefund],  evaluator: vercelEvaluator({    provider: createTypeSafeAi({ apiKey: process.env.TYPESAFE_AI_API_KEY }),    model: 'jev-latest',  }),})

Any AI SDK provider with an evaluationModel(id) factory works the same way. The demo host reads TYPESAFE_AI_API_KEY this way and uses the stub without it.