Skip to content
Gilmend
Engine used for new reports:Muse Spark 1.3 (Meta Model API)
Report a damaged parcel

Evaluation

How the agent holds up

The six demo cases, each run 3 times on Muse Spark 1.3 (Meta Model API), scored against the outcome we expected, with how much the runs varied.

6 cases × 3 runs: a sanity check on the demo, not a benchmark. That is too few runs to put a rate on: 3 of 3 rules out very little, and such a case could still fail often on other runs. These are also the six cases the agent was built and tuned on (its prompt, guardrails and SAM concepts were adjusted while looking at them), so this shows how repeatable it is on them, not how it does on parcels it has never seen. What counts as right is what we wrote down ourselves in data/demo-cases.json; the orders are fictitious and the photos are AI-generated stand-ins. Amounts and deadlines come from code, not from the model, and the SAM column counts the regions it outlined, not whether the outline is right.

Measured on commit 0eab783; the agent deployed here (9cc2002) is not that one. Its code may have changed since, so these figures describe the measured version, not necessarily the agent you can try on this site. The command below re-runs the evaluation on whatever code you check out.

Summary

Every expectation met

17/18

runs: decision, carrier claim, human review and claim eligibility all as expected

Same answer every run

5/6

cases that gave the same decision, claim and human-review answer in every run (right or wrong)

Median run

28.2 s

from report to finished case, including the SAM outline and the claim document

Estimated cost

$1.18

for all 18 runs, an estimate at Meta's published starting price ($1.25/$4.25 per M tokens in/out), cache discounts ignored: 766.9k tokens (425.2k served from the provider's cache) and 22 SAM requests

By expected field, over 18 runs: decision 18, carrier claim 18, human review 17, claim eligibility 18. 18 of 18 runs reached a decision.

Case by case

  1. demo-1-vase-broken

    Broken vase (Colissimo): reship and claim the carrier

    3 of 3 runs meet every expectation

    Expected

    Decision
    Reship
    Carrier claim
    Yes
    Human review
    No
    Claim eligible (by code)
    Yes

    Observed

    Decision
    Reship ×33 of 3 runs acceptable
    Carrier claim kept
    3 of 3 runs3 of 3 runs as expected
    Human review asked
    0 of 3 runs3 of 3 runs as expected
    Claim amount (by code)
    €52.90
    Median duration
    29.7 s
    SAM outline
    2 regions (median), 3 of 3 runs

    Same answer every run

    1. Run 1: Reship, claim
    2. Run 2: Reship, claim
    3. Run 3: Reship, claim

  2. demo-2-carton-crushed

    Crushed carton, product intact: no claim

    3 of 3 runs meet every expectation

    Expected

    Decision
    No action
    Carrier claim
    No
    Human review
    No
    Claim eligible (by code)
    No

    Observed

    Decision
    No action ×33 of 3 runs acceptable
    Carrier claim kept
    0 of 3 runs3 of 3 runs as expected
    Human review asked
    0 of 3 runs3 of 3 runs as expected
    Claim amount (by code)
    No claim amount
    Median duration
    28.8 s
    SAM outline
    1 region (median), 3 of 3 runscalled by code, not the model, in 2 of 3 runs

    Same answer every run

    1. Run 1: No action
    2. Run 2: No action
    3. Run 3: No action

  3. demo-3-photo-blurry

    Blurry photo: ask for another one

    2 of 3 runs meet every expectation

    Expected

    Decision
    Ask for more photos
    Carrier claim
    No
    Human review
    No
    Claim eligible (by code)
    No

    Observed

    Decision
    Ask for more photos ×33 of 3 runs acceptable
    Carrier claim kept
    0 of 3 runs3 of 3 runs as expected
    Human review asked
    1 of 3 runs2 of 3 runs as expected
    Claim amount (by code)
    No claim amount
    Median duration
    23.3 s
    SAM outline
    None

    The runs differ

    1. Run 1: Ask for more photos, human. Missed human review
    2. Run 2: Ask for more photos
    3. Run 3: Ask for more photos
    What the agent said in the run that missed

    • Run 1

      • Order #1003 found
      • Photo 0 is heavily motion-blurred and shows no verifiable crack or damage
      • Damage cannot be confirmed without clearer photos

  4. demo-4-opened-missing

    Parcel opened at the relay, item missing (Mondial Relay, 72 h window)

    3 of 3 runs meet every expectation

    Expected

    Decision
    Reship or Refund
    Carrier claim
    Yes
    Human review
    No
    Claim eligible (by code)
    Yes

    Observed

    Decision
    Reship ×33 of 3 runs acceptable
    Carrier claim kept
    3 of 3 runs3 of 3 runs as expected
    Human review asked
    0 of 3 runs3 of 3 runs as expected
    Claim amount (by code)
    €25
    Median duration
    27.8 s
    SAM outline
    5 regions (median), 3 of 3 runs

    Same answer every run

    1. Run 1: Reship, claim
    2. Run 2: Reship, claim
    3. Run 3: Reship, claim

  5. demo-5-expensive-escalation

    Expensive item: hand over to a human

    3 of 3 runs meet every expectation

    Expected

    Decision
    Refund or Reship
    Carrier claim
    Yes
    Human review
    Yes
    Claim eligible (by code)
    Yes

    Observed

    Decision
    Reship ×33 of 3 runs acceptable
    Carrier claim kept
    3 of 3 runs3 of 3 runs as expected
    Human review asked
    3 of 3 runs3 of 3 runs as expected
    Claim amount (by code)
    €250
    Median duration
    30.5 s
    SAM outline
    1 region (median), 3 of 3 runs

    Same answer every run

    1. Run 1: Reship, claim, human
    2. Run 2: Reship, claim, human
    3. Run 3: Reship, claim, human

  6. demo-6-late-report

    Reported too late: refund, no carrier claim possible

    3 of 3 runs meet every expectation

    Expected

    Decision
    Refund or Reship
    Carrier claim
    No
    Human review
    No
    Claim eligible (by code)
    No

    Observed

    Decision
    Refund ×33 of 3 runs acceptable
    Carrier claim kept
    0 of 3 runs3 of 3 runs as expected
    Human review asked
    0 of 3 runs3 of 3 runs as expected
    Claim amount (by code)
    No claim amount
    Median duration
    24.9 s
    SAM outline
    1 region (median), 3 of 3 runs

    Same answer every run

    1. Run 1: Refund
    2. Run 2: Refund
    3. Run 3: Refund

Reproduce it

Needs a MODEL_API_KEY in .env.local. It spends real credits, refuses the demo mock, and stops when the projected cost would go over $5 (add --yes to go on). Reasoning effort and API host come from the environment, not from the command: set them as shown below to match.

npm run eval -- --runs 3
Engine
Muse Spark 1.3 (Meta Model API)
Model id
muse-spark-1.3
Run on
27 Oct 2026, 18:43
Code
0eab783
Results file
data/eval/2026-10-27-muse-spark-1.3.json
Reasoning effort
not recorded in this file
API host
not recorded in this file
Cost estimate based on
Meta Model API pricing page (dev.meta.ai), read 27 Oct 2026; cache discounts ignored