Eval & Observability
Real telemetry from every model call.
FrictionLens scores app reviews across five sentiment dimensions using Gemini. This page shows the eval intuition layer behind that pipeline — Spearman correlation per dimension against a hand-labeled golden set, plus real cost, latency, and reliability metrics from production traffic.
Calls (30d)
43
Total model invocations
Spend (30d)
$0.13
Sum of input + output cost
p95 latency
33546ms
Weighted by call volume
Error rate
34.9%
1 errors · 14 rate-limited
Cost by model
Total USD across last 30 days
- gemini-2.5-flash-lite$0.1023
- gemini-3.1-flash-lite$0.0287
Latency by prompt
p50 / p95 / p99 across last 30 days
- batch-review-v131 callsp5024766msp9542095msp9951598ms
- report-v56 callsp5010442msp9512193msp9912193ms
- report-v33 callsp5012635msp9517212msp9917212ms
- review-v12 callsp501208msp951349msp991349ms
- report-v41 callsp5010034msp9510034msp9910034ms
Latest eval run
Spearman correlation per dimension vs. hand-labeled golden set
- Prompt
- review-v1
- Model
- gemini-2.5-flash
- n
- 19 reviews · golden vv1
- Date
- 2026-05-14
| Dimension | Spearman ρ | MAE (0–10 scale) | Verdict |
|---|---|---|---|
| love | 0.946 | 0.63 | strong |
| frustration | 0.951 | 0.95 | strong |
| loyalty | 0.971 | 0.68 | strong |
| momentum | 0.912 | 0.89 | strong |
| wom | 0.987 | 0.37 | strong |
Prompt registry
Deployed prompts with version, call volume, and last-deployed date
| Prompt ID | Name | Version | Calls (30d) | Deployed | Notes |
|---|---|---|---|---|---|
| report-v1 | report | v1 | 0 | 2026-05-13 | Initial aggregate Vibe Report synthesis prompt. |
| review-v1 | review | v1 | 2 | 2026-05-13 | Initial scoring rubric for single-review analysis. |
| batch-review-v1 | batch_review | v1 | 31 | 2026-05-13 | Same rubric as review-v1, applied across a batch of reviews. |
| report-v2 | report | v2 | 0 | 2026-05-13 | Adds verdict, the_one_thing, citations, confidence, wishlist/dealbreaker bucketing, vagueness rejection, de-duplication. |
| report-v3 | report | v3 | 3 | 2026-05-14 | Adds strict review-ID binding: cited_review_ids must use only the rNNN IDs supplied, never invented. Enables the click-to-see-receipts UI. |
| report-v4 | report | v4 | 1 | 2026-07-12 | Adds topic continuity: reuse the baseline analysis's friction/churn topic names verbatim so period-over-period deltas pair the same topic instead of naming drift. |
| report-v5 | report | v5 | 6 | 2026-07-12 | Topic continuity v2: v4's buried instruction was ignored live (0/10 topics reused). Moves the topic list to the TOP of the user prompt, adds a reinforcement at the produce-list decision point, and hardens the system rule. |
Recent traces
Last 20 model calls (anonymized — no user identifiers)
| When | Prompt | Model | Tokens (in / out) | Latency | Cost | Source | Status |
|---|---|---|---|---|---|---|---|
| 13d ago | batch-review-v1 | gemini-3.1-flash-lite | 6956 / 7856 | 17360ms | $0.0135 | — | success |
| 13d ago | batch-review-v1 | gemini-3.1-flash-lite | 7890 / 7286 | 17114ms | $0.0129 | — | success |
| 13d ago | batch-review-v1 | gemini-3.1-flash-lite | 865 / 346 | 1398ms | $0.0007 | — | success |
| 13d ago | review-v1 | gemini-3.1-flash-lite | 773 / 189 | 1208ms | $0.0005 | — | success |
| 13d ago | batch-review-v1 | gemini-3.1-flash-lite | 829 / 224 | 1106ms | $0.0005 | — | success |
| 13d ago | review-v1 | gemini-3.1-flash-lite | 773 / 189 | 1349ms | $0.0005 | — | success |
| 13d ago | report-v5 | gemini-2.5-flash-lite | 5179 / 2004 | 11890ms | $0.0013 | report_api | success |
| 13d ago | batch-review-v1 | gemini-2.5-flash-lite | 4165 / 6555 | 51199ms | $0.0030 | — | success |
| 14d ago | report-v5 | gemini-2.5-flash-lite | — / — | 9247ms | — | pipeline | rate_limit |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 4905 / 6809 | 20506ms | $0.0032 | pipeline | success |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 10046 / 13743 | 39737ms | $0.0065 | pipeline | success |
| 14d ago | report-v5 | gemini-2.5-flash-lite | — / — | 12254ms | — | pipeline | rate_limit |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 7321 / 10097 | 26626ms | $0.0048 | pipeline | success |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 9527 / 13209 | 32112ms | $0.0062 | pipeline | success |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | — / — | 42461ms | — | pipeline | error |
| 14d ago | report-v5 | gemini-2.5-flash-lite | — / — | 11118ms | — | pipeline | rate_limit |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 7321 / 10130 | 31789ms | $0.0048 | pipeline | success |
| 14d ago | batch-review-v1 | gemini-2.5-flash-lite | 8556 / 13645 | 38311ms | $0.0063 | pipeline | success |
| 15d ago | batch-review-v1 | gemini-2.5-flash-lite | — / — | 6326ms | — | pipeline | rate_limit |
| 15d ago | batch-review-v1 | gemini-2.5-flash-lite | — / — | 30657ms | — | pipeline | rate_limit |