Skip to content

feat(billing): add posthog code token spend analysis banner#2224

Open
pauldambra wants to merge 5 commits into
mainfrom
posthog-code/llma-spend-analysis-banner
Open

feat(billing): add posthog code token spend analysis banner#2224
pauldambra wants to merge 5 commits into
mainfrom
posthog-code/llma-spend-analysis-banner

Conversation

@pauldambra
Copy link
Copy Markdown
Member

@pauldambra pauldambra commented May 19, 2026

Problem

PostHog Code users on the Plan & usage page have no way to see how their LLM token spend breaks down. The data exists in PostHog's LLM analytics — we just haven't surfaced it back inside the app. This PR adds an in-app banner that runs the analysis on demand and prints the result inline, doubling as an upsell for PostHog's LLM analytics product.

Changes

  • New TokenSpendAnalysisBanner rendered at the top of Settings -> Plan & usage. Idle state is an upsell callout with an "Analyse my spend" button; click runs the analysis, swaps to inline tables and heuristic suggestions on completion.
  • New useTokenSpendAnalysis hook (renderer-side useState/useEffect pattern matching other cloud-API callers) and a new getPostHogCodeSpendAnalysis(days) method on PostHogAPIClient that hits GET /api/llm_analytics/posthog_code_spend/.
  • Response types in apps/code/src/renderer/features/billing/types/spend-analysis.ts, mirroring the backend serializer.
  • Footer links: PostHog LLM analytics docs and the exploring-llm-costs skill in the skill store.
  • Gated behind posthog-code-spend-analysis (always on in dev). The BILLING_FLAG that gates Plan & usage itself is also forced on in dev so the section is reachable without per-developer flag setup.

Backend endpoint: PostHog/posthog#59017.

Screenshots

Banner idle state — Settings → Plan & usage:

CleanShot 2026-05-19 at 16 19 20@2x

How did you test this code?

I'm an agent. I did not perform manual UI testing — the screenshot above is from the human author's local run.

  • pnpm --filter=@posthog/code typecheck passes on the rebuilt branch.
  • pnpm biome check --write --unsafe clean on the touched files.

The banner is reachable in dev at Cmd+, → Plan & usage.

Publish to changelog?

no — backend dependency lands first; user-facing announcement comes later.

🤖 Agent context

Authored with PostHog Code (Claude Opus 4.7).

  • Approaches considered and rejected:
    • Letting an agent task handle the analysis (background task with results in side panel) — adds friction for a one-shot upsell view.
    • Calling MCP directly from the renderer — wrong scope; MCP is per-agent auth.
    • Querying the customer's own PostHog project's HogQL — wrong project; the data lives in PostHog's internal cloud analytics team.
  • Chose a fixed backend report + inline tables. Faster, deterministic, no LLM in the loop for data fetching.
  • Suggestions are hardcoded heuristics (Bash share, no-tool share, top-trace share). Easy to swap for an LLM-generated narrative later if useful.

Surfaces a self-serve spend analysis inside Settings -> Plan & usage.
Calls the new /api/llm_analytics/posthog_code_spend/ endpoint, renders
totals + breakdowns by ai_product / tool / model + top traces, and
generates inline heuristic suggestions on where to optimise. Footer
links to PostHog LLM analytics docs and the exploring-llm-costs skill.

Gated behind the posthog-code-spend-analysis feature flag (always on
in dev). Plan & usage section itself is also forced on in dev so the
banner is reachable without billing-flag setup.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 19, 2026

Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
apps/code/src/renderer/features/billing/hooks/useTokenSpendAnalysis.ts:28-33
Refresh errors are silently swallowed after the first successful load. Because `data` is never cleared on failure, the banner always takes the `if (data)` branch in `TokenSpendAnalysisBanner`, so the `if (error)` branch is unreachable once data has been fetched. When a refresh fails the user sees the stale previous result with no indication anything went wrong.

```suggestion
      const result = await client.getPostHogCodeSpendAnalysis(days);
      setData(result);
      setError(null);
    } catch (err) {
      const message = err instanceof Error ? err.message : "Unknown error";
      log.warn("Failed to fetch spend analysis", { error: message });
      setData(null);
      setError(message);
```

### Issue 2 of 3
apps/code/src/renderer/features/billing/components/TokenSpendAnalysisBanner.tsx:23-24
The `amount < 1` branch is a strict subset of `amount < 100` and performs the exact same operation (`toFixed(2)`), so it is dead code that adds no distinction. This violates the "no superfluous parts" simplicity rule.

```suggestion
  if (amount < 100) return `$${amount.toFixed(2)}`;
```

### Issue 3 of 3
apps/code/src/renderer/features/billing/types/spend-analysis.ts:22-28
`SpendAnalysisModelRow` is defined and `by_model` is included in `SpendAnalysisResponse`, but the component never renders a model breakdown table. The data is fetched over the wire and typed, yet discarded. Either a `ModelTable` is missing from the banner, or the interface and field should be dropped to satisfy the "no superfluous parts" rule.

Reviews (1): Last reviewed commit: "feat(billing): add posthog code token sp..." | Re-trigger Greptile

Comment thread apps/code/src/renderer/features/billing/hooks/useTokenSpendAnalysis.ts Outdated
Comment thread apps/code/src/renderer/features/billing/components/TokenSpendAnalysisBanner.tsx Outdated
Comment thread apps/code/src/renderer/features/billing/types/spend-analysis.ts
- Clear data state on fetch error so error UI is reachable on refresh failures.
- Render the model breakdown table that was already typed and fetched but unused.
- Remove redundant amount < 1 branch in formatUsd.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
@pauldambra pauldambra added the Stamphog This will request an autostamp by stamphog on small changes label May 19, 2026
Copy link
Copy Markdown
Contributor

@joshsny joshsny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - wondering if it should be an MCP tool and we send them into the chat, probably not since they want to look at this when they ran out of $ so that would be frustrating

Renames the client method to getPersonalSpendAnalysis, takes an options
object with optional `product` query param, and points at the new
/api/llm_analytics/personal_spend/ endpoint. Banner passes
`product: "posthog_code"` so the tool / model / trace breakdowns stay
PostHog Code-scoped; the by_product breakdown shows the cross-product
distribution.

SpendAnalysisSummary drops posthog_code_cost_usd / posthog_code_event_count
in favour of generic scoped_cost_usd / scoped_event_count fields, matching
the backend shape.

Hook renamed useTokenSpendAnalysis -> useSpendAnalysis.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
…aginated

Mirrors the backend changes:

- URL: getPersonalSpendAnalysis now calls /api/llm_analytics/@me/spend/
- Response shape: each breakdown is `{ items, truncated }`; banner reads
  `.items` and tables stay the same.
- SpendAnalysisToolRow gains `share_of_scoped` (float 0-1). Banner
  suggestions use this directly, no longer divide cost_usd by
  scoped_cost_usd (which can over-count for multi-tool generations).

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
Tracks the backend rename:

- Client method now takes `{ dateFrom, dateTo, product }` instead of
  `{ days, product }`. Maps to the `date_from` / `date_to` query params
  the backend now accepts.
- SpendAnalysisSummary swaps `period_days` for `date_from` / `date_to`
  ISO strings.
- Banner derives the "Window" stat card from the two timestamps.

Generated-By: PostHog Code
Task-Id: f9d5d152-49c6-46cf-8fde-079105ba2e67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Stamphog This will request an autostamp by stamphog on small changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants