The ingest API

Ingest listens on port 3000. /v1/track, /v1/page, /v1/identify and /v1/batch all authenticate with the write key in the x-lyraflow-write-key header.

Endpoints

/health (liveness), /ready (readiness), and /metrics (Prometheus text format) are also served, and are not authenticated. Neither are GET /v1/shared/:token and POST /v1/shared/:token/tiles/:index/run, which serve a shared dashboard to whoever holds its link; see Sharing a dashboard.

GET /v1/project

Server-key authenticated (x-lyraflow-server-key), unlike every endpoint above. Returns {"name", "slug", "write_key"} — the project’s own identity, including the write key, which create-project otherwise prints only once and nothing else serves. This is not a widening of what the server key can already do: it is a secret, hashed at rest and unrecoverable, that authenticates every read path in this API — a caller holding it can already read every person, event and segment in the project, so handing back a value that ships unhidden in the browser bundle of every instrumented page changes nothing about what that caller can reach. lyraflow snippet (see packages/cli/README.md) is the intended way to use this endpoint: it prints a paste-ready install snippet with the write key already filled in, rather than a caller reading this response by hand. lyraflow snippet --rotate rotates the key first and prints the snippet that carries the new one.

POST /v1/project/rotate-write-key

Server-key authenticated. Replaces the project’s write key and returns the new one: {"write_key", "previous_write_key_expires_at"}. The body is optional and takes {"grace_hours"}, an integer from 0 to 720 (default 24): for that long the key being replaced keeps working, so pages still serving the old snippet keep collecting while their caches turn over. 0 is a hard swap. There is only ever one previous key — rotating again inside the grace retires the older one immediately.

Once the grace ends, a page still carrying the old key gets 401 invalid_write_key on its next flush and the SDK stops sending for the life of that page, silently apart from one console warning; that is the browser SDK’s own rule for 401, documented under Responses. Lyraflow caches each key’s project for a minute, so a retired key can be accepted for up to a minute past its expiry by a server that looked it up just before — the same window lyraflow projects delete waits out.

Rotation is the remedy for a leaked write key. It is not a remedy for a leaked server key, which is hashed at rest and cannot be rotated in place; that remains a new project.

PATCH /v1/project

Also server-key authenticated. Body is {"retention_months"}, {"monthly_event_quota"}, or both — this is what the Web UI’s Settings screen calls, and it is the only API surface documented under Retention and Quotas below; there is no longer a raw SQL statement to run for either. A field’s absence means “leave it alone”; it is not the same as sending it explicitly. monthly_event_quota: null sets unlimited; omitting the key entirely changes nothing about the existing quota. Sending neither field is a 400. retention_months is 1–120, matching the column’s own check constraint; monthly_event_quota is a positive integer or null — 0 is rejected rather than accepted and silently misread as a limit, since 0 is what isOverQuota treats as “no limit configured” and refusing it here is cheaper than that ambiguity reaching ingest. Returns the row’s current {"retention_months", "monthly_event_quota"} on 200, and invalidates the 60-second project cache the retention worker and the ingest quota check both read from, so a new limit is in force immediately rather than up to a minute later.

GET /v1/project/usage

Server-key authenticated. Returns the active project’s counters for the current calendar month — {"month", "events_accepted", "events_rejected", "events_throttled", "events_bot", "monthly_event_quota"} — all zero for a project with no row yet this month, which is the ordinary state for a brand-new one. events_bot counts events dropped as crawler traffic and is reported apart from events_rejected (malformed input), because a large rejection count means the integration is broken and a large bot count does not. This is what the Settings screen’s usage card reads.

The project list: GET, POST, PATCH and DELETE /v1/projects

Session-cookie authenticated, not server-key — these are instance-scoped (“which projects exist”, “create one”) rather than project-scoped, so a server key (which names one project) cannot answer them, and accepting one would let a single project’s credential enumerate every other project on the install. In practice this means: the CLI’s create-project and these routes are the only ways to create a project, and only an admin signed into the Web UI (or holding its session cookie) can list every project or create a new one over HTTP. GET /v1/projects returns {"projects": [...]}, wrapped rather than a bare array, with each entry shaped {"id", "name", "slug", "created_at", "retention_months", "monthly_event_quota", "disabled_at", "deleting_at"} and no key of either kind — the one response in this API that names every project at once, so a key leaking here would leak the whole install rather than one project. POST /v1/projects takes {"name"}, slugifies it the same way create-project does, and returns {"name", "slug", "write_key", "server_key"} — the server key shown once, exactly as create-project prints it once, and never served again by anything.

PATCH /v1/projects/:id takes any of {"name"} and {"archived": true|false} and returns the updated entry; it never changes the slug. DELETE /v1/projects/:id takes {"slug"} in the body as the confirmation — a mismatch is 409 slug_mismatch, a project already being deleted is 409 already_deleting — and answers 202 {"id", "project_id", "status": "pending"}. The teardown runs in the background; GET /v1/project-deletions/:id reports pending, in_progress, completed or failed (with the last error). All four are session-cookie routes for the reason above. Retention and quota are per-project settings and live on PATCH /v1/project, the server-key route, not here.

GET /v1/meta

What release this install is running, as {"version": "0.15.0"}. The Settings screen’s Install card reads it, which is where an operator finds the number to quote into a bug report or to compare against the latest release.

Session-cookie authenticated, like the two routes above, and instance-scoped for the same reason — “what version is this” names no project, so a server key cannot answer it. It is deliberately not on /health: a version number tells a caller which published advisories apply to the install, and /health answers anything that can reach the port. Requiring a signed-in admin is the difference between an operator reading their own version and the internet reading it.

/v1/meta rather than /v1/version because the path is the expensive half to change later and the body is not. It carries one field today; a second is a decision about what an install discloses about itself, not a field appended in passing.

Sent directly from browser JavaScript (as opposed to a server-side SDK),

Page views, and the $ prefix

Every page view is stored under the event name $page, named or not. A name becomes the $page_name property rather than the event name, so “how many page views” is one query and page('signup') cannot be confused with track('signup').

It works that way for a storage reason as much as a naming one: event_name is a LowCardinality(String) column and the second key of the schema catalogue’s sort order, while page names are unbounded by construction — one per URL. Under the old behaviour every page name also claimed its own property-key budget instead of all page views sharing one.

$ is reserved for Lyraflow. A property key you send that begins with $ is dropped, in both the string and the numeric map — dropped rather than refused, because ingest degrades rather than failing an otherwise valid event, and because the write key is public and refusing here would hand a visitor a way to make a site’s events disappear. Everything else about the key is untouched: price_$ and a$b are ordinary keys and are stored.

Events already stored under a page name stay as they are. This changes what is written from now on, not history — so a project that used page(name) before this release has its old page views under their old names and its new ones under $page.

/v1/track, /v1/page, /v1/identify and /v1/batch are CORS-preflighted requests. By default Lyraflow answers that preflight for any origin — set LYRAFLOW_ALLOWED_ORIGINS (comma-separated) to restrict it. On the shipped Compose stack that means a line in .env, which the lyraflow service’s environment: block passes through; anywhere else it is an environment variable on the server process. Confirm it took effect rather than assuming it did: the server states which mode it is in on every boot, so docker compose logs lyraflow | grep 'ingest CORS' answers it in one line. Getting ingest CORS unrestricted back after setting the variable means the value never reached the process — see When the allowlist does not take effect.

This is not a security boundary: the write key already ships in page source, and any non-browser client ignores CORS entirely. What it buys is tamper-evidence — stopping someone from pasting your write key on their own site and quietly polluting your data — not access control. Leave it unset and any origin is allowed, which is why a fresh install’s tracking snippet works on first paste with no configuration.

Payload fields

Control characters in an event name or a property key are escaped when the event is written, not when it is displayed. A byte in the C0 range (0x00–0x1f), DEL (0x7f) or C1 (0x80–0x9f) is stored as a printable \xNN escape instead — so signup followed by an ESC is stored as signup\x1b, and what you read back can never itself be a terminal instruction, an HTML break or a log-line split.

This matters because a write key is public by construction — it ships inside the browser bundle and is readable in devtools on any instrumented page — so anyone who visits an instrumented site can choose these bytes. Escaping once, at ingest, is what stops every consumer from having to remember: the terminal output of the CLI, the web UI, exports, and whatever you build next.

The consequence to know about: a stored name with a control character in it does not match the bytes that were sent. The escape is chosen to be legible rather than lossy — \x1b tells you exactly which byte arrived, where dropping it would silently merge sign+newline+up and signup into one event — but if you are reconciling against your own logs, that is where a difference comes from. Property values are never rewritten, only names and keys: a value is your data, and a newline in a support message is content rather than a formatting accident.

A payload that declares one of Lyraflow’s server-side SDKs is never filtered as a bot. That matters because the HTTP clients those SDKs use announce themselves as python-requests, okhttp or curl/ — indistinguishable from a scraper, and dropped as one before this field existed. The browser SDK does not send this field; an absent library is filtered exactly as before.

A server-side SDK is judged on the visitor agent it forwards. If a payload declares a server-side library and carries context.user_agent, that value is what the bot filter reads, and what device_type, os and browser are parsed from. So a backend passing through Googlebot/2.1 has that crawler filtered rather than recorded as a person — and a backend passing through a real visitor’s agent stops recording an unknown device. A declared SDK that forwards nothing is exempt exactly as before.

context.user_agent is only consulted for a declared server-side library. Everything else — every browser payload — is judged and enriched from the request’s own User-Agent header, as it always has been.

Bot filtering is data hygiene, not a security boundary. The write key ships inside the browser bundle, so any client can claim to be a server-side SDK — or simply send a browser’s User-Agent, which has always been possible. What the filter removes is incidental traffic: crawlers, uptime monitors, link-preview fetchers. None of those declare a library.

Reading a forwarded agent does not widen that. It is consulted only for callers already exempt, so it can only ever cause more filtering, never less: there is no payload it lets through that could not already get through by declaring a library and forwarding nothing.

Property and trait values may be strings, numbers, booleans, or null. Numbers are stored in a numeric column and everything else as text, so 3 and "3" are not interchangeable. Nested objects and arrays are not supported. An event may carry up to 250 properties.

A boolean is stored as the text true or false, and there is no boolean type. That is more than a storage detail, because it means true and the string "true" are the same value once ingested — indistinguishable, with no way to tell them apart afterwards. Three consequences worth knowing before you instrument anything:

• /v1/schema/properties reports string for such a property, never boolean, so autocomplete cannot tell a caller the underlying value is two-valued.
• A segment filter must use the string form: is_subscribed = "true", not is_subscribed = true.
• "True", "1" and "yes" are three different values in that same column, from callers who each reasonably believed they were sending a boolean. Nothing rejects, warns, or reports the coercion, so an integration looks correct and the divergence only surfaces when someone builds a segment months later.

If you send booleans, send them consistently and expect to filter on "true".

A context field over its limit costs the whole event, not just that field: the event fails validation, is dead-lettered, and the response still says 202 — with rejected counting it. This is easier to hit than it looks; an OAuth callback URL carrying a redirect_uri clears 2048 characters without trying. The browser SDK truncates url, path, referrer and user_agent to these limits before sending, and warns on the console when it does. If you are calling the HTTP API directly, truncate them yourself.

Client clocks are frequently wrong, so an explicit timestamp is clamped to within 24 hours of server time.

Responses

• 202 — accepted. Also returned for malformed events and for events dropped as bot traffic, deliberately: a tracking endpoint that returns an error breaks the customer’s site. Malformed events are recorded in the events_dead_letter table with the reason; bot traffic is simply counted and discarded. /metrics reports the accepted, rejected, throttled, over-quota and bot totals, so a 202 that stored nothing is still visible there.
• 401 — missing or unknown write key, or a project that has been archived, which answers {"error":"project_archived"}, or one being deleted, which answers {"error":"project_deleted"}. Both stop collection deliberately; the status is 401 rather than 403 because the browser SDK treats 401 as final and stops, while any other status is retried indefinitely by every bundle already deployed on a page.
• 429 with {"error":"quota_exceeded"} — the project has used its monthly event quota. No retry-after, deliberately: unlike a 503, this does not clear on its own shortly. It holds until the month rolls over or an operator raises the limit, so retrying is pointless. No project has a quota until an operator sets one; see Quotas under Operations.
• 503 with retry-after: 5 — the server is saturated or shutting down. Retry.
• 400 / 413 — malformed JSON, or a body over 1 MiB. Retrying will not help.

/v1/batch always answers with counts: {"accepted":n,"rejected":n,"throttled":n,"over_quota":n,"bot":n}. It returns 503 if the buffer saturates part-way through, with the counts describing how far it got; retry the whole batch. It never returns 429: a batch answers 202 with over_quota counting the events refused, because its contract is a body carrying the tally rather than a wholesale failure over one event. Those events are not worth retrying either. Read over_quota even when the status is 202 — for a batch, it is the only signal that events were refused. bot counts items dropped as bot traffic, the same outcome /metrics reports above. Single-event routes (/v1/track, /v1/page, /v1/identify) are unchanged: they still answer {"status":"accepted"} regardless of outcome and carry no such count.

Retries

Retry a 503 with the same message_id; it becomes the event’s id (see Payload fields above). A replayed event is never double-counted as long as your query selects DISTINCT event_id (or otherwise aggregates by it) — a plain count(*) can see it as two rows, and ClickHouse’s FINAL modifier does not rescue that when the retry omitted timestamp (see below). There is no query API yet, so this is on you: it is the same discipline any ClickHouse client of this table needs.

If you also send an explicit timestamp and replay it unchanged, the storage engine’s own row collapse — deterministic under FINAL, eventual otherwise — removes the duplicate outright, so the retry costs no extra disk. Omit timestamp and the server stamps each attempt with its own receipt time; because that receipt time is part of the table’s sort key, the two rows never collapse — deduplicated only by querying event_id yourself, correct but not free. Long-lived retry queues should send timestamp.

That collapse has a 24-hour shelf life, and it expires silently. The clamp above rewrites any timestamp more than 24 hours from server time to the boundary — a value computed from now, so it is different on every attempt. A queue that drains within 24 hours of the original event collapses as described. One that drains later does not: each retry is clamped to a different instant, lands as another permanent row, and is also misdated to the clamp boundary rather than when it happened. Nothing reports this. If your retry queue can outlive a day, aggregate by event_id and treat the engine collapse as an optimisation you do not have.