Operations

Serving over HTTPS

Lyraflow speaks plain HTTP and has no TLS of its own. For a local trial that is fine. For anything else it is not, and not for the reason you would expect first:

• The snippet will not load. It arrives as a <script src>. On a page served over https://, a script tag pointing at http:// is active mixed content, which browsers block outright with no warning and no override. Nothing is collected and nothing says why.
• Your server key crosses the internet in clear. The write key is public by design. The server key — which reads, exports and deletes people — is on every read call you make.

So give the installer a hostname that already resolves to the server:

./install.sh analytics.example.com

A fourth container joins the stack. It takes ports 80 and 443, obtains a certificate from Let’s Encrypt on its own, renews it on its own, and forwards to Lyraflow — which stops being reachable from anywhere but the machine itself. Nothing else about the install changes, and every example in this document works against https://analytics.example.com in place of http://localhost:3000.

lyraflow snippet picks this up automatically: with a hostname configured this way, -e LYRAFLOW_HOST=... can be dropped from the docker compose exec call in Put the snippet on your website — the command defaults to https:// plus the domain you gave the installer, so the one place a wrong scheme silently produces a broken (mixed-content) snippet no longer needs typing out by hand. Every other command still needs --host/LYRAFLOW_HOST set explicitly (see packages/cli/README.md).

Leaving the hostname out keeps today’s behaviour exactly: three containers, port 3000, no certificate. That is the right choice if you already run a reverse proxy — put it in front of port 3000 as you would anything else.

The certificate and the account key live in a Docker volume, so restarts and upgrades keep them. docker compose down -v throws them away along with your data, and the next start asks for a new certificate — worth knowing before you reach for -v repeatedly, because certificate authorities rate-limit re-issuing for the same name.

Re-running ./install.sh analytics.example.com on an install that already serves that name is fine — it picks up a new image and restarts the stack. It will not change a domain that is already in .env; nothing in the installer rewrites a value that file already holds.

To go back to a local install, remove all three of the settings the installer added — LYRAFLOW_DOMAIN, COMPOSE_PROFILES and LYRAFLOW_PUBLISH — leaving the passwords alone, since they are the only copy. Then:

docker compose --profile tls down
docker compose up -d

You are left with the three containers and port 3000 again, and your data where it was.

Both halves matter, in ways that are easy to get wrong:

• Remove all three settings, not just the domain. With COMPOSE_PROFILES=tls still in .env, Caddy is still started — now with no domain to serve, so it fails to parse its configuration and restarts forever. And with LYRAFLOW_PUBLISH still there, the app stays bound to loopback, which is not reachable once Caddy is gone.
• --profile tls on the down, and no -v. Removing the settings makes Compose stop listing the caddy service at all, so a plain docker compose down — even with --remove-orphans — walks straight past the running container and leaves it holding 80 and 443. Naming the profile is what brings it back into view long enough to remove it. -v would take your database volumes with it.

Behind Cloudflare, or any other proxy

If the record is proxied — Cloudflare’s orange cloud, or an equivalent — the automatic certificate may not issue, and whether it does depends on settings that Lyraflow cannot see. The challenge is an ordinary HTTP request, so a proxy that passes port 80 through to your server will let it through; one set to redirect that traffic, or to refuse unencrypted connections to the origin, will not. The failure is quiet either way — the site simply never starts serving, and nothing says why.

The dependable answer is not to rely on that question having a good answer. Give Caddy a certificate directly, and issuance stops involving the proxy at all. For Cloudflare that means an Origin CA certificate: create one in the dashboard, save the pair on the server, and add a file to docker/caddy/tls.d/:

tls /etc/caddy/certs/origin.pem /etc/caddy/certs/origin.key

Mount the directory holding them into the caddy service, and set Cloudflare’s SSL/TLS mode to Full (strict).

You can instead grey-cloud the record until the automatic certificate issues and turn the proxy back on. That works, but it is not finished: renewal happens on its own schedule months later and meets whatever conditions exist then. A certificate that issued once behind a grey cloud is not evidence the next one will.

One thing worth being explicit about, because the setting sounds like it solves the problem and does not: Cloudflare’s Full mode does not remove the need for a certificate here. It still requires your server to speak HTTPS — it only stops checking which certificate you present. The mode that needs no certificate at all is Flexible, and it leaves the leg between Cloudflare and your server unencrypted, carrying your server key and your event data. Your visitors would see a padlock that stops being true partway.

Admin login

One admin account, a session-cookie login (POST /v1/auth/login), and the project-scoped routes it protects — including the Web UI’s own sign-in form, which calls this same endpoint. It matters to what you expose even if you never open the UI, so it is documented here rather than only in the section about the screen that uses it.

./install.sh generates the admin account the same way it generates the database passwords: a random one, written into .env, and printed once at the end of a successful install — the only time you will see it.

Resetting the login. There is one admin account, and one command that resets it. It takes the email address as its argument and the new password on stdin, and it sets both: whatever address you pass becomes the login, whether or not it matches the current one, and every signed-in browser is signed out. Forgetting the password and forgetting the address are therefore the same situation with the same fix, and neither needs the other to recover:

read -rsp 'password: ' P; echo
printf '%s' "$P" | docker compose exec -T lyraflow \
  node packages/cli/dist/index.js reset-admin-login you@example.com
unset P

reset-admin-login and set-admin-password are the same command under two names; the second is the original and is what the UI’s first-run screen and older docs print.

The command takes the password on stdin, never as an argument — an argument lands in shell history and in ps output for every user on the box. read -rs is there for the same reason and is not merely tidier: an echo 'a new password' | ... keeps the argument off ps but writes the credential straight into your shell history, which is most of the problem back again.

-T matters too. docker compose exec allocates a TTY by default and then ignores piped stdin, so without it the password never arrives and the failure is silent.

If you are upgrading an install that predates the admin account, its .env has no LYRAFLOW_ADMIN_PASSWORD — the installer only ever writes it into a brand-new .env, and an upgrade keeps the .env you already have. The app still boots; it logs a warning at startup and there is nothing to sign in with until you run the command above, once.

Stated plainly, because it is a real trade and not an oversight: the admin login is served on the exact same public origin as ingest. There is no separate port, no separate host, and no network boundary between them — an install reachable from the internet for /v1/track is reachable from the internet for /v1/auth/login too, protected by the password above and nothing else. That is the price of an install this simple. At minimum, put it behind HTTPS, and treat the admin password with the same care as the server key.

Retention

A background worker drops events older than each project’s own retention_months — 13 months by default for a new project. That default lives on the projects table and applies only going forward: it changed where a fresh install starts, not what an existing project is already configured with, so upgrading never quietly shortens anyone’s retention. Change it after creation with PATCH /v1/project (see The ingest API above), which the Web UI’s Settings screen also calls — both take the same range, 1–120, enforced again by the column’s own check constraint so a value the API validated can never fail at the database for a reason the caller wasn’t already told.

Retention is month-granular, not day-granular — a floor, not an exact promise. ClickHouse’s events and device_index tables are both partitioned (project_id, month), and the worker drops whole partitions, not individual rows. A project on 13 months therefore holds between 13 and 14 months of data depending on where in the current month you ask: the oldest surviving partition is always at least 13 months old, but it is not dropped until its entire month has aged past the boundary.

What survives, and why. Two tables are deliberately outside retention’s reach:

• person_traits — the latest known value for each trait a person has ever had (identify()’s payload), partitioned by project only, with no time dimension to expire against. A person past retention keeps their traits and their identity links (identity_bindings, in Postgres, is untouched by this worker entirely) — but not retrievably. GET /v1/persons/:id and GET /v1/persons/:id/export (see Privacy: deletion and export above) both decide whether a person exists at all from the same query, an event count, and answer 404 person_not_found when it is zero — identically to an id that was never recorded. Once retention has dropped every partition holding this person’s events, that count is zero, so both routes 404, not a profile with traits and no event lines. The traits and identity links are still there, physically, in person_traits and identity_bindings; nothing in this API can read them back out once every event is gone. If you are answering a data-subject access request for someone past retention, the honest answer this API can give is “no record found” — state that plainly rather than reading the 404 as proof the person was never recorded.
• event_schema — the distinct event and property names Lyraflow has ever seen, used for autocomplete (see Autocomplete: event and property names under Segments above). It is not partitioned by time at all, so an event name can keep showing up as a suggestion long after every event that used it has aged out and been dropped — autocomplete can offer a name that now returns nothing.

A person past retention also leaves the segment base population. Every segment’s base population is built from device_index (base.last_seen, in particular, is derived from it — see Segments above), so once a person’s last remaining device_index partition is dropped, they no longer appear in any segment count or member list — not because they were deleted, but because the aggregate row retention just removed was the only thing that put them there.

Two environment variables control the worker, and a third decides whether its work leaves any record:

Both retention variables, like every other setting the server reads, must go in the environment: block of the lyraflow service in docker-compose.yml — Compose passes only what that block lists, and a variable added to .env alone is used for substitution inside the compose file and never reaches the server.

Disabling it means retention is nobody’s job unless you make it somebody’s. LYRAFLOW_RETENTION_ENABLED=false is a legitimate choice for an operator who prunes ClickHouse some other way, but Lyraflow will not do it for you, silently or otherwise, once it is off — the server logs a line at startup saying so, precisely so that choice is visible in the boot log rather than merely absent. A disabled worker also reports 0 on both metrics below, forever — it never runs, so lyraflow_retention_last_run_timestamp_seconds never leaves 0 and lyraflow_retention_partitions_dropped_total never leaves 0 either. If you disable retention deliberately, disable or exclude the alert on the first metric too, or it will fire permanently for a state you chose on purpose.

Two /metrics series exist to alert on:

• lyraflow_retention_last_run_timestamp_seconds — the Unix timestamp of the worker’s last completed run; 0 before the first one. This is the metric to alert on, and the thing to watch is it going stale, not its value. A worker that has silently stopped — crashed, wedged, never started — looks exactly like one that is healthy and simply has nothing left to expire: neither shows up as an error anywhere else. A timestamp that stops moving is the only signal that tells the two apart, and by the time it is noticed the wrong way, the failure it exists to prevent (partitions never dropped, disk quietly filling) has already been arriving, unannounced, since the worker stopped. This timestamp still advances even on a run where every single project’s drop failed — the worker moves on to the next project and reports each failure through its own error log rather than aborting the run, so a completed run (this metric’s whole definition) is not the same claim as “something was actually dropped”. If you need to know that drops are succeeding, not merely that the worker is alive, watch the error log and the counter below together with this timestamp, not this timestamp alone.
• lyraflow_retention_partitions_dropped_total — a counter of partitions actually dropped since process start. A dry run or a run that found nothing expired does not advance it.

Retention trusts Postgres for its list of projects, and deletion is what keeps the two stores in step. The worker sweeps the projects in the Postgres projects table, so a project row removed by hand — DELETE FROM projects, a partial restore — would leave its ClickHouse partitions out of retention’s reach and out of both metrics above. That is why project deletion is a real operation rather than a row delete: lyraflow projects delete and DELETE /v1/projects/:id tear ClickHouse down first, confirm nothing is left, and only then remove the row (see Deleting a project). A deletion that keeps failing stays in the projects table with deleting_at set, and the worker keeps sweeping it, so a half-finished delete is never invisible. If you do remove a row by hand, drop that project’s partitions in ClickHouse yourself at the same time.

Quotas

A quota is off by default, and no project has one until you set it. The projects.monthly_event_quota column is nullable, NULL means unlimited, and NULL is what every project carries — both a new one and every existing one, which the upgrade rewrote on purpose rather than starting to enforce a limit nobody had opted into. Set one with PATCH /v1/project (see The ingest API above) — the same route the Web UI’s Settings screen calls — or direct SQL if you would rather:

-- 5,000,000 accepted events per calendar month for one project.
UPDATE projects SET monthly_event_quota = 5000000 WHERE slug = 'acme';

-- Back to unlimited.
UPDATE projects SET monthly_event_quota = NULL WHERE slug = 'acme';

The value must be positive (a check constraint enforces it, and the API rejects 0 and negative values the same way); send null over the API or NULL over SQL, never 0, to mean unlimited. The month is the UTC calendar month, so the budget resets at 00:00 UTC on the 1st, not on a rolling 30-day window and not in the server’s local timezone.

Understand what you are turning on before you turn it on. The write key ships in your browser bundle and is readable by anyone who visits an instrumented page. With no quota, the worst that key buys an abuser is your storage and your bandwidth. With a quota, it also buys them an off switch for your own analytics: valid events count, so a few minutes of scripted traffic can spend the month’s budget, after which your real events are refused until the 1st — by design, since that is what a quota means. Nothing here distinguishes a customer’s browser from a script; both hold the same key.

So a quota protects a bill, not a service, and it does so by trading availability for cost. Set one where an unbounded bill is the greater risk — and size it well above any month you would actually want, since a quota that is merely generous still ends in a month of silence once it is spent. If you need protection against abuse rather than against cost, that belongs in front of the ingest (a rate limit at your proxy or CDN, per IP), which the quota does not attempt and cannot replace.

A change takes up to a minute to take effect. Each server process caches the project row — quota included — for 60 seconds against the write key it arrived with, so events can still be refused for about that long after you raise a limit, and for about that long after you lower one they will still be accepted. Nothing needs restarting; wait it out.

Only accepted events count toward a quota. Malformed events, events refused by the cardinality limits, bot traffic, and events dropped when the buffer saturates all leave it untouched. That is deliberate and it is a security property, not a convenience: if rejected traffic consumed the budget, anyone holding the write key — which ships in the browser bundle — could exhaust a project’s month with payloads that are never stored as events, and silence its real analytics until the 1st.

Malformed events are not free of storage, though: each one writes a row to events_dead_letter, kept for 30 days by that table’s own TTL and bounded by nothing else. The row’s detail and payload are capped at 1000 and 8000 characters, which is not the same as bytes — a payload of non-Latin text weighs about three times its character count in UTF-8, so budget for roughly 24 KB per row rather than 9 KB. A flood of nonsense therefore costs disk whatever the quota says. What it cannot do is consume the budget.

Enforcement is a bound with known slack, not an exact cliff. Each server process keeps its recent counts in memory, folds them into Postgres every 10 seconds, and caches the persisted total for 5 seconds, so the figure the check acts on can trail reality by roughly those two intervals of that project’s own traffic — about 15 seconds’ worth. A project can therefore overshoot its quota before refusals begin: against a quota of 10, 15 events being accepted is normal and expected, not a bug. Neither interval is configurable. Running several server processes widens the same window by roughly a factor of the process count, because each holds its own pending tally and its own cache. Set a quota you can afford to exceed by a few seconds of peak traffic.

The slack is bounded by that project’s own rate over those seconds, and not by how many requests arrive at once: a burst of simultaneous requests is decided one at a time, each seeing the one before it. So the number to plan against is a project’s peak events per second, not its peak concurrency.

Once a project is over, /v1/track, /v1/identify and /v1/page answer 429 {"error":"quota_exceeded"} with no retry-after, and /v1/batch answers 202 with the refused events counted in over_quota (see Responses).

The browser SDK learns about the quota from the 202 body, never from a status code. It posts only to /v1/batch, so the 429 above is not a response it can receive at all. When a batch comes back with over_quota above zero, the SDK drops those events — a quota refusal does not clear on its own, so holding them would only wedge the queue behind events the server will refuse all month — and warns on the console naming the quota, which is the only signal a developer gets. A 429 reaching the SDK from anywhere else is treated as an ordinary rate limit: the batch is kept and retried with backoff.

A refusal is recorded in two places, and they answer different questions. lyraflow_ingest_events_total{outcome="over_quota"} on /metrics counts individual events refused since process start, across every project — it carries no project label and it resets on restart, so it tells you that refusals are happening, not who they belong to. That makes it the thing to alert on. The durable record is ingest_counters.events_over_quota in Postgres, one row per project per month, which each server process folds its tally into every 10 seconds; query that to find out which project ran out and by how much. Neither is events_dead_letter: over-quota events are deliberately kept out of it, because that table records data that could not be parsed, and filling it with valid events refused by policy would bury the bad-data signal it exists to carry.

To be warned before the cliff, alert on lyraflow_ingest_quota_used_ratio. It is a gauge, labelled by project_id, carrying this month’s accepted events as a fraction of that project’s quota:

lyraflow_ingest_quota_used_ratio{project_id="7"} 0.83

The threshold is yours to pick — 0.8, 0.95, both — which is why this is a ratio rather than a built-in warning level. The counter above tells you that events have already been lost; this one tells you they are about to be.

Three things about it are worth knowing rather than discovering:

• Only projects that have a quota appear. null is unlimited and is the default, and a ratio against unlimited is not a number. A deployment that has never set a quota emits the HELP and TYPE lines and no series, and pays nothing for them.
• A project appears only once it has sent an event this month, because the figure comes from the ingest path’s own cache rather than from a query — a scrape costs no database read, on an endpoint that is unauthenticated and scraped on a schedule the server does not control. A project that has gone quiet has no series until it sends again.
• It can exceed 1.0. A batch is admitted or refused as a whole, so a project can finish one slightly past its limit. That is not clamped, because crossing the line is the transition worth being able to see afterwards.

For the month’s durable totals, read ingest_counters; for one project’s current consumption on demand, lyraflow usage or GET /v1/project/usage.

If Postgres is unreachable, the quota is not enforced from persisted state. The usage read falls back to the last known figure for the current month, or to zero, leaving only the process’s own in-memory tally counting against the limit — a database blip must not turn into a project-wide refusal of events that were well inside their budget. The server logs quota usage read failed once per project per cache TTL while that lasts, which is the only signal that enforcement has degraded.

Backup and restore

Two scripts sit beside install.sh. backup.sh is the one you schedule; restore.sh is the one you run once, under pressure, and it is the only thing in this repository that deletes any of your data.

Taking a backup

./backup.sh /var/backups/lyraflow

It stops only the app container, waits for its grace period so the ingest buffer drains, backs up ClickHouse and Postgres with nothing writing, restarts the app, and writes:

/var/backups/lyraflow/2026-08-10T041500Z/
    clickhouse.zip     the ClickHouse database
    postgres.dump      pg_dump custom format
    MANIFEST           versions, per-table row counts, SHA-256 of each file

Ingest is refused while it runs, and so are queries. Events are delayed rather than lost: the browser SDK queues in localStorage and retries. What you get for that pause is a guarantee that fits in one sentence — the backup is a point-in-time image of both stores with no writes in flight.

How long that pause is depends on how much data you hold. On a small deployment it is about eight seconds (8.3s, 8.6s and 8.3s on three consecutive runs of the stack this repository ships); most of it is the app’s shutdown drain rather than the copying, so it grows with your data but not from a standing start. Measure your own before you decide what time of night to run it.

A nightly cron entry:

17 4 * * *  cd /srv/lyraflow && ./backup.sh /var/backups/lyraflow >>/var/log/lyraflow-backup.log 2>&1 || docker compose ps

The trailing || docker compose ps is not decoration. backup.sh restarts the app from an exit trap on every path it can control, but a SIGKILL — an OOM kill, a docker kill, a hard systemctl stop — runs no trap at all and leaves the app stopped. The ps puts the state in your log where you will see it.

If you pipe the script anywhere, test PIPESTATUS, not $?.

Rotation, off-site copies and encryption are yours to choose. find -mtime, restic and rclone all do these better than we would, and Lyraflow deliberately does none of them.

The backup file is a credential

Beyond your personal data and your projects’ write keys in plaintext, the archive contains your Postgres password. Lyraflow’s identity dictionaries live inside the ClickHouse database, and their definitions embed the credential they use to read Postgres — so three files inside clickhouse.zip carry it.

backup.sh writes everything 0600 inside a 0700 directory. Treat a backup directory exactly as you would treat the database itself, and think about that before you sync it to a bucket.

What a restore cannot give you back is a server key. Only its hash is stored, by design. If you have lost yours, no backup recovers it and the remedy is a new project.

Restoring

./restore.sh /var/backups/lyraflow/2026-08-10T041500Z

You will be asked to type the backup’s timestamp. There is no --force.

Three things are checked before anything is destroyed: the artefacts match their checksums, the backup is not newer than the image you are running, and you confirmed. Any of them refusing leaves your running system untouched — not even stopped.

Then the ClickHouse database is dropped and refilled, and the Postgres public schema is dropped and refilled. Everything written since the backup is gone.

Both stores are always restored together, and there is no flag to do one. suppressed_persons — the record that a person exercised their right to erasure — lives in Postgres, while the events it hides live in ClickHouse. A Postgres older than its ClickHouse partner brings deleted people back into every query. ClickHouse is restored first so that an interrupted restore fails on the safe side.

If a restore is interrupted part-way, the app is deliberately left stopped. The script tells you which store is in which state and asks you to run it again with the same backup, which is safe and idempotent. It does not restart the app for you, because a Lyraflow serving a half-restored database can be answering queries with no suppression rows at all — a down site is loud, and that is not.

Two smaller things worth knowing. Restoring a backup older than a project’s retention_months brings back events the policy has already expired; the next retention sweep drops them again, harmlessly. And restore.sh drops and recreates the public schema, which assumes the role Lyraflow connects with owns it — true of the stack this repository ships, and not necessarily true of a managed Postgres or a deployment where the application role is deliberately not an owner. The restore now checks this before it destroys anything and refuses with an explanation if the role cannot drop the schema, rather than failing partway through. To check for yourself:

SELECT nspowner::regrole FROM pg_namespace WHERE nspname = 'public';

Behind a CDN: recording the visitor’s IP

Caddy will not read a forwarded header from a peer it has not been told to trust, and that refusal is the right default — any client can send X-Forwarded-For, so believing it unconditionally would let a visitor choose their own apparent address.

So behind Cloudflare or any other intermediary, the address Lyraflow sees is the intermediary’s. Name the ranges you actually sit behind by dropping a file into docker/caddy/proxy.d/:

trusted_proxies static 173.245.48.0/20 103.21.244.0/22

Those directives land inside the reverse_proxy block, which is why they go in proxy.d/ rather than tls.d/ — trusted_proxies is a sub-directive of the proxy, not of the site.

The ranges are your CDN’s published egress list and they change; Cloudflare publishes theirs at https://www.cloudflare.com/ips/. A stale list fails quietly rather than loudly: an unlisted range is simply untrusted, and visitors arriving through it record the CDN’s address instead of their own.

Do not use 0.0.0.0/0. Trusting everyone is the same as having no check at all — it lets any client claim any IP by setting a header.

This has no visible effect today: GeoIP returns an empty country, region and city for every event, so nothing currently reads the client address. It matters from the moment that changes.