Server Side Analytics

The events that never touch a browser, and are usually the ones that matter most.

What only your backend knows

A browser can tell you someone clicked Subscribe. Only your backend knows whether the payment cleared, whether the webhook was retried four times, whether the nightly export finished, and whether the subscription silently churned ninety days later.

Server-side events are also the trustworthy ones. They cannot be blocked by an extension, lost to a closed tab, or forged by a client. If a number needs to be correct rather than indicative, send it from your server.

There is no SDK to install. It is an HTTPS POST with a JSON body, and the batch limit of 1000 events per request means a job summarising a whole day costs a handful of calls.

Wiring a service to the portal

No library, no agent, no build change — the integration is a URL and two identifiers.

  1. Create a site for the service

    Add your ClickHouse module under Infralytiqs → Settings and give the service its own siteId — acme-etl, acme-checkout-api. Services usually deserve their own id rather than sharing the website’s, because their event vocabulary and their retention needs are different.

    Settings

  2. Post to the collector

    POST {serverUrl}/il/analytics/{tenantId}/{siteId}/events with Content-Type: application/json and either a single object or an array of up to 1,000. There is no Authorization header — the registered tenant and site pair in the path is the authorisation, and an unregistered pair returns 403.

    Set device_type to server and device_os to something meaningful for your runtime. It keeps backend traffic trivially separable from browser traffic in every report.

    Contract

  3. Decide what carries the identity

    Server-side events have no browser to supply a session. Use user_id where you know the human, and put the job or correlation id in session_id where you do not — that is what lets you group every event from one pipeline run.

    Design

  4. Send the same correlation id to Loki

    Put the job id into your logging MDC as well, and add it to the log line pattern. One identifier in both systems is what turns “this batch reported 14 failures” into “here are the 14 log lines”, and it costs one line of code.

    Correlate

  5. Verify, then chart it

    A 201 with { success: true, count: N } means committed. Then build the panel: throughput per run, p95 duration, failure counts by reason. A scheduled job that has quietly processed nothing for three weeks looks exactly like a healthy one until it is on a chart.

    Verify

What to instrument

Ordered roughly by how quickly each one earns its keep.

Business transactions

Orders, subscriptions, refunds, plan changes. Emit these from the code that commits the transaction, so the event exists if and only if the thing actually happened.

Webhook traffic

Inbound webhooks from payment providers and partners, with the delivery attempt number and the outcome. The first thing you want when a partner claims they sent something.

Scheduled jobs

Start, finish, duration and records processed. A job that has silently done nothing for three weeks looks identical to a healthy one until you chart it.

Errors with context

Not a replacement for your error tracker, but errors as events let you group failures by customer, plan or region — the dimension an error tracker usually lacks.

Authentication and authorisation

Logins, failures, token refreshes, permission denials. A spike in denials is a product problem as often as a security one.

Machine consumers

API keys, integration partners and internal services. Whole classes of usage never appear in browser analytics at all.

Worked example: a Spring Boot ETL service

One service, both halves — analytics events to Infralytiqs, log streams to Loki, correlated by the same run id. This is the shape the AEM integrator uses, written out for an ordinary Spring application.

<!-- Nothing Infralytiqs-specific is required: the collector is plain
     JSON over HTTPS, so RestClient or WebClient is enough.

     Loki is the only added dependency. For a standalone Spring app the
     loki-logback-appender is the least intrusive route -- Logback
     pushes to /loki/api/v1/push directly and nothing in your code has
     to know about it. (Inside AEM you would instead deploy the
     aem-loki-integrator bundle, which is the OSGi equivalent.) -->

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>

  <dependency>
    <groupId>com.github.loki4j</groupId>
    <artifactId>loki-logback-appender</artifactId>
    <version>1.5.2</version>
  </dependency>
</dependencies>

The event vocabulary in the example

Small, deliberate and stable. Two event types carry the whole pipeline.

Event Subtype Dimensions Metrics
etl_run etl_run_started pipeline, trigger run_signal
etl_run etl_run_status_success pipeline rows_total, batches_failed, run_duration_ms
etl_batch etl_batch_status_success pipeline, source_system, batch_id rows_processed, batch_duration_ms
etl_batch etl_batch_status_failed pipeline, source_system, batch_id, failure_reason failure_signal

Why the run id is the whole trick

The example above does one thing that is easy to skip and hard to retrofit: it puts the same run_id into the logging MDC and into session_id on every analytics event.

That single shared value is what makes the two systems one system. The report tells you that last night’s run committed 40,000 fewer rows than usual and that three batches failed with SocketTimeoutException. You copy the run id out of the drill-down, paste it into the Logs page as {app="acme-etl"} |= "sync-2026-07-27", and you are looking at the stack traces from exactly those three batches — not the whole night’s output.

Note the division of labour. Analytics events are small, structured and countable: an event per batch, with a reason as a dimension and a duration as a metric. Log lines are unstructured and verbose: the exception, the stack, the query that timed out. Trying to make either do the other’s job is the usual reason teams end up with expensive logging and useless dashboards.

The other half, in the console

Log streams from the same service, filtered to the run that went wrong.

The Infralytiqs live log tail showing ETL pipeline log lines with one entry expanded The Infralytiqs live log tail showing ETL pipeline log lines with one entry expanded
Logs page

Live tail during a run

Expanding an entry shows its stream labels, so you can see which application and environment produced it. During a nightly run, live mode with a level filter is usually the fastest way to watch a pipeline behave.

Clients

Each one batches, retries with backoff and flushes on shutdown. The Node example is the fullest; the others are the same shape.

/**
 * Infralytiqs server-side client.
 *
 * Batches in memory, flushes on a timer, and drains on shutdown so an
 * in-flight batch is not lost on a rolling deploy.
 */
interface InfralytiqsEvent {
  event_type: string;
  event_subtype?: string;
  event_time?: string;
  user_id?: string;
  anonymous_id?: string;
  session_id?: string;
  page_url?: string;
  country_code?: string;
  device_type?: string;
  device_os?: string;
  utm_source?: string;
  language_iso_code?: string;
  custom_dimensions?: Record<string, string>;
  custom_metrics?: Record<string, number>;
}

interface Options {
  serverUrl: string;
  tenantId: string;
  siteId: string;
  /** Server hard limit is 1000. */
  batchSize?: number;
  flushIntervalMs?: number;
  globalDimensions?: Record<string, string>;
  onError?: (error: unknown, dropped: number) => void;
}

export class Infralytiqs {
  private readonly url: string;
  private readonly batchSize: number;
  private readonly globalDimensions: Record<string, string>;
  private readonly onError: (error: unknown, dropped: number) => void;
  private queue: InfralytiqsEvent[] = [];
  private timer?: NodeJS.Timeout;
  private flushing = false;

  constructor(private readonly options: Options) {
    this.url =
      `${options.serverUrl}/il/analytics/` +
      `${options.tenantId}/${encodeURIComponent(options.siteId)}/events`;
    this.batchSize = Math.min(options.batchSize ?? 100, 1000);
    this.globalDimensions = options.globalDimensions ?? {};
    this.onError = options.onError ?? (() => {});

    this.timer = setInterval(() => void this.flush(), options.flushIntervalMs ?? 5000);
    this.timer.unref?.();   // never hold the process open

    for (const signal of ['SIGTERM', 'SIGINT'] as const) {
      process.once(signal, () => void this.shutdown());
    }
  }

  track(event: InfralytiqsEvent): void {
    this.queue.push({
      // Stamp at capture. If the flush is delayed by a slow batch, the event
      // should still claim the moment it actually happened.
      event_time: new Date().toISOString(),
      ...event,
      custom_dimensions: { ...this.globalDimensions, ...event.custom_dimensions },
    });

    if (this.queue.length >= this.batchSize) {
      void this.flush();
    }
  }

  async flush(): Promise<void> {
    if (this.flushing || this.queue.length === 0) { return; }
    this.flushing = true;

    const batch = this.queue.splice(0, this.batchSize);

    try {
      await this.post(batch);
    } catch (error) {
      // Three attempts have already failed inside post(). Holding the batch
      // forever would grow the heap without bound, so drop it loudly.
      this.onError(error, batch.length);
    } finally {
      this.flushing = false;
    }
  }

  private async post(batch: InfralytiqsEvent[], attempt = 1): Promise<void> {
    const response = await fetch(this.url, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(batch),
      signal: AbortSignal.timeout(10_000),
    });

    if (response.ok) { return; }

    // 4xx other than 429 will never succeed — a malformed payload or an
    // unknown tenant. Retrying is pointless and hides the real problem.
    if (response.status >= 400 && response.status < 500 && response.status !== 429) {
      throw new Error(
        `Infralytiqs rejected ${batch.length} events: ` +
        `${response.status} ${await response.text()}`
      );
    }

    if (attempt >= 3) {
      throw new Error(`Infralytiqs unavailable after ${attempt} attempts: ${response.status}`);
    }

    // Exponential backoff with jitter, so a fleet of instances does not
    // retry in lockstep after a shared outage.
    const delay = 2 ** attempt * 250 + Math.random() * 250;
    await new Promise((resolve) => setTimeout(resolve, delay));
    return this.post(batch, attempt + 1);
  }

  /** Drain everything. Call before the process exits. */
  async shutdown(): Promise<void> {
    if (this.timer) { clearInterval(this.timer); }
    while (this.queue.length > 0) { await this.flush(); }
  }
}

// ── Usage ──────────────────────────────────────────────────────────────
export const analytics = new Infralytiqs({
  serverUrl: process.env.IL_SERVER_URL!,
  tenantId:  process.env.IL_TENANT_ID!,
  siteId:    process.env.IL_SITE_ID!,
  globalDimensions: {
    service:     'checkout-api',
    environment: process.env.NODE_ENV ?? 'development',
    release:     process.env.GIT_SHA ?? 'unknown',
  },
  onError: (error, dropped) =>
    logger.warn({ err: error, dropped }, 'infralytiqs batch dropped'),
});

// Emit from the code that commits the transaction, so the event exists if
// and only if the order does.
await db.transaction(async (tx) => {
  const order = await tx.orders.create(payload);

  analytics.track({
    event_type: 'order_completed',
    user_id: order.customerId,
    custom_dimensions: {
      order_id: order.id,
      plan:     order.plan,
      currency: order.currency,
      country_code: order.billingCountry,
    },
    custom_metrics: {
      order_value: order.total,
      item_count:  order.items.length,
    },
  });
});

Backfilling history

Because event_time is a field you control, importing your existing history is just a batched loop.

"""Backfill historical events from your own database.

event_time is settable, so history is not a special case — it is the same
endpoint with older timestamps.

Before you start: the table ships with TTL event_date + INTERVAL 24 MONTH,
so anything older than that will be dropped at the next merge. Extend the
TTL first if you are importing further back:

    ALTER TABLE website_events
        MODIFY TTL event_date + INTERVAL 120 MONTH DELETE;
"""

import itertools
from datetime import timezone

BATCH = 1000  # server hard limit


def batched(iterable, size):
    iterator = iter(iterable)
    while chunk := list(itertools.islice(iterator, size)):
        yield chunk


def to_event(row):
    return {
        "event_type": "order_completed",
        "event_time": row.created_at.astimezone(timezone.utc)
        .isoformat(timespec="milliseconds")
        .replace("+00:00", "Z"),
        "user_id": str(row.customer_id),
        "country_code": row.billing_country,
        "custom_dimensions": {
            "order_id": str(row.id),
            "plan": row.plan,
            "currency": row.currency,
            "backfill": "true",          # so you can tell imported from live
        },
        "custom_metrics": {
            "order_value": float(row.total),
            "item_count": row.item_count,
        },
    }


rows = db.query(
    "SELECT * FROM orders WHERE created_at >= %s ORDER BY created_at",
    (start_date,),
)

sent = 0
for chunk in batched(rows, BATCH):
    events = [to_event(row) for row in chunk]

    # Send synchronously and stop on failure. A partial backfill you know
    # the boundary of is recoverable; one you do not is not.
    response = requests.post(ENDPOINT, json=events, timeout=30)
    response.raise_for_status()

    sent += response.json()["count"]
    print(f"backfilled {sent} events, through {chunk[-1].created_at}")

print(f"done: {sent} events")

# Verify against your own ClickHouse, and confirm the partitions you expect
# actually exist:
#
#   SELECT toYYYYMM(event_date) AS month, count() AS events
#   FROM website_events
#   WHERE custom_dimensions['backfill'] = 'true'
#   GROUP BY month ORDER BY month;

Server-side questions

No, and every client on this page is built that way — bounded queues, non-blocking enqueue, and dropping with a log line when the queue is full. An outage in your analytics pipeline must never become an outage in your product.
Pass the anonymous_id or session_id from the browser through to your backend — a header on the API call is the usual route — and include it on the server-side event. Then a funnel from click to confirmed payment is one query. Without it, user_id is your join key, which only works after login.
No. event_id defaults to a fresh UUID, so a retried batch inserts duplicates. If exactly-once matters, send a deterministic event_id derived from your own primary key and deduplicate at read time with argMax, or keep that event type in a ReplacingMergeTree table.
One per process, with a global dimension naming the service. That gives each process its own queue and flush timer while letting you group or filter by service in reports. Sharing a client across processes is not possible anyway, since the queue is in memory.
Batching does not help you there. Send synchronously before the handler returns, and accept the added latency — or write events to a queue your own infrastructure owns and forward them from a long-lived consumer, which is usually the better shape.
You can, and you should think carefully first. It is your database, so this is your policy decision rather than ours — but the table keeps rows for 24 months by default, and Map columns are awkward to selectively redact. Sending an opaque id and joining to your own user table at query time is almost always the better choice.

Now run the database properly

The ClickHouse guide covers a production setup: least-privilege users, TLS, retention, backups and what actually breaks.

Logs Collector Custom Dimensions and Metrics