Alerts

The Alerts page tells RAGGuard where to send a message when an audit finds something new, and shows what happened to every message. You give it one alert webhook (a web address that accepts messages, such as a Slack incoming webhook) and a minimum severity. After each finished audit RAGGuard sends one message listing the issues that are new, that came back, or that got worse and that are at or above that severity. This chapter explains the card, both fields, how to create a Slack webhook, exactly when a message is and is not sent, what the message contains, and how to read the Recent deliveries table.

On this page


Purpose, who needs it, how to open it

Purpose. You should not have to open RAGGuard every day to learn that something broke. With an alert webhook, the news comes to the place your team already watches.

Who needs it. Anyone who wants to be told about new problems, usually the owner of the pipeline or an on-call channel.

How to open it. Choose Alerts in the left menu. The page header reads "Get told when an audit finds something new."


The page at a glance

The Alerts page: the Alert webhook card with its two fields and the Check settings and Save buttons

The Alert webhook card on the left, with Webhook URL and Minimum severity.

From top to bottom:

  1. A two-column area on a wide screen (stacked on a narrow one): the Email and webhook card on the left (the screenshots below still call it Alert webhook) and the How alerts work card on the right.
  2. Recent deliveries: a table with a Show filter.
  3. More ways to be notified: two dashed cards marked Not available yet.

Notifications and email

After every audit that finishes, RAGGuard tells you what happened and what to do next, in two places.

In the app. The bell at the top right shows a red number when you have unread notifications. Click it to see the latest ten; See all notifications opens the Notifications page. Each notification says what the audit found and has buttons for the next step:

The audit...The buttons
finished with no issuesSee the run
finished with issuesReview critical issues (or high issues, or all of them), Verify deletions (only when leftover vectors of deleted documents were found), See the run
stopped part-wayResume audit, Review the issues found so far, See the run
failedRun it again, Check connections, See the run

Resume audit continues the audit from where it stopped. Run it again starts a new audit. Verify deletions checks that deleted documents are really gone from the index and tells you what it found. After you press a button that does something, the notification shows it as done.

By email. The same message is emailed to the sign-in address of everyone in your workspace. The Email me setting chooses when: always (after every audit, the default), attention (only when an audit failed, stopped part-way or found a new issue at or above the minimum severity) or never. A link in the email opens a page. A button that does something (Resume audit, Run it again, Verify deletions) opens the Notifications page, where the button is, so opening an email can never change anything by itself.

Who sets up the mail. Email is sent from one mail account that the operator of this RAGGuard sets up. If the operator has not set it up, no email is sent, the notification in the app still appears, and the audit's record says why. At most 12 emails an hour are sent per workspace.


The Alert webhook card

The header reads "Email and webhook", with the line "Choose when you get the email. The optional Slack-compatible webhook is saved encrypted and never shown again." and a badge: Configured (green) or Not configured (grey). Configured means you have saved at least one value in this group. It does not prove the webhook works, and saving only Minimum severity also counts as configured.

The form is on the card itself (there is no dialog). Its parts, top to bottom:

ElementWhat it does
Email meWhen the email after an audit is sent: always, attention or never
Webhook URLThe address messages are posted to (secret)
Minimum severityThe lowest severity that triggers a message
Saved line"Not saved yet." or "Saved 3 h ago." (hover for the exact time). No test history is kept for this group
Note"Checks that the values are valid and the address is allowed. It does not send a message."
"No unsaved changes."Appears when the form matches what is saved. It disappears as soon as you change something
Check settingsValidates the values without sending anything (see below)
SaveStores your changes. Greyed out until something changed. A message "Alerts saved" appears when it works

The shared form controls (required markers, the secret box and Remove saved value, inline errors) work as described in Connect your data sources.

The fields

LabelRequiredSecretDefaultExample
Email meNoNoalwaysattention
Webhook URLNoYesNonehttps://hooks.slack.com/services/T0000/B0000/XXXX
Minimum severityNoNohigh (shown as Use server default)high

Email me (drop-down: always, attention, never)

  • What it is: when RAGGuard emails you after an audit finishes. The notification in the app is always created, whatever you choose.
  • What it changes: always emails after every audit. attention emails only when an audit failed, stopped part-way, or found a new, reopened or escalated issue at or above the minimum severity. never sends no email.
  • Common mistakes: expecting email when the operator has not set up a mail account (the audit's record then says why no email was sent).

Webhook URL (secret)

  • What it is: the address of a Slack-compatible incoming webhook. RAGGuard sends it one JSON message per audit (when there is something to say). It is treated as a secret because the address itself acts as the password: anyone who has it can post to your channel. It is stored encrypted and never shown again.
  • What it changes: with an address, alerts are on. With none, alerts are off, and each audit records "No alert webhook was configured." To switch alerts off later, press Remove saved value under the field, then Save.
  • Rules: it must start with https:// or http://, must not contain a user name or password, and must point to a public host (see public addresses). Because the field is secret, the browser only checks that it is not empty; the server checks the format when you save or press Check settings, and a rejection appears in a red box starting with "Webhook URL: ".
  • Common mistakes: pasting a Slack channel link instead of a webhook address; an address from a deleted or revoked webhook (deliveries then fail); a private address.

Minimum severity (drop-down: Use server default, critical, high, medium, low)

  • What it is: the lowest severity that is worth a message. The order, highest first, is critical, high, medium, low.
  • What it changes: an issue only triggers a message if its severity is at or above this level. critical means only the most serious. low means almost everything. The default, high, covers critical and high. If you leave the drop-down on Use server default, high applies.
  • Common mistakes: choosing low and being flooded by minor bookkeeping issues; choosing critical and missing high-severity problems such as missing or stale vectors.

Check settings

Check settings answers one question: "are these values acceptable?" It checks the format, and that the address is allowed. It does not post a message, so it cannot tell you whether Slack will accept the webhook. A pass shows a green Test passed panel with the line "Settings: valid". If you have saved nothing and typed nothing, you see "Nothing to check yet. Fill in a field first." A rejected value (for example a private address) shows a red message instead, for example "Webhook URL: localhost is not a public host name."

To prove that the whole chain works, post to your webhook yourself with the command in Set up a Slack incoming webhook, or lower the minimum severity temporarily and run an audit that finds something new.

What Save does

Only the fields you changed are stored. The webhook address is encrypted. A message "Alerts saved" appears, and the card's badge and saved line update. The new settings are used by the next audit.


Set up a Slack incoming webhook

These steps use Slack's own screens, which may be named slightly differently over time. Slack's documentation for "incoming webhooks" is the authority.

  1. In Slack, open the apps page (api.slack.com/apps) and create a new app from scratch. Give it a name such as "RAGGuard alerts" and choose your workspace.
  2. In the app's settings, open Incoming Webhooks and turn the feature on.
  3. Choose Add New Webhook to Workspace, pick the channel that should receive the alerts and allow it.
  4. Copy the Webhook URL. It starts with https://hooks.slack.com/services/.
  5. In RAGGuard, open Alerts, paste it into Webhook URL, choose a Minimum severity, press Check settings, then Save.
  6. Optional: confirm that Slack accepts the address by posting a test message from a terminal (replace the address with yours):
curl -X POST -H "Content-Type: application/json" \
  -d '{"text": "Test message from my terminal"}' \
  "https://hooks.slack.com/services/T0000/B0000/XXXX"

If the message appears in the channel, the webhook works.

Other tools. Any tool that accepts Slack-style incoming webhooks (a POST with JSON that has a text field) can receive RAGGuard's alerts. Check that tool's documentation for how to create the address. RAGGuard also sends a few extra fields next to text; a tool that is strict about unknown fields may not accept them.


How alerts work

The How alerts work card with its five points and the request body by shape

The How alerts work card: five points and the shape of the request body.

The card lists six points. The first, In the app and by email, is explained under Notifications and email above. The other five are about the Slack webhook; here they are in plain words, with the details behind each.

1. When it runs

After every audit that finishes, RAGGuard compares the issues that audit found with the issues it already knew about. That includes full audits, scoped audits (one tenant or a few documents) and the verification audits that run after a repair. A scoped audit can only report on its own scope. An audit that fails does not alert.

2. What triggers an alert

Only issues that are new, reopened or escalated, and only if their severity is at or above the minimum severity. An issue that is still open and unchanged never alerts again.

SituationAlert?
An audit finds a problem RAGGuard has not recorded before, at or above the minimum severityYes (new)
A problem that was marked resolved is found againYes (reopened)
An open issue's severity goes up (for example medium to high) and the new severity is at or above the minimumYes (escalated)
An open issue stays open at the same severityNo
An open issue's severity goes downNo
An issue you ignored appears againNo
An issue is below the minimum severityNo

"New" means new to RAGGuard's record, not new to your webhook. If earlier audits already recorded an issue, adding a webhook later does not make that issue alert. If you save the webhook before the first audit, that first audit treats everything it finds as new, so expect one larger message.

3. What is sent

One JSON request to your webhook. The text field is a readable summary for Slack:

*RAG Integrity*: 3 new issue(s) at or above `high` (health 87%)
• [CRITICAL] Document has no embeddings in the index (pipeline claims success) - `acme/policies/leave.pdf`
• [HIGH] Embeddings are from an older version of the document - `acme/runbooks/restart.md`
• [HIGH] ...

(The numbers, titles and document names here are only an example, and the product separates the title from the document with a long dash. The first line says how many issues, the minimum severity and the health score of that audit. After it come up to 10 issues, most severe first, each as the severity in capitals, the title and tenant/document. If there are more than 10 a last line says "and N more". The summary still uses the name RAG Integrity, the product's earlier name.)

The same request carries more structured data, for tools that read it:

{
  "text": "<short summary for Slack>",
  "run_id": "<audit run id>",
  "summary": { "new_issues": 3, "health_score": 87 },
  "issues": [
    { "id": "...", "type": "...", "severity": "...", "tenant_id": "...", "document_id": "...", "title": "..." }
  ]
}
FieldMeaning
textThe readable summary described above
run_idThe audit run that found these issues. Open it in Audit history
summary.new_issuesHow many issues qualified (not capped)
summary.health_scoreThe health score of that audit, as a percentage
issuesUp to 25 qualifying issues. Each has id, type (the issue type, for example missing_embedding), severity, tenant_id, document_id and title

4. If delivery fails

RAGGuard makes one attempt and waits up to 10 seconds. It does not retry. A failed delivery never fails the audit: the error is stored on the run and listed under Recent deliveries, and the audit and its results are unaffected. Fix the webhook and the next audit that has something new will deliver.

5. When nothing is sent

If no webhook is saved, or no issue qualifies, nothing is sent, and the reason is recorded on the run (it shows in the Detail column below).


Recent deliveries

The Recent deliveries table with a Show filter, the Audit run, Outcome, Issues included and Detail columns

Recent deliveries: what happened to the alert after each of the latest audits.

This card shows "What happened to the alert after each of the latest audits." It reads the 50 most recent audits and lists the ones that recorded an alert result. An audit that failed has none, so it is not listed.

The Show filter

One drop-down labelled Show:

ChoiceLists
Delivered or failed (the default)Audits where RAGGuard tried to send something and either succeeded or failed
Every auditEvery audit with an alert result, including those where nothing was sent. Use it to find out why nothing arrived
DeliveredOnly successful deliveries
FailedOnly failed deliveries
Not sentAudits where nothing was attempted

When the filter is not the default, a Clear filters button appears. The choice is kept in the page address (as outcome), so you can bookmark or share a view.

The columns

ColumnWhat it showsHow to read it
Audit runThe date and time the audit started, as a link, with a line under it such as "Full audit, 5 min ago" (the kind is Full, Scoped or Verify)Click the date to open that run in Audit history. Sortable. The table starts with the newest first
OutcomeA badge: Delivered (green), Failed (red) or Not sent (grey)See below. Sortable
Issues includedThe number of issues in the message, or "None"How many issues the alert listed. Sortable
DetailA sentenceThe reason or the error. Red for a failure

Outcomes and the text in Detail

OutcomeMeaningText in Detail
DeliveredThe webhook accepted the message"Sent to the alert webhook."
FailedRAGGuard tried and the webhook did not accept it, or could not be reached, or took longer than 10 secondsThe error: its type, then the first part of its message with every web address removed, for example HTTPStatusError: Client error '404 Not Found' for url '<webhook>'. See the note below
Not sentNothing was attempted"No alert webhook was configured." (no address was saved), or "No new issues at or above the minimum severity." (the audit found nothing that qualifies), or the recorded reason, or "Nothing to send."

Common reasons for a failure. A revoked or deleted Slack webhook (the webhook answers with an error status), a mistyped address (not found), a network problem, or a slow endpoint (more than 10 seconds).

The address is never shown. The webhook address is a secret, and the error a web library produces quotes it. RAGGuard replaces it before the error is stored, shown or logged: the address you saved appears as <webhook>, and any other web address in the message (including where a redirect pointed, even a relative one) appears as <url>. The error type and the status code stay, so you can still tell a 404 (deleted or mistyped address) from a 500 or a refused connection. The same applies to the error of a failed repair delivery. Audits recorded by an earlier version of RAGGuard could hold the address in this column; RAGGuard now removes any web address from that text whenever it reads a run, so old rows are cleaned too. If you ever pasted such a run into a ticket or a chat, rotate the webhook.

Paging and counts

Under the table: "Showing 1 to 10 of N" and the note "From the latest 50 audits.", a Rows drop-down (10, 25, 50 or 100; it starts at 10) and Previous page and Next page arrows with "Page 1 of N".

Empty states

WhereWhat you see
No audit has an alert result at all"No alerts have been sent yet": "After every audit RAGGuard checks for new issues. Once an alert webhook is saved and an audit finds something new, each delivery is listed here."
Audits exist, but none attempted an alert, and the filter is the default"No alerts have been sent yet": "Audits have run, but none produced an alert. Choose Every audit to see why."
You chose another filter and nothing matches"No audits match this filter": "Choose another outcome to see more."
The audits could not be loaded"We couldn't load this" with a Try again button

While loading, grey placeholder rows are shown.


More ways to be notified: not available yet

Two dashed cards at the bottom are each marked Not available yet. Nothing here can be configured.

CardWhat it says
Per-issue-type rules"The only filter is the minimum severity."
Digest and quiet hours"An alert is sent right after each audit that finds something new."

Single-user mode

On a server that runs for one team without accounts (live or demo mode) the card is read-only. It shows "Email and webhook" with the line "Set in the server's environment." and the note "Configured by the server's environment variables (RAGI_*)": for email, RAGI_SMTP_HOST, RAGI_SMTP_FROM (and RAGI_SMTP_USER, RAGI_SMTP_PASSWORD if the server needs a login), RAGI_ALERT_EMAIL_TO for who gets it, and RAGI_ALERT_EMAIL_WHEN; for Slack, RAGI_ALERT_WEBHOOK_URL, and RAGI_ALERT_MIN_SEVERITY to choose the minimum severity. How alerts work, Recent deliveries and the Not available yet cards work the same way.


States you may see

StateWhat you seeWhat to do
LoadingA tall grey placeholder where the card goes, and grey rows in the tableWait a moment
Not configuredA grey Not configured badge, and every audit records "No alert webhook was configured."Paste a webhook address and Save
ConfiguredA green badgeCheck Recent deliveries after the next audit
A rejected valueA red box under the form beginning with "Webhook URL: "Fix the address
Cannot load"We couldn't load this" and Try againPress Try again

Tips and common mistakes

  • Start with high. It keeps the channel quiet enough that people read it. Lower it only if you want to see everything.
  • A steady state is silent. If your channel is quiet, either nothing new happened or nothing reached the minimum severity. Choose Every audit under Show to see which.
  • Check settings does not send anything. To test delivery, use the curl command above.
  • The webhook address is a secret. If it leaks, delete the webhook in Slack, create a new one and replace it here.
  • Hosted RAGGuard does not run audits on a timer. To get alerts every morning, schedule a call to the API with an API key (see Integrations).
  • To switch alerts off, use Remove saved value under Webhook URL, then Save.