Alerts
Configure metric-based alerts to get notified when your applications experience issues like high CPU usage, memory problems, or other conditions.
Overview
The alerting system in Quave ONE consists of two components:
- Contact Points - Notification destinations where alerts are sent (Slack, Microsoft Teams, Google Chat, PagerDuty, Webhook, Email)
- Alerts - Conditions that trigger notifications when met
Contact points are managed at the account level and can be shared across all environments in your account. Alerts are configured per environment.
Setting Up Alerts
Step 1: Create a Contact Point
Before creating alerts, you need at least one contact point to receive notifications.
- Open Contact Points from the main navigation
- Click Create Contact Point
- Choose a type and provide the required configuration
After saving a Slack contact point, use its Test action to send a real notification to the configured destination. Confirm with the destination owner before testing. Test is shown only for Slack.
Available Contact Point Types
Slack
Send alerts to a Slack channel via webhook
Setup: Create an Incoming Webhook in your Slack workspace (Apps → Incoming Webhooks → Add New Webhook). Alert payloads follow the Grafana alert format and are forwarded directly to your channel.
Required: webhookUrl | Optional: channel, mentionUsers
Microsoft Teams (Workflows webhook)
Send alerts to a Microsoft Teams channel through a Workflows (Power Automate) webhook
Setup: In Teams, open the channel → Workflows → "Post to a channel when a webhook request is received" (Power Automate "When a Teams webhook request is received") and copy the HTTPS URL it generates. The old Office 365 connector URLs (webhook.office.com) were retired by Microsoft and are not supported.
Required: webhookUrl
Google Chat
Send alerts to a Google Chat space via an incoming webhook
Setup: In Google Chat, open the space → Apps & integrations → Webhooks → Add webhook, then copy the URL (it starts with https://chat.googleapis.com/v1/spaces/...).
Required: webhookUrl
PagerDuty
Send alerts to PagerDuty for incident management
Setup: Create an Events API v2 integration: Go to Integrations → Generic Integrations (v2), click New Integration, name it (e.g., "Quave ONE Alerts"), then copy the Integration Key.
Required: integrationKey | Optional: severity
Custom Webhook
Send alerts to any HTTP endpoint
Setup: Provide your webhook URL and optionally configure HTTP method and Basic Auth credentials. See the Grafana webhook documentation for payload format details.
Required: url | Optional: httpMethod, username, password
Send alerts via email
Setup: Enter one or more email addresses (comma-separated) to receive alert notifications.
Required: addresses
Step 2: Create an Alert
- Navigate to your app environment
- Go to the Alerts tab
- Click Create Alert
- Configure the alert condition and select a contact point
Alert Configuration
Metrics
Choose what to monitor. Metrics come in five categories:
Combined Metrics (Aggregate)
These metrics combine values across all pods/containers, giving you an overall view:
| Metric | Description | Default Threshold |
|---|---|---|
| CPU % (combined) | CPU usage as percentage of allocated limit | 80% |
| Memory % (combined) | Memory usage as percentage of allocated limit | 80% |
| CPU millicores (combined) | CPU usage in millicores (1000 millicores = 1 CPU core) | 500m |
| Memory MB (combined) | Memory usage in megabytes | 512MB |
Individual Metrics (Per Pod)
These metrics return values for each pod separately. Use with max/min aggregation to detect issues in specific pods:
| Metric | Description | Best For |
|---|---|---|
| CPU % (individual) | CPU usage per pod - use with max aggregation to find busiest pod | Finding the busiest pod |
| Memory % (individual) | Memory usage per pod - use with max aggregation to find busiest pod | Finding memory-heavy pods |
| CPU millicores (individual) | CPU usage per pod in millicores | Finding the busiest pod |
| Memory MB (individual) | Memory usage per pod in MB | Finding memory-heavy pods |
Ingress Metrics (HTTP Traffic)
These metrics come from the NGINX Ingress Controller and monitor HTTP traffic patterns hitting your application. They are not available for database environments.
| Metric | Description | Default Threshold |
|---|---|---|
| Request Rate (RPS) | Total HTTP requests per second through the ingress | 1000req/s |
| 5xx Errors (per second) | Server error (5xx) responses per second | 1err/s |
| 5xx Error % | Percentage of requests returning 5xx status codes | 5% |
| 4xx Errors (per second) | Client error (4xx) responses per second | 10err/s |
| 4xx Error % | Percentage of requests returning 4xx status codes | 10% |
| P95 Response Time | 95th percentile response time in seconds | 2s |
| P99 Response Time | 99th percentile response time in seconds | 5s |
| P95 Response Time (GET, no WebSocket) | 95th percentile response time of GET requests in seconds, excluding WebSocket upgrades and long-poll POSTs | 2s |
| P99 Response Time (GET, no WebSocket) | 99th percentile response time of GET requests in seconds, excluding WebSocket upgrades and long-poll POSTs | 5s |
Database Metrics (MongoDB)
These metrics come from the database exporter and are only available for MongoDB environments.
| Metric | Description | Default Threshold |
|---|---|---|
| MongoDB Replication Lag (seconds) | Seconds the most lagging replica set member trails the primary's last applied operation, as seen by the primary. Idle replica sets read up to about 20s because the primary records a no-op only every 10s | 60s |
Storage Metrics
These metrics come from the persistent volume statistics and are only available for environments with their own persistent volume: databases with storage and apps with a local volume. Shared volumes are not supported.
| Metric | Description | Default Threshold |
|---|---|---|
| Disk usage % | Used space of the environment's fullest persistent volume, as a percentage of its capacity | 85% |
Individual metrics are especially useful for databases where the primary may be busy while replicas are idle. Use with Worst (Max) aggregation to alert when any single pod exceeds the threshold.
Combined percentages compare the usage of the running pods with those same pods' limits. During a rolling deploy, old and new pods both count on each side, so the overlap does not inflate the value. A spike on a single pod raises the combined value only by its share, so use an individual metric to catch it.
To detect a sudden drop in traffic (e.g., after a bad deployment), use the Request Rate (RPS) ingress metric with the less than operator. For example: "Request Rate less than 10 req/s for 5 minutes."
Ingress (HTTP Traffic) metrics are only available for web application environments. They are not shown for database environments since databases do not receive traffic through the NGINX Ingress Controller.
WebSocket connections are recorded as one request that lasts until the socket closes, and long-poll requests (for example Meteor's SockJS fallback) are held open for up to 30 seconds. On apps that use them, P95 and P99 sit at the 10-second top of the latency histogram, so a P95 or P99 alert either fires constantly or, with a higher threshold, never fires. Use P95/P99 Response Time (GET, no WebSocket) instead. It measures only GET requests and skips WebSocket upgrades, so it follows page, asset and API read latency.
MongoDB Replication Lag shows how many seconds the most lagging member of the replica set trails the primary. An idle replica set reads up to about 20 seconds, because the primary records a no-op only every 10 seconds. A threshold of 60 seconds with Average aggregation over 10 minutes catches a secondary that stops keeping up without paging on idle periods. The same value is plotted as Replication Lag in the MongoDB Grafana dashboard.
Disk usage % is the used space of the environment's fullest persistent volume, as a percentage of its capacity. It is offered for databases with storage and for apps with a local volume, whether or not disk autoresize is enabled. Environments without their own volume, including those using a shared volume, cannot use it. The default threshold of 85% matches the in-app Volume almost full notification, but this alert is delivered to your contact points. The same value is plotted in the Storage Volumes panel of the Grafana dashboard.
Condition
Define when the alert should fire:
Operators
| Operator | Symbol | Description |
|---|---|---|
gt | > | Greater than |
lt | < | Less than |
Threshold
The value to compare against. Can be any number appropriate for the metric.
Duration
How long the condition must be true before the alert fires. This helps prevent false alarms from brief spikes.
| Duration | Description |
|---|---|
1m | 1 minute |
5m | 5 minutes (default) |
10m | 10 minutes |
15m | 15 minutes |
30m | 30 minutes |
1h | 1 hour |
For example, "CPU greater than 80% for 5 minutes" only fires if CPU remains above 80% continuously for 5 minutes.
Aggregation
When multiple containers exist, aggregation determines how to combine their values:
| Aggregation | Description | Use Case |
|---|---|---|
| Average (default) | Average across all containers | General monitoring |
| Worst (Max) | Highest value among containers | Detect problems in any container |
| Best (Min) | Lowest value among containers | Ensure all containers are active |
| Last | Most recent value | Real-time monitoring |
Query Range
How far back to look for metric data when evaluating the alert:
| Range | Description |
|---|---|
5m | Last 5 minutes |
10m | Last 10 minutes (default) |
15m | Last 15 minutes |
30m | Last 30 minutes |
1h | Last 1 hour |
2h | Last 2 hours |
Longer query ranges smooth out short spikes but may delay detection of new issues.
Alert States
Alerts have five possible states:
| State | Description | Color |
|---|---|---|
| Normal | Alert condition is not met | green |
| Pending | Alert condition met, waiting for duration | yellow |
| Firing | Alert is active and notifications sent | red |
| No Data | No data received for alert query | gray |
| Error | Error evaluating alert query | orange |
Missing Data and NaN Handling
Some metrics can produce NaN (Not a Number) values when there is no traffic — for example, error percentages involve a division that becomes 0/0, and latency percentiles cannot be computed from an empty histogram.
Quave ONE handles this automatically at two levels:
- Query level: Percentage queries guard the denominator so that zero traffic returns no data instead of NaN. Latency queries filter out NaN results.
- Evaluation level: The reduce step replaces any remaining NaN with 0 before comparing against your threshold. This prevents false-positive alerts when your application has no traffic.
This means:
- "Greater than" alerts (the most common case) will not fire during zero-traffic periods, since 0 is below any reasonable threshold.
- "Less than" alerts (e.g., request rate drops below 10 req/s) will correctly fire when traffic is zero, since that is a genuine drop.
- Container metrics (CPU, memory) are unaffected because Kubernetes always reports values for running pods.
If the monitoring system has no data at all for your environment, or if the Prometheus query fails or times out, Quave ONE treats that evaluation as OK instead of sending a threshold alert. This prevents monitoring outages from generating false-positive customer alerts. The alert fires only when the metric query succeeds and the measured value breaches your threshold.
Managing Alerts
Enable/Disable Alerts
You can temporarily disable an alert without deleting it. Disabled alerts:
- Stop evaluating the condition
- Don't send notifications
- Preserve your configuration for when you re-enable
This is useful during maintenance windows or known issues.
Refresh Alert State
Click the refresh button to get the latest state directly from the monitoring system. This is helpful when:
- Verifying an alert is back to normal after fixing an issue
- Getting the latest state without waiting for automatic refresh
Delete Alerts
Deleting an alert permanently removes it. This action cannot be undone.
Best Practices
Start with Basic Alerts
Begin with these essential alerts:
- CPU Alert: CPU % (combined) greater than 80% for 5 minutes
- Memory Alert: Memory % (combined) greater than 80% for 5 minutes
- 5xx Error Alert: 5xx Error % greater than 5% for 5 minutes
- Latency Alert: P95 Response Time greater than 2s for 5 minutes
Use Appropriate Durations
- 1 minute: Only for critical, time-sensitive issues
- 5 minutes: Good default for most alerts
- 15-30 minutes: For non-urgent, trend-based alerts
Choose the Right Aggregation
- Use Average for overall health monitoring
- Use Worst (Max) with individual metrics to catch problems in any single pod
- Use Best (Min) to ensure all pods are functioning
Organize Contact Points
- Create dedicated contact points for different severity levels
- Use Slack for informational alerts
- Use PagerDuty for critical, on-call alerts
- Consider separate channels for different environments (staging vs production)
Avoid Alert Fatigue
- Set thresholds that represent actual problems
- Use appropriate durations to filter noise
- Start with fewer alerts and add more as needed
- Review and tune alerts regularly
Programmatic Access
You can manage alerts programmatically via:
- Public API - REST API for automation
- MCP Tools - AI-assisted management
Troubleshooting
Alert Shows "No Data"
- Verify the environment has running containers
- Check that metrics are being collected (view in Metrics dashboard)
Alert Not Firing
- Check if the alert is enabled
- Verify the threshold is set correctly
- Ensure the duration has passed since the condition was met
- Click refresh to get the latest state
Contact Point Not Receiving Notifications
- Use Test on the Contact Points page to send a real notification to a supported destination and see the result immediately.
- A Test checks the destination only when you request it. It is separate from Grafana's delivery-failure detection and never pauses the contact point.
- Verify the contact point configuration (webhook URL, API key, etc.)
- Check that the contact point is selected in the alert
- For Slack, ensure the webhook URL is valid and the channel exists. If Slack still sends to the webhook's original channel, create the incoming webhook for the intended destination channel because modern Slack webhooks can ignore channel overrides.
- For PagerDuty, verify the integration key is correct
- If the Contact Points page shows Delivery paused, Slack reported that the webhook is no longer active. Automations still run and the result stays in the dashboard. Update the destination and save to resume delivery. Grafana will not keep posting to the dead webhook while the contact point is paused.
- Grafana alert delivery death is detected on a daily last-notify poll, not instantly. Grafana stores only the last error, in memory, so a quiet dead webhook stays unnoticed until an alert tries to notify, and HA Grafana can miss or overwrite that error.