Cloud Platform · production · shared release

DeployAngel checks each production deploy and says when it's safe to stop watching.

Try it free →
Cleared

run-260 cleared for normal traffic at 00:19 EST (17 min after rollout)

Code release · 9381337004ee · released Nov 10, 00:02 EST · deployed by GitHub Actions

  1. Released00:02 EST
  2. Rolled out00:02 EST
  3. Cleared00:19 EST

Cleared for normal traffic, which doesn't cover everything:

  • Not assessed: HTTP p95 latency (needs 200 requests to show it's as fast as before; had 45, enough only to catch a large slowdown, and none was)
  • Not assessed yet: volume of UpdatePathHealthJob, MonitorScheduledJobsJob, GET /api/v1/deployments, and 18 more, too quiet to judge by clearance; compared since the rollout until Nov 10, 21:47 EST
  • Not run since the deploy: critical flows notifications and sign_in
  • Changed within limits: job duration p95 351 ms → 1103 ms (3.1×; it warns at 2.0× and +1000 ms)
  • Not checked: results. A release that returns wrong answers without changing error rates, latency, volumes, or exceptions passes; a checkpoint on what matters catches that.
Confidence
Medium
required signals passed; high needs exact data, not approximate
Coverage
100%
of normally active routes and jobs, run at least once
Critical flows
3 of 5
run or checked since rollout
Cleared in
17 min
after rollout
Baseline
62d2ef6

Compared with baseline

Before is the baseline (62d2ef6), after is run-260 since rollout.

27 signals

✓ No failing or warning signals. No new exceptions.

All signals: 27 (22 passing, 5 without enough data)
Signal Status
Job failure rate

Job failure rate · EvaluateVerificationsJob

Rule: needs 20 attempts

EvaluateVerificationsJob

0.00% → 0.00% · 13 samples

 Not enough data

needs 20 attempts
p95 latency

p95 latency

Rule: needs 200 requests in baseline and observation on routes the baseline saw; on fewer, fails only when >= 20% of requests are over their own route's usual p95 by >= 1.5x and +150 ms at p < 0.000001, sustained 3 evaluations (2.2% were, 0.0% usually)

63 ms → 351 ms · 45 samples

 Not enough data

needs 200 requests in baseline and observation on routes the baseline saw; on fewer, fails only when >= 20% of requests are over their own route's usual p95 by >= 1.5x and +150 ms at p < 0.000001, sustained 3 evaluations (2.2% were, 0.0% usually)
p95 latency

p95 latency · GET /api/v1/deployments

Rule: needs 50 requests in baseline and observation

GET /api/v1/deployments

43 ms → 48 ms · 11 samples

 Not enough data

needs 50 requests in baseline and observation
p95 latency

p95 latency · POST /api/v1/telemetry

Rule: needs 50 requests in baseline and observation

POST /api/v1/telemetry

57 ms → 164 ms · 33 samples

 Not enough data

needs 50 requests in baseline and observation
Route 5xx rate

Route 5xx rate · GET /api/v1/deployments

Rule: needs 20 route requests

GET /api/v1/deployments

0.00% → 0.00% · 13 samples

 Not enough data

needs 20 route requests
Checkpoint

Checkpoint · telemetry.ingested

Rule: expected from the baseline, per request or per job as recorded; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.8

telemetry.ingested

~40 expected → 45

 Pass
HTTP 5xx rate

HTTP 5xx rate

Rule: >= 30 requests, >= 80% of expected routes exercised 3+ times, zero 5xx

0.00% → 0.00% · 86 samples

 Pass approx.
Job duration p95

Job duration p95

Rule: warning only: >= 2.0x and >= +1000 ms

351 ms → 1103 ms · 192 samples

 Pass
Job failure rate

Job failure rate

Rule: failing: p < 0.001 and >= 2x and >= +1 pp; warning: p < 0.01 and >= 1.5x

p = 1

0.00% → 0.00% · 192 samples

 Pass
Job failure rate

Job failure rate · IngestTelemetryJob

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

IngestTelemetryJob

0.00% → 0.00% · 158 samples

 Pass
Recurring job

Recurring job · clear_solid_queue_finished_jobs

Rule: expected by 01:27 EST (declared schedule)

clear_solid_queue_finished_jobs

every hour → ran

 Pass
Recurring job

Recurring job · evaluate_verifications

Rule: expected by 00:28 EST (declared schedule)

evaluate_verifications

every minute → ran

 Pass
Recurring job

Recurring job · ingest_telemetry

Rule: expected by 00:27 EST (declared schedule)

ingest_telemetry

every 5 seconds → ran

 Pass
Recurring job

Recurring job · monitor_error_spikes

Rule: expected by 00:30 EST (declared schedule)

monitor_error_spikes

every 5 minutes → ran

 Pass
Recurring job

Recurring job · monitor_new_errors

Rule: expected by 00:30 EST (declared schedule)

monitor_new_errors

every 5 minutes → ran

 Pass
Recurring job

Recurring job · monitor_scheduled_jobs

Rule: expected by 00:30 EST (declared schedule)

monitor_scheduled_jobs

every 5 minutes → ran

 Pass
Recurring job

Recurring job · monitor_silence

Rule: expected by 00:30 EST (declared schedule)

monitor_silence

every 5 minutes → ran

 Pass
Recurring job

Recurring job · monitor_volume

Rule: expected by 00:30 EST (declared schedule)

monitor_volume

every 5 minutes → ran

 Pass
Recurring job

Recurring job · record_daily_stats

Rule: expected by 01:20 EST (declared schedule)

record_daily_stats

every hour → ran

 Pass
Recurring job

Recurring job · update_path_health

Rule: expected by 00:30 EST (declared schedule)

update_path_health

every 5 minutes → ran

 Pass
Queue backlog

Queue backlog

Rule: above 2x the largest backlog before the release and +100, with workers busy, and higher than 5 minutes earlier; failing: 15 minutes in a row; warning: 5

3 → 0 · 10 samples

 Pass
Queue latency p95

Queue latency p95

Rule: warning only: >= 3.0x and >= +30000 ms

1002 ms → 1103 ms · 192 samples

 Pass
Route 5xx rate

Route 5xx rate · POST /api/v1/telemetry

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

POST /api/v1/telemetry

0.00% → 0.00% · 42 samples

 Pass
Volume

Volume · all jobs

Rule: the usual job runs for these hours, scaled down by web traffic when it's below usual; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.9

all jobs

~62 expected → 72

 Pass
Volume

Volume · EvaluateVerificationsJob

Rule: expected from the baseline, scaled by the rest of the app's job runs, and at most the usual count for these hours; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.53

EvaluateVerificationsJob

~13 expected → 13

 Pass
Volume

Volume · IngestTelemetryJob

Rule: expected from the baseline, scaled by the rest of the app's job runs, and at most the usual count for these hours; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.39

IngestTelemetryJob

~162 expected → 158

 Pass
Volume

Volume · POST /api/v1/telemetry

Rule: expected from the baseline, scaled by the rest of the app's requests, and at most the usual count for these hours; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.42

POST /api/v1/telemetry

~44 expected → 42

 Pass

What production exercised

What normally runs during this window, from the last 14 days. Each item counts once however busy it is, and one run is enough. Clearing needs 50% coverage, and high confidence needs 80%.

Normally active routes 2 of 2 · 100%
Regular job classes 2 of 2 · 100%
Recurring jobs due 10 of 10 · 100%

Not observable: none.

Critical flows

Flows that matter more than their traffic suggests, counted since this release rolled out.

  • Deploy registration

    POST /api/v1/deploymentsPOST /integrations/heroku/webhooks/:token

    Ran once

  • Notifications

    job:DeliverNotificationJob

    Not run yet

  • Sign in

    POST /session

    Not run yet

  • Telemetry ingestion

    POST /api/v1/telemetrycheckpoint:telemetry.ingested

    Ran 87 times

  • Verification

    job:EvaluateVerificationsJob

    Ran 13 times

Rarely used, not yet exercised

Not covered by this clearance. DeployAngel watches their first use.

  • AnalyzeReleaseJob last used about 10 hours ago

History

  1. First telemetry from this release

    Nov 10, 00:04 EST

  2. Confidence Low → Medium

    Nov 10, 00:10 EST

  3. Cleared, still watching recurring jobs

    Nov 10, 00:19 EST

  4. Stopped watching: a newer release rolled out

    Nov 10, 21:46 EST