Active
TMM Created
Disk Utilization & Remediation
Proportional disk response with switch routing
Description
Servers run out of disk space at different urgency levels. At 72% you do nothing. At 85% you reclaim cache and old logs before it hurts. At 97% you add capacity. On EC2, expand the EBS volume, grow the partition, and extend XFS. Each path posts a tier-specific Mattermost message so the team knows what ran.
Why it matters
- One check playbook and one switch replace a nest of conditional jobs. Disk tier drives the right response automatically.
- {"Remediation scales with severity"=>"log and exit at low usage, cleanup at warning, expand capacity at critical instead of one-size-fits-all scripting."}
- Tier-specific Mattermost notifications tell operators exactly what ran, so on-call does not have to dig through job logs.
AO building blocks
Automation orchestrator step types this workflow uses on the canvas. See all building blocks
-
Triggers
Manual trigger
Starts a workflow when a user clicks Run in the UI or calls the workflow execution API — the simplest entry point for on-demand runs like demos, break-glass remediation, or operator-initiated changes. No extra configuration is required, and runs appear in history with the triggering user as the source.
-
Logic
Switch step
Evaluates conditions and routes execution to the first matching branch, supporting up to 100 ordered paths plus a fallback when nothing matches. Use it to branch on severity tiers, environment (dev/staging/prod), or AI agent output categories where a binary conditional isn't enough.
-
AAP execution
AAP job template
Launches a single Ansible Automation Platform job template from the workflow, polling until the job completes and exposing results to downstream steps. The standard bridge between automation orchestrator reasoning and AAP execution — one playbook with resolved inventory, credentials, limits, and extra vars.
Interactive Experience
Workflow
flowchart LR
A@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/manual-trigger.png", label: "Manual trigger", pos: "b", h: 96, constraint: "on" } --> B@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Disk Utilization Check", pos: "b", h: 96, constraint: "on" }
B --> C@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/switch-node.png", label: "Switch on disk_use_percent", pos: "b", h: 96, constraint: "on" }
C -->|"< 80%"| D@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Continue", pos: "b", h: 96, constraint: "on" }
C -->|"80–95%"| E@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Cleanup logs & cache", pos: "b", h: 96, constraint: "on" }
C -->|"> 95%"| F@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Expand EBS volume", pos: "b", h: 96, constraint: "on" }
C -->|default| G@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Fallback: manual review", pos: "b", h: 96, constraint: "on" }
D --> H@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Notify", pos: "b", h: 96, constraint: "on" }
E --> H
F --> H
G --> H
classDef noBorder fill:transparent,stroke:transparent
class A,B,C,D,E,F,G,H noBorder
Demo walkthrough — test every switch path
You do not need to fill the disk to show the switch. The Check job accepts a test_disk_use_percent extra var that skips live df and publishes a simulated disk_use_percent artifact for routing.
- In automation orchestrator, open the Disk Utilization Check step.
- Under extra vars, set
test_disk_use_percentto one of the values in the table below (number, not a string). - Run the workflow. The switch reads
disk_use_percentfrom Check and takes the matching branch — Continue, Cleanup, Expand, or Fallback. - Change the value and re-run to walk the next path. Clear or remove
test_disk_use_percentwhen you want a realdfcheck.
Tip: The exported workflow ships with test_disk_use_percent set to 50 so a first run safely takes Continue. Change it before you demo Cleanup or Expand.
| Path | test_disk_use_percent | Switch condition | What runs |
|---|---|---|---|
Continue |
75 |
< 80% | Log OK → Mattermost (healthy) |
Cleanup |
85 |
80–95% | Reclaim dnf cache + old logs → Mattermost (warning) |
Expand |
96 |
> 95% | Grow EBS + filesystem → Mattermost (critical) |
Fallback |
omit or non-numeric |
default | Manual-review path → Mattermost (unsupported) |
Playbooks
| Playbook | What it does | Runs on |
|---|---|---|
check_disk.yml
|
Reads filesystem usage on the target mount and publishes disk_use_percent and disk_tier artifacts for the switch. | RHEL EC2 node |
remediate_disk_continue.yml
|
Healthy path when disk is below the warning threshold. Logs status and publishes notify artifacts with no changes to the host. | RHEL EC2 node |
remediate_disk_cleanup.yml
|
Reclaims dnf package cache and old log archives, then publishes before/after usage stats for the warning notify tier. | RHEL EC2 node |
remediate_disk_expand.yml
|
Increases the root EBS volume in AWS, then runs growpart and xfs_growfs on the host for the critical expand path. | AWS (localhost) + RHEL EC2 node |
remediate_disk_fallback.yml
|
Handles unexpected switch tiers (e.g. boundary values) with no remediation and publishes artifacts for the unsupported notify template. | RHEL EC2 node |
notify_chatroom.yml
|
Includes the tier-specific Mattermost template (ok, warn, critical, unsupported) and posts the remediation summary to chat. | localhost (Mattermost API) |