Active
TMM Created
Novel Incident Remediation
Deterministic routing with AI fallback for unknown incidents
Description
Known incident types route directly to existing AAP job templates with no AI overhead. When an unknown or unclassified incident arrives, the workflow pivots to an AI agent that investigates using gathered system facts, enriches the ServiceNow ticket with findings, and generates a concrete remediation plan. An operator approves the plan, and the workflow executes the recommended automation. The same workflow handles both the deterministic and novel paths — no separate pipelines.
Why it matters
- Deterministic routing for known incident types means zero AI latency on the paths that matter most — disk, service, and config drift remediations run immediately.
- The AI fallback handles genuinely novel incidents without a human having to manually classify and route them — it investigates, explains, and recommends.
- A single unified workflow covers both known and unknown incidents. Operations teams maintain one automation pipeline instead of separate flows for routine and edge-case events.
- The approval gate keeps humans in control on the AI path — operators see the AI's reasoning and recommended automation before anything runs.
AO building blocks
Automation orchestrator step types this workflow uses on the canvas. See all building blocks
-
Triggers
Webhook trigger
Starts a workflow when an external system sends an authenticated HTTP POST to a generated endpoint URL, with the request body available as trigger input for downstream steps. Commonly used to integrate CI/CD pipelines, monitoring tools, or Splunk alerts — each workflow gets its own unique URL path.
-
Logic
Switch step
Evaluates conditions and routes execution to the first matching branch, supporting up to 100 ordered paths plus a fallback when nothing matches. Use it to branch on severity tiers, environment (dev/staging/prod), or AI agent output categories where a binary conditional isn't enough.
-
AAP execution
AAP job template
Launches a single Ansible Automation Platform job template from the workflow, polling until the job completes and exposing results to downstream steps. The standard bridge between automation orchestrator reasoning and AAP execution — one playbook with resolved inventory, credentials, limits, and extra vars.
-
AI & integrations
Task agent
Adds LLM reasoning to a workflow — the step receives a natural-language prompt, can call enabled tools, and returns structured output that downstream steps reference through expressions. Use it for unstructured data (logs, tickets, alerts) to summarize, classify, triage, and recommend remediation actions; pair with an approval step for human oversight.
-
Approval
Approval step
Pauses the branch and creates an approval request; an approver reviews context, approves or rejects, and the workflow continues on the matching path. Place approvals before destructive or production changes, with configurable approver users/groups, a decision window, and fallback behavior if no one responds in time.
Interactive Experience
Workflow
flowchart LR
A@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/manual-trigger.png", label: "SNOW Webhook", pos: "b", h: 96, constraint: "on" } --> B@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/switch-node.png", label: "Route by Incident Type", pos: "b", h: 96, constraint: "on" }
B -->|known| C@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Run Remediation JT", pos: "b", h: 96, constraint: "on" }
C --> D@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Update SNOW Ticket", pos: "b", h: 96, constraint: "on" }
B -->|unknown| E@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Gather System Facts", pos: "b", h: 96, constraint: "on" }
E --> F@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/task-agent.png", label: "AI: Investigate", pos: "b", h: 96, constraint: "on" }
F --> G@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/task-agent.png", label: "AI: Remediation Plan", pos: "b", h: 96, constraint: "on" }
G --> H@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/approval-node.png", label: "Approve", pos: "b", h: 96, constraint: "on" }
H -->|Approved| I@{ img: "https://ansible-tmm.github.io/aap-orchestrator-demos/assets/icons/launch-aap-job-template.png", label: "Execute Automation", pos: "b", h: 96, constraint: "on" }
I --> D
classDef noBorder fill:transparent,stroke:transparent
class A,B,C,D,E,F,G,H,I noBorder
Switch routing
| Incident Type | Path | What Happens |
|---|---|---|
capacity_issue |
Deterministic | Disk Capacity Cleanup JT runs, SNOW ticket updated and closed |
service_down |
Deterministic | Impact report generated, Service Recovery JT runs, ticket closed |
config_drift |
Deterministic | Impact report generated, escalated to Change Management |
unknown |
AI agent | Gather facts, AI investigates, enriches SNOW, AI generates remediation plan, operator approves, recommended automation executes |
Playbooks
| Playbook | What it does | Runs on |
|---|---|---|
gather_incident_facts.yml
|
Collects memory usage, disk utilization, failed service count, web service health check, and recent syslog errors from the affected host. Publishes all metrics as set_stats artifacts consumed by the AI agent. | RHEL target host |
remediate_disk_cleanup.yml
|
Clears app session storage, removes large log files and compressed archives older than 7 days, forces log rotation, and publishes before/after disk stats as artifacts. | RHEL target host |
linux_service_recovery.yml
|
Restarts the failed service, waits for it to become active, validates the web endpoint returns HTTP 200, and logs the recovery to syslog. | RHEL target host |
notify_snow.yml
|
Looks up the ServiceNow incident by ticket ID, updates the ticket state, and posts a work note with the provided title and body. Adds close code and close notes when resolving (state 6). | localhost (ServiceNow API) |
incident_impact_report.yml
|
Gathers system facts from the affected host, assesses service and resource impact, and posts a structured impact report as a work note on the ServiceNow ticket. | RHEL target host |
incident_log_and_report.yml
|
Logs the unrecognized incident details and posts a work note to the ServiceNow ticket flagging it for manual triage. | localhost (ServiceNow API) |
Demo ideas
- Fire a capacity_issue incident and show the disk cleanup job template running with no AI in the loop
- Fire a service_down incident to walk the impact report and service recovery path
- Fire an unknown incident to show the AI agent activating, enriching the SNOW ticket, and generating the remediation plan
- Compare the unknown path approval step — show the AI reasoning in the SNOW ticket before approving