Make the defense
visible.

01 / FIELD NOTES

A defense you can explain is a defense you can operate.

A dashboard turns red. An IP address appears in a list. Someone says the attack was blocked. Fine. But what actually happened? Which service produced the signal? Which address did the host identify? What changed in the kernel? And can a legitimate user still get their work done?

That is where a useful security exercise begins. I want people learning infrastructure operations to leave with more than a screenshot. I want them to explain the whole sequence, challenge their first interpretation and restore the machine without improvising. SysWarden is a practical way to work through that discipline: Linux host defense, native enforcement, observable decisions and a recovery path you can inspect.

The scenarios below are intended for educational use. They move from a healthy baseline to authentication abuse, suspicious web requests, temporary enforcement, narrow access exceptions, log provenance and incident handover. The traffic stays small. The questions get progressively more interesting.

If you need the build and installation path first, start with the SysWarden source installation guide. That guide identifies its own reviewed revision. Keep those revisions explicit: a recipe reviewed against one commit does not automatically validate another.

SysWarden.FROM SIGNAL TO EVIDENCE

1. 01ProduceOne bounded, intentional event
2. 02ObserveA real service log and its source
3. 03VerifyDecision, native state and traffic
4. 04RestoreA clean baseline and a useful report

02 / FIELD NOTES

Build a small lab with a clear way back.

Use a disposable Linux target with SysWarden already installed, a traffic generator and a separate operator workstation. The copyable service commands below use a systemd target with Bash, OpenSSH, curl and nftables tooling. Debian or Ubuntu is a convenient teaching environment. Alpine uses OpenRC; do not paste systemctl commands there and conclude that the product is broken.

Take a restorable snapshot before installation or policy changes, verify console access, and write down the original firewall and service state. Use a static web page with no database, login form or application backend for the HTTP exercise. Keep the target reachable only by the roles needed for the lab. A service that was already unreachable cannot prove that a new defensive decision worked.

For that public variant, let the instructor approve both endpoints and the effective source address before any traffic. NAT can merge several people into one address. A reverse proxy can make the proxy look like the client. These are operational facts, not small details. Do not spoof forwarding headers or weaken the target policy to force a dramatic result.

Work in two passes. First observe. Then, only when the signal and scope are understood, enable enforcement on the disposable target. Review waap.enforcement_mode in the modular configuration: audit records a simulated decision for an otherwise eligible target; enforcing permits the native mutation. Protected sources can remain shadow observations even in audit mode.

sudo /usr/local/bin/syswarden config-get waap.enforcement_mode
sudo /usr/local/bin/syswarden config-get waap.bruteforce_threshold
sudo /usr/local/bin/syswarden config-get waap.bruteforce_window_seconds
sudo /usr/local/bin/syswarden config-get waap.bruteforce_logs
sudo /usr/local/bin/syswarden config-get waap.modsec_logs

A configuration key reported as unset is not proof that the running core has no default. Resolve the effective configuration before testing. Use the configuration editor to merge changes into the appropriate existing module or user override. Do not overwrite a complete configuration with a three-line sample. Validate before reloading, and stop if validation fails. A reload can restart the core and affect enforcement, so retain console access and your baseline.

sudo /usr/local/bin/syswarden config
sudo /usr/local/bin/syswarden config validate --path /etc/syswarden/config && \
  sudo /usr/local/bin/syswarden reload

The source defaults are five tracked matches within sixty seconds. Your installed configuration and rule catalog take precedence. One connection can produce more than one relevant record, so count matched events rather than guessing from the number of commands you typed. Confirm the core has loaded the intended mode, input paths and configuration; a CLI configuration read alone does not prove that a running process has reloaded them.

Source: Enforcement modes, defaults and native collector, reviewed at f334bead.

Source: Protected dynamic targets, reviewed at f334bead.

03 / FIELD NOTES

Scenario 01 / Prove that normal service works.

Objective: establish a baseline that makes later failures meaningful. This is the quiet exercise that prevents a lot of noisy conclusions.

On the target, record the UTC time, validate configuration, check the core service and inspect the listening sockets. Review the native ruleset before generating traffic. Keep that output private: real addresses and policy details do not belong in a public class report by default.

date -u
sudo /usr/local/bin/syswarden config validate --path /etc/syswarden/config
sudo systemctl is-active syswarden-core
sudo ss -lnt
sudo nft -a list ruleset
sudo /usr/local/bin/syswarden audit

From the traffic generator, set the exact target IPv4 address approved for this exercise. This variable belongs to the current Bash session. Repeat this small setup whenever you open a new terminal. There are no shell prompts hidden in the code blocks; the Copy button copies only the commands.

read -r -p "Approved lab target IPv4 address: " lab_target
python3 - "$lab_target" <<'CHECK'
import ipaddress, sys
address = ipaddress.ip_address(sys.argv[1])
assert address.version == 4, "This exercise uses an IPv4 target"
print("Target:", address)
CHECK

After the address check succeeds, make one request to the static lab page. The -q option, placed first, avoids a hidden curl configuration. The direct request bypasses proxy environment settings, has a short time budget and does not follow redirects to another host.

curl -q --noproxy '*' --connect-timeout 3 --max-time 5 \
  -sS -o /dev/null -w 'HTTP %{http_code}\n' \
  "http://${lab_target:?Set lab_target first}/"

Open a fresh authorized SSH session from the operator workstation as well. An already established session is useful for recovery, but is a weak test of whether new connections are allowed. Identify the real source address in the target logs and confirm the benign HTTP request reached the intended service.

Negative control: a successful login and a normal page request should not be presented as failed authentication or a SQL injection attempt. Cleanup: none beyond closing test connections. Keep the baseline and record any setup correction.

Source: Operator diagnostics and interpretation, reviewed at f334bead.

04 / FIELD NOTES

Scenario 02 / Follow a failed SSH login all the way through.

Situation: a host receives repeated failed authentication events. You want to distinguish recognition from enforcement without running a password-cracking tool or flooding a service.

Have the instructor prepare a dedicated lab account and an authentication method that can produce a real failed-password record. Do not change production SSH settings, disable key authentication or loosen a hardened host just to obtain a screenshot. If the server is key-only, treat the failure format as a diagnostic exercise and document what it actually emits.

On the target, watch SysWarden alerts in one terminal. In another, inspect the actual SSH service journal or configured native authentication file. On Debian, the service is commonly ssh; on RPM systems it is commonly sshd. Select the unit that exists on your host. File-based ingestion still needs the native log file configured and collected, even if journald shows the event.

sudo /usr/local/bin/syswarden alerts

sudo journalctl -u ssh -u sshd --since '5 minutes ago' --no-pager
sudo journalctl -u syswarden-core --since '5 minutes ago' --no-pager

From the generator, enter the prepared lab username. Verify the SSH host-key fingerprint against the console before accepting it. Type one deliberately incorrect lab password when prompted. No password is included in the command, shell history or article.

read -r -p "Prepared lab username: " lab_user
ssh -o ConnectTimeout=5 \
  -o PreferredAuthentications=password \
  -o PubkeyAuthentication=no \
  -o KbdInteractiveAuthentication=no \
  -o NumberOfPasswordPrompts=1 \
  "${lab_user:?Set lab_user first}@${lab_target:?Set lab_target first}"

Start with one attempt. Check the native line, its timestamp, the source address and the matched ssh-auth rule. The reviewed catalog recognizes constrained native SSH failure formats, including sshd-session and the Alpine timestamp envelope. A connection-closed message is not interchangeable with a failed-password message. Appending an invented line to a file is not proof of a real authentication attempt.

For a threshold exercise, agree a small event budget before starting. With a confirmed five-match threshold, make at most five manual attempts during the configured window, stopping as soon as the relevant decision appears. Count the service records and matched events. If the expected pattern is missing, investigate the collector and authentication method instead of increasing the volume.

Active variant: use only the approved public generator with a separate operator path. Correlate the native failure, the SysWarden decision, the exact IP in native state and a new connection attempt from that generator. Compare a fresh operator connection at the same time. A timeout by itself could be routing, a cloud firewall or a stopped service.

Negative control: perform a valid login from the operator and verify it does not count as a failed-password event. Cleanup: stop attempts, close the alert stream, then use the runtime cleanup procedure in Scenario 04 if a dynamic ban was created. Never clear the whole firewall to make the lab green.

Source: Native SSH signature and trusted source capture, reviewed at f334bead.

Source: Native authentication provenance, reviewed at f334bead.

05 / FIELD NOTES

Scenario 03 / Detect a suspicious request without a vulnerable application.

Situation: a web server receives a request containing a familiar SQL injection marker. The learning target is the defensive chain, so a static Nginx page is sufficient. There is no database to attack and no reason to deploy an intentionally vulnerable application for this exercise.

Before starting, the instructor must verify that the target writes a native, trusted access log in a supported format and that SysWarden actually collects it. The source policy rejects unsafe log ownership and permissions. Do not make a log world-writable to fix a collector error. Keep a direct path to the target in this first exercise, without a CDN or reverse proxy obscuring the client address.

SysWarden.TWO DIFFERENT PATHS

1. 01HTTP requestClient reaches the web service
2. 02Access logThe service writes what it received
3. 03AnalysisSysWarden recognizes the request target
4. 04Host ruleAn eligible source can be blocked

After the normal request from Scenario 01 succeeds, send this single marker request from the generator. The static server treats the query as data; it does not execute SQL. The marker is deliberately recognizable in the reviewed signature catalog.

curl -q --noproxy '*' --connect-timeout 3 --max-time 5 \
  -sS -o /dev/null -w 'HTTP %{http_code}\n' \
  "http://${lab_target:?Set lab_target first}/?lesson=UNION%20SELECT"

Find the exact request in the native access log, then correlate the sqli match, source address, enforcement mode and outcome in SysWarden. The request may return the static page normally. That does not disprove log detection. It does mean you must not claim that the application was protected from that already delivered request.

Use the private observation topology first. For an authorized active variant, use the separate public generator and inspect native state before making one fresh benign request from the same source. At the same time, confirm that the operator can still reach the page. If another firewall already blocks that source, restore a known baseline before attributing the result to SysWarden.

Negative control: repeat the ordinary / request from a clean source and verify that it is not classified as the marker. Cleanup: stop traffic and remove only the lab-created runtime claim if applicable. Retain the paired benign and suspicious records with their timestamps.

For a later architecture discussion, compare this flow with an inline WAF such as BunkerWeb. The WAF can act in the request path; SysWarden contributes host-level decisions and enforcement. Explain the trust boundary and client-address provenance before connecting the two. The SysWarden and BunkerWeb article provides the wider context.

Source: Request-target SQL marker catalog, reviewed at f334bead.

Source: Native log collection and outcomes, reviewed at f334bead.

06 / FIELD NOTES

Scenario 04 / Prove that a temporary defense has a lifecycle.

Situation: a temporary runtime ban exists after an approved active exercise. The incident is over. The operator must understand what owns that ban, when it expires and what a removal command actually removes.

Use the public source that produced the previous event. From the operator terminal on the target, enter that exact source IP. Do not enter the protected target, your own administration address or an arbitrary address seen in a screenshot.

read -r -p "Approved generator source IPv4 address: " lab_source
python3 - "$lab_source" <<'CHECK'
import ipaddress, sys
address = ipaddress.ip_address(sys.argv[1])
assert address.version == 4 and address.is_global, "Use the approved public generator"
print("Source to inspect:", address)
CHECK

The address check is a syntax and broad scope check; it does not prove ownership or reproduce every SysWarden target-policy exclusion. Continue only after verifying the source against the approved lab plan. Then inspect the entry and the history view. In the terminal UI, press h to open runtime history.

sudo /usr/local/bin/syswarden check "${lab_source:?Set lab_source first}"
sudo nft -a list ruleset
sudo /usr/local/bin/syswarden tui

Choose one branch of the exercise. In the expiry branch, record the actual deadline and leave the source quiet; fresh events can affect the lifecycle. Observe after that deadline, then observe again after the required reconciliation delay. Do not assume a universal one-minute ban or change the system clock to hurry things along.

In the operator-removal branch, remove the lab-created local runtime claim with the command below. This is appropriate only after checking that it is the claim you intend to remove. Inspect the result and repeat the legitimate connection probe.

sudo /usr/local/bin/syswarden runtime-unblock "${lab_source:?Set lab_source first}"
sudo /usr/local/bin/syswarden check "${lab_source:?Set lab_source first}"
sudo nft -a list ruleset

runtime-unblock is not an instruction to erase every reason an address might remain blocked. A persistent operator entry or another independent claim can still apply. The separate unblock command handles persistent block policy as well. Likewise, manually adding an address with block does not create an authentic attack event or turn a persistent policy entry into a temporary detection.

Pass condition: the recorded state, exact native entry and new traffic agree with the branch you chose. Negative control: explain why a separate persistent claim would still prevent access. Cleanup: return the lab source to its recorded baseline, preserving unrelated policy and the full private lifecycle state.

Source: Runtime states, TTL and claim ownership, reviewed at f334bead.

07 / FIELD NOTES

Scenario 05 / Give support exactly the access it needs.

Situation: a support engineer needs temporary SSH access. Opening every port or globally trusting an address is a quick way to create a second problem while solving the first.

This is a supervised policy exercise. Use an approved, distinct support source and a baseline that intentionally denies its new SSH connection. Verify that the target service is listening, the routing works and the outer firewall permits the lab path. If the cloud firewall blocks the connection first, a host-level exception cannot establish the intended result.

Record any pre-existing exception for that source before changing anything. Keep the operator and console paths separate. On the target, prompt for the instructor-approved support address and the actual SSH port. The example deliberately does not assume every host uses port 22.

read -r -p "Approved support source IP: " support_ip
read -r -p "Target SSH TCP port: " support_port
sudo /usr/local/bin/syswarden allow-ssh \
  "${support_ip:?Set support_ip first}" "${support_port:?Set support_port first}"

The allow-ssh command uses the dedicated SSH exception registry. It is distinct from a global whitelist. After applying it, verify a fresh SSH connection from the support source and inspect the corresponding native rule. A successful command response does not replace this traffic check.

Negative control: compare an unrelated service port whose baseline denies the support source and is known to be listening. That port should not become accessible merely because SSH was granted. A connection-refused response from a nonexistent service is not evidence of filtering. Do not scan a port range; select the one approved control service.

When support is finished, remove only the exception created for this exercise. If an exception existed beforehand, restore its original value instead of revoking it. Then test another new connection and confirm that the original denied state has returned.

sudo /usr/local/bin/syswarden revoke-ssh "${support_ip:?Set support_ip first}"
sudo nft -a list ruleset

An established session can survive a policy change because connection tracking and rule ordering matter. Use new connections for the comparison. Record the service, protocol, source, duration and owner of the exception. That short record is often more useful to the next operator than a large export with no explanation.

Source: Supported access commands, reviewed at f334bead.

Source: Dedicated SSH exception handling, reviewed at f334bead.

08 / FIELD NOTES

Scenario 06 / Refuse a security signal you cannot trust.

Situation: someone suggests watching a custom log file. The path looks plausible. Its contents look convincing. But an ordinary application account can edit it. Should that file be allowed to influence host enforcement? This is a useful negative test, and a very practical design discussion.

Use a separate disposable clone in audit mode for this scenario. The instructor prepares two custom input fixtures: one real regular file satisfying the documented ownership and permissions, and one deliberately unsafe fixture writable by an untrusted account. Neither fixture is the live SSH or Nginx log. Label them as synthetic inputs in the evidence.

The exercise is about admission to the collector, not about fabricating a successful native attack. Inspect owner, group, mode, file type and the parent directory chain. For a root-owned custom input, the reviewed policy requires a regular file with no group or other write permission. An appropriate readable mode such as 0640 can be valid; a symlink or a writable substitute is not equivalent.

read -r -p "Instructor-prepared custom log path: " fixture_path
sudo stat -- "${fixture_path:?Set fixture_path first}"
sudo namei -l -- "${fixture_path:?Set fixture_path first}"

Have the instructor add only the selected fixture to the intended custom log configuration, validate the modular configuration and perform the normal reload. Observe the collector diagnostics. Configuration validity and source admission are separate checks: a syntactically correct file path does not prove that the collector accepts the file behind it.

Compare with the eligible fixture, then inspect the documented behavior for rotation or replacement. Direct collection rechecks file identity and trust during following; a bridge generated at configuration time has its own admission boundary. Do not collapse those different checks into a claim that every transport has identical runtime guarantees.

Negative control: a believable message in the unsafe file must not be reported as a genuine SSH login failure. Cleanup: remove the exercise-only input reference, restore the cloned configuration and retain the rejection evidence. Never delete or rewrite real authentication history.

The lesson is broader than log permissions. A detection system needs a trustworthy origin, a constrained parser and an authoritative source address. If any local user could write an enforcement command disguised as a log message, the defender would become a tool for disruption.

Source: Custom inputs, native writers and transport trust, reviewed at f334bead.

09 / FIELD NOTES

Scenario 07 / Recover the service and hand over the evidence.

Situation: the first operator leaves halfway through an incident. A second operator must determine what happened, what remains blocked and whether the machine is back in a known state. This is where a lab starts to look like real operations.

Work in pairs if possible. The first operator runs one of the earlier scenarios and hands over a short evidence pack. The second gets the pack and the approved console access, but no verbal hints about the intended result. Ask them to identify the source, rule, decision, remaining claim and next safe action.

For an optional recovery extension, the instructor can restart the core on a disposable target while a known temporary claim is active. Record its original deadline and remaining lifetime first. After restart, compare the native state and history with that deadline. A live claim should be evaluated through the documented lifecycle recovery path; the exercise must not assume that every restart grants it a fresh full lifetime.

date -u
sudo systemctl restart syswarden-core
sudo systemctl is-active syswarden-core
sudo journalctl -u syswarden-core --since '5 minutes ago' --no-pager
sudo nft -a list ruleset

If recovery reports inconsistent state, preserve the error and use the planned restore procedure. Do not delete the lifecycle journal or anchors to manufacture a healthy-looking result. The runtime directory contains related state that must remain consistent; a rollback needs a complete compatible recovery point, not a convenient selection of files.

This service-restart exercise is not a high-availability qualification. A real HA campaign requires two native nodes, explicit roles, authenticated replication, failure handling and its own acceptance criteria. A second process on the same laptop cannot establish those properties. Keep HA as a separate advanced workshop unless you have prepared that environment properly.

Pass condition: the second operator can justify the outcome from the evidence and restore the baseline without guessing. Negative control: include one ambiguous timeout or rejected input and see whether they correctly mark it inconclusive. Cleanup: verify fresh operator access, normal service traffic, the expected firewall association and local policy, then close the temporary lab window.

A failed observation is still useful evidence. Keep the first attempt, explain the cause and show what changed before a retry. Hash the private evidence files, restrict access and redact the public teaching report. A hash detects a changed file; it does not by itself prove who produced it or whether the underlying claim is true.

Source: Consistent restart and rollback requirements, reviewed at f334bead.

Source: Separate HA prerequisites, reviewed at f334bead.

10 / FIELD NOTES

Assess the reasoning, not the number of red alerts.

A useful lab report answers six questions: what was supposed to happen, what actually happened, which source produced the evidence, what changed in native state, what legitimate behavior remained available, and how the baseline was restored. If one answer is missing, the right next step is usually another observation, not more traffic.

This is also how I approach SysWarden development. Integration connects real components. Verification checks that the implementation meets its requirements. Validation asks whether the operator can accomplish the intended job. Qualification binds the result to a specific candidate, platform, procedure and body of evidence. Those words describe work, not marketing labels.

Open source makes that work inspectable. It lets a learner follow a command into the implementation, challenge a claim and propose a better test. Independence is valuable, but it comes with a responsibility to describe the limits just as clearly as the strengths. A product becomes useful when people can operate it confidently and explain why they trust the result.

Keep the first workshop small. Run the baseline, one SSH failure and one suspicious HTTP request. Read the evidence together. Then add lifecycle recovery and narrow access exceptions. The goal is not to create the loudest attack demonstration. It is to build the habit of making a decision, checking its effect and leaving the next operator a clean, understandable system.

Continue with the source installation guide, inspect the reviewed source, or explore the other SysWarden articles. For tool option details, consult the official curl manual and the OpenSSH manual.