# scribe-distill This skill should be used when the user says "distill this finding", "create a detection module", "encode this finding", "make a sigil from this", "automate detection for this", "build a sigil from this", "scribe distill", "/scribe-distill", or when a confirmed finding should be transformed into a reusable automated detection module. Analyzes a finding's vulnerability pattern, assesses automation feasibility, and creates a detection module (sigil) in the project spellbook. - **Kind:** skill - **Source:** https://github.com/JoranHonig/grimoire - **Page:** https://forefy.com/skills/bce90bab-aa81-4bfb-bbca-a83eb2bacad7 - **API (JSON + files):** https://forefy.com/api/asr/bce90bab-aa81-4bfb-bbca-a83eb2bacad7 --- ## SKILL.md --- name: scribe-distill description: >- This skill should be used when the user says "distill this finding", "create a detection module", "encode this finding", "make a sigil from this", "automate detection for this", "build a sigil from this", "scribe distill", "/scribe-distill", or when a confirmed finding should be transformed into a reusable automated detection module. Analyzes a finding's vulnerability pattern, assesses automation feasibility, and creates a detection module (sigil) in the project spellbook. user_invocable: true --- # Scribe Distill Transform confirmed findings into reusable automated detection modules. ## Philosophy Every confirmed finding is a potential lesson for future audits. The Scribe's distill workflow determines whether that lesson can be encoded as automated detection and, if so, builds the detection module. Static analysis is always preferred — it is deterministic, fast, and cheap. Agentic checks are a fallback for patterns that require reasoning. Some findings simply cannot be automated; for those, encode the knowledge as a reference artifact rather than letting it be forgotten. A low false-positive rate is more important than coverage. A detection module that fires on every audit but produces false positives is worse than one that misses edge cases but is always right when it fires. ## Workflow When this skill is activated, create a todo list from the following steps. Mark each task in_progress before starting it and completed when done. ``` - [ ] 1. Load finding and context - [ ] 2. Check for existing coverage - [ ] 3. Extract generalizable pattern - [ ] 4. Assess automation feasibility - [ ] 5. Determine sigil type - [ ] 6. Create detection module - [ ] 7. Validate detection module - [ ] 8. Assess variant analysis potential - [ ] 9. Present results and suggest follow-ups ``` --- ### 1. Load Finding and Context Read the finding file. Extract: - Title, severity, type - Affected code locations (from `context` frontmatter field) - Root cause (from Description and Details sections) - Impact (from Description) Read `GRIMOIRE.md` for codebase context — architecture, crown jewels, technology stack. Check in with the user before continuing. ### 2. Check for Existing Coverage Run `bash skills/checks/scripts/index-checks.sh grimoire/spells/checks/` to list existing checks. Read any that might cover the same vulnerability class. If a check already covers this exact pattern: - Report the overlap to the user - Stop unless the user wants a more specific variant or the existing check is too broad If partial overlap exists, note which aspects are already covered so the new module can focus on the gap. If the directory does not exist, no existing checks exist — skip to step 3. ### 3. Extract Generalizable Pattern Strip instance-specific details from the finding: - Remove specific contract/class/function names unique to this codebase - Remove hardcoded values, addresses, or identifiers - Preserve the structural code shape and the invariant violation Express the pattern as: "Code that does X without Y, in the context of Z." Example: A reentrancy finding in `Vault.withdraw()` becomes "Functions that make external calls before updating state in contracts holding user funds." Check in with the user to confirm the generalized pattern captures the essence of the vulnerability. ### 4. Assess Automation Feasibility Consult `references/feasibility-criteria.md` for detailed guidance. Classify the pattern: **Static-feasible** — The pattern is identifiable by searching for specific strings, function calls, code shapes, or AST patterns. Proceed to check creation. **Agentic-feasible** — The pattern is identifiable by grep but requires reading surrounding context to assess whether a match is a real issue. Create a check with assessment criteria that guide the applying agent. **Not automatable** — The pattern requires deep business logic understanding, external state knowledge, or full-codebase reasoning that cannot be reduced to a check. Create a knowledge artifact instead. Present the feasibility assessment to the user. ### 5. Determine Sigil Type Consult `references/sigil-types.md` for the type taxonomy and decision flowchart. For now, the only implementable types are: - **Check** — a markdown file in checks format, applied by spawning a subagent - **Knowledge artifact** — a reference doc for patterns that cannot be automated If the pattern would be better served by semgrep, slither, or codeql, note this as a future improvement but create a check as a fallback. ### 6. Create Detection Module **For checks (static-feasible and agentic-feasible):** Follow the check format from `skills/checks/references/check-format.md`: - YAML frontmatter: name, description, languages, severity-default, confidence, tools, tags - If the finding or pattern originated from external research (librarian results, audit reports, known registries like SWC), add `attribution-name` and `attribution-url` to credit the source - Body: Patterns section (grep-able search patterns), Assessment section (how to evaluate matches, severity adjustment, benign cases) Create the file in `grimoire/spells/checks/` with a slugified filename. Consult `skills/checks/examples/` for worked examples at different complexity levels. **Gnome delegation:** Spawn a Gnome agent to build the check. Provide it with the generalized pattern, target language(s), severity/confidence guidance, and assessment criteria. The Gnome handles file creation, format compliance, and validation. **For knowledge artifacts (not automatable):** Create a markdown file in `grimoire/spells/knowledge/` (create directory with `mkdir -p` if needed) with: - YAML frontmatter: name, description, vulnerability-class, languages - Body: what to look for (natural language), why it matters, known-bad patterns, assessment guidance ### 7. Validate Detection Module For checks, run: ```bash bash skills/checks/scripts/validate-check.sh ``` If validation fails, fix the issues and re-validate. For knowledge artifacts, verify: frontmatter is well-formed, description is clear and actionable, the document is self-contained. Check in with the user. ### 8. Assess Variant Analysis Potential Consider whether the pattern is likely to recur elsewhere in the current codebase: - How common is the code shape? (many similar components → high variant potential) - Is this a systemic issue or a one-off mistake? - How many files/contracts follow a similar structure? If variant analysis is warranted, suggest spawning a variant sigil with the generalized pattern from step 3. ### 9. Present Results and Suggest Follow-ups Summarize what was created using the Scribe distill output format from `agents/scribe.md`. Suggest follow-ups: - **Variant sigil** — if variant analysis is warranted (from step 8) - **End-of-audit merge** — when wrapping up, promote generalizable sigils to the personal grimoire via scribe Mode 2a - **Additional checks** — if the pattern has sub-variants not covered by this module - **Sigil application** — run the new check immediately against the codebase via the checks skill (Mode 2: Apply) ## references ``` ``` ## references/feasibility-criteria.md # Automation Feasibility Criteria Guide for determining whether a confirmed finding can be encoded as automated detection and which approach to use. ## Static Analysis (strongly preferred) The pattern is **static-feasible** when: - **Syntactic match:** The vulnerability involves a specific function call, keyword, import, or literal that can be grepped for directly. Examples: `debug_assert!` instead of `assert!`, `transfer()` before state update, `eval()` with user input. - **Structural match:** The vulnerability involves a code shape that can be identified by AST pattern matching (function signature shape, modifier absence, inheritance pattern). Even if not grep-able, tools like semgrep can match these. - **Low ambiguity:** Most matches of the search pattern are real issues, not benign uses. If the pattern has many benign matches, consider adding context constraints to narrow it. Estimated false positive rate for static patterns should be under 20%. If higher, narrow the detection or fall back to agentic. ## Agentic Check (fallback) The pattern is **agentic-feasible** when: - **Grep-able but context-dependent:** The pattern involves a searchable indicator (function call, import, annotation) but each match requires reading surrounding context to determine if it is a real issue. Example: `external` calls in Solidity are grep-able, but whether each constitutes a reentrancy risk depends on state update ordering. - **Multi-file assessment:** Determining vulnerability requires checking 2-3 related files (e.g., interface vs implementation, config vs usage). - **Severity depends on context:** The same code shape might be Critical in one context and Informational in another, requiring the agent to assess impact. Agentic checks should still have concrete grep patterns as starting points. The assessment section provides the reasoning framework the applying agent follows. ## Not Automatable (knowledge artifact) The pattern is **not automatable** when: - **Business logic dependency:** Understanding whether the code is vulnerable requires knowing the intended business rules, which are not expressed in code. Example: "the fee calculation is wrong" requires knowing the intended fee structure. - **External state dependency:** Exploitation depends on oracle prices, off-chain state, governance decisions, or runtime configuration not visible in the source. - **Full-codebase reasoning:** The vulnerability emerges from the interaction of many components and cannot be reduced to a local pattern. Example: "the protocol is economically exploitable via flash loans" requires understanding the full token flow. - **One-off design flaw:** The issue is unique to this specific system's design choices and has no generalizable pattern. For these, create a knowledge artifact: a reference document that describes what to look for in natural language so the researcher remembers to investigate this class of issue in future audits. ## False Positive Estimation When proposing a detection approach, estimate the expected false positive rate: | Rate | Meaning | Action | |--------|---------|--------| | Low | <5% of matches are benign | Strong candidate for automation | | Medium | 5-20% of matches are benign | Acceptable with assessment criteria | | High | >20% of matches are benign | Narrow the pattern or downgrade to knowledge artifact | A high-FP detection module wastes more researcher time than it saves. When in doubt, narrow the detection pattern at the cost of missing some true positives. ## references/sigil-types.md # Sigil Types Taxonomy of detection modules the Scribe can create, and guidance for selecting the right type. ## Available Now ### Check A markdown file in the checks skill format. Contains grep-able search patterns and assessment criteria for an applying agent to follow. - **Format:** YAML frontmatter + Patterns section + Assessment section - **Location:** `grimoire/spells/checks/.md` - **Applied by:** Spawning a subagent per check via the checks skill (Mode 2) - **Best for:** Patterns identifiable by grep that need context assessment, patterns where the applying agent can follow explicit criteria to determine true vs false positives - **Reference:** `skills/checks/references/check-format.md` ### Knowledge Artifact A reference markdown document encoding researcher knowledge for patterns that cannot be automated. Not a detection module — a reminder to investigate. - **Format:** YAML frontmatter (name, description, vulnerability-class, languages) + body describing what to look for, why it matters, known-bad patterns, assessment guidance - **Location:** `grimoire/spells/knowledge/.md` - **Surfaced by:** Scribe utilities (Mode 3) and during summon as manual review suggestions - **Best for:** Business-logic-dependent patterns, design-level flaws, patterns requiring full-codebase reasoning ## Planned (not yet implementable) ### Semgrep Rule A YAML rule for the Semgrep static analysis engine. AST-level pattern matching — deterministic, fast, zero false positives for well-written rules. - **Best for:** Syntactic patterns with clear bad/good code shapes - **Depends on:** semgrep skill (not yet implemented) - **Fallback:** Create a check with the grep patterns that a semgrep rule would formalize ### Slither Detector A Python detector for the Slither static analysis framework (Solidity-specific). Control flow, data flow, and inheritance analysis. - **Best for:** Solidity-specific patterns involving state variable access ordering, inheritance resolution, or cross-contract interactions - **Depends on:** slither skill (not yet implemented) - **Fallback:** Create a check targeting Solidity-specific code shapes ### CodeQL Query A QL query for the CodeQL analysis engine. Inter-procedural data flow and taint tracking. - **Best for:** Complex data flow patterns across function boundaries, taint propagation from source to sink - **Depends on:** codeql skill (not yet implemented) - **Fallback:** Create a check with multi-file assessment criteria ## Decision Flowchart ``` Is the pattern syntactically matchable (specific strings/calls)? ├── Yes → Is most of the target language Solidity? │ ├── Yes → Slither detector (planned). Fallback: check. │ └── No → Semgrep rule (planned). Fallback: check. └── No → Does it require reading context to assess matches? ├── Yes → Can matches be found by grep first? │ ├── Yes → Check (agentic, with assessment criteria). │ └── No → Does it require inter-procedural data flow? │ ├── Yes → CodeQL query (planned). Fallback: check. │ └── No → Knowledge artifact. └── No → Knowledge artifact. ``` When a planned type is the ideal choice but not yet available, create a check as a fallback and note the ideal type in the check's body. When the corresponding skill becomes available, these checks are candidates for upgrade.