# openhound-development Use for OpenHound collector and OpenGraph extension development work, including planning, graph schema, source collection, assets, lookup/preproc, registration, validation, and collector template adaptation under the BloodHound plugin. - **Kind:** skill - **Source:** https://github.com/SpecterOps/skills - **Page:** https://forefy.com/skills/3789fb9a-09a7-4861-b1f6-e54285adcbfd - **API (JSON + files):** https://forefy.com/api/asr/3789fb9a-09a7-4861-b1f6-e54285adcbfd --- ## SKILL.md --- name: openhound-development description: Use for OpenHound collector and OpenGraph extension development work, including planning, graph schema, source collection, assets, lookup/preproc, registration, validation, and collector template adaptation under the BloodHound plugin. metadata: author: "GhostWorks" --- # OpenHound Development Use this skill for any OpenHound collector task. ## Critical Rules (Always apply) 1. Read `../../standards/openhound/openhound.md` before editing collector code when working from this plugin, or `.agents/standards/openhound.md` inside a generated collector repo. This includes important standards and best practices for OpenHound collector development that are not repeated in the reference docs. 2. Based on the requested task, select the matching reference from the routing table. These contain specific rules and examples for different types of collector work. 3. For broad collector work, or when creating a new collector, read `../../standards/openhound/workflow.md` when working from this plugin, or `.agents/standards/workflow.md` inside a generated collector repo. 4. Important: Before finishing collector and graph behavior changes read `references/validate-extension.md`. ### Route By Task | Task | Reference | |---|---| | Plan a new collector from API docs, sample responses, or requirements | `references/plan-collector.md` | | Define graph base classes, common properties, node IDs, or edge properties | `references/graph-schema.md` | | Register `collect`, `preproc`, `convert`, metadata, or entry points | `references/register-extension.md` | | Implement API clients, auth, DLT resources, transformers, or secrets | `references/source-collection.md` | | Add or modify models, node assets, edge assets, kind constants, or exports | `references/add-asset.md` | | Add DuckDB transforms, lookup methods, lookup registration, or `self._lookup` usage | `references/preproc-lookup.md` | | Validate collector changes before finishing | `references/validate-extension.md` | ### Routing Rules - If the task touches multiple areas, read every matching reference. - If adding or changing collector behavior, always read `references/validate-extension.md` before finishing. - If the task involves a new collector or broad redesign, start with `references/plan-collector.md`. - If models use `self._lookup`, also read `references/preproc-lookup.md` and `references/register-extension.md`. - If adding a new collected resource, usually read `references/source-collection.md`, `references/add-asset.md`, and `references/validate-extension.md`. ## agents ``` ``` ## agents/openai.yaml ```yaml interface: display_name: OpenHound Development short_description: "Build and validate OpenHound collectors." icon_small: ./assets/icon.svg icon_large: ./assets/icon.png brand_color: '#8E92EB' default_prompt: Use $openhound-development for OpenHound collector and OpenGraph extension development workflows. policy: allow_implicit_invocation: true ``` ## assets ``` ``` ## assets/icon.png ``` ``` ## assets/icon.svg ``` ``` ## references ``` ``` ## references/add-asset.md # OpenHound Add Asset Use this reference when adding a new resource model under `src//models/` or changing how a model emits nodes and edges. ## Files Usually Touched - `src//models/.py` - `src//models/__init__.py` - `src//kinds/nodes.py` - `src//kinds/edges.py` - `src//source.py` - `src//main.py` if lookup/preproc table registration is required - `src//transforms.py` and `src//lookup.py` if cross-table lookup is required ## Model Structure Every collected asset should define two things: 1. A dataclass named `Properties` extending the extension base `NodeProperties`. 2. A Pydantic `BaseAsset` subclass named `` decorated with `@app.asset(...)`. The raw fields on the `BaseAsset` class should mirror the collected JSONL schema. The dataclass properties should describe the OpenGraph node properties emitted during conversion. ## Property Dataclasses Every OpenGraph property field must be documented in the class docstring's `Attributes` section: ```python from dataclasses import dataclass from openhound_.graph import EXNodeProperties @dataclass class AssetProperties(EXNodeProperties): """Properties for the Asset node. Attributes: hostname: The asset hostname. """ hostname: str ``` Prefer nullable types over placeholder sentinels when data can be absent: ```python group_name: str | None = None ``` Do not use empty strings to mean missing data unless the upstream API explicitly distinguishes empty string from null. ## Asset Classes Use `@app.asset(...)` with `NodeDef` for node-bearing assets and `EdgeDef` for relationships emitted by that same class. ```python from openhound.core.asset import BaseAsset, EdgeDef, NodeDef from openhound.core.models.entries import Edge, EdgePath from openhound_.graph import EXNode from openhound_.kinds import edges as ek from openhound_.kinds import nodes as nk from openhound_.main import app @app.asset( node=NodeDef(kind=nk.ASSET, properties=AssetProperties, description="Example asset", icon="cog"), edges=[ EdgeDef( start=nk.ASSET, end=nk.GROUP, kind=ek.MEMBER_OF, description="Asset belongs to group", ) ], ) class Asset(BaseAsset): id: str name: str groups: list[str] @property def as_node(self) -> EXNode: properties = AssetProperties( node_id=self.id, name=self.name, displayname=self.name, environmentid=self._extras["environmentid"], ) return EXNode(properties=properties, kinds=[nk.ASSET]) @property def edges(self): for group_id in self.groups: yield Edge( kind=ek.MEMBER_OF, start=EdgePath(value=self.as_node.id, match_by="id"), end=EdgePath(value=group_id, match_by="id"), ) ``` ## Edge Definition Alignment Only declare `EdgeDef(...)` entries on the asset class that emits those edges from its `edges` property. If a node-bearing asset creates only a node, use only `node=NodeDef(...)` and emit no edges. If relationships are represented by a separate edge-only asset, put the `EdgeDef(...)` declarations on that edge-only asset. If a helper emits a shared edge, such as a root/environment containment edge, declare the matching `EdgeDef(...)` on every asset that yields it. ## Edge Emission Style Prefer generators: ```python @property def _access_edges(self): for target_id in self.target_ids: yield Edge(...) @property def edges(self): yield from self._access_edges ``` Avoid building `edges = []`, appending items, and returning the list unless there is a specific reason. ## Conditional Edge Paths Prefer `ConditionalEdgePath` when creating an edge to an existing node that should be resolved by node properties instead of a known OpenGraph ID. Use `PropertyMatch` entries for the stable property constraints needed to uniquely identify the target node. Avoid weak matchers, such as only `name`, unless the upstream domain guarantees uniqueness. Use `EdgePath(value=..., match_by="id")` when the exact stable node ID is already known. ```python from openhound.core.models.entries import ConditionalEdgePath, Edge, EdgePath, PropertyMatch from openhound.core.models.entries_dataclass import EdgeProperties yield Edge( kind=ek.MEMBERSHIP_SYNC, start=ConditionalEdgePath( kind=nk.GROUP, property_matchers=[ PropertyMatch(key="tenant_domain", value=source_domain), PropertyMatch(key="type", value="OKTA_GROUP"), PropertyMatch(key="name", value=self.profile.name.upper()), ], ), end=EdgePath(value=self.id, match_by="id"), properties=EdgeProperties(traversable=True), ) ``` ## Lookup Usage Use `self._lookup` inside `as_node` or `edges` only when `preproc` creates and registers the required lookup data. ```python @property def as_node(self) -> EXNode: org_node_id = self._lookup.org_id_for(self.org_name) properties = AssetProperties(node_id=self.id, org_id=org_node_id) return EXNode(properties=properties, kinds=[nk.ASSET]) ``` ## Checklist - Create or update `src//models/.py`. - Add node kind constants to `kinds/nodes.py` if the asset emits nodes. - Add edge kind constants to `kinds/edges.py` if the asset emits edges. - Assets returning a node must pass the emitted property dataclass into `NodeDef(properties=...)`. - Export the model from `src//models/__init__.py`. - Add or update a resource or transformer in `source.py`. - Wire the resource or transformer into the source return tuple. - If lookup data is needed, update `transforms.py`, `lookup.py`, and the `preproc` table map in `main.py`. - Read `references/validate-extension.md` before finishing. ## references/graph-schema.md # OpenHound Graph Schema Use this reference when editing `src//graph.py` or changing extension-wide OpenGraph node and edge property behavior. ## Purpose `graph.py` defines the OpenHound extension's base dataclass types for nodes, node properties, and edge properties. These types are imported by model files and lookup-related code. ## Naming Use the extension prefix consistently: - `NodeProperties` - `Node` - `EdgeProperties` For example, GitHub might use `GHNodeProperties`, `GHNode`, and `GHEdgeProperties`. ## Base Node Properties The base properties class should extend `NodeProperties` from `openhound.core.models.entries_dataclass`. Add only fields that every node in the extension should have. ```python from dataclasses import dataclass from openhound.core.models.entries_dataclass import NodeProperties as BaseProperties @dataclass class EXNodeProperties(BaseProperties): """Base properties for all nodes in the extension. Attributes: node_id: The platform native unique identifier. """ node_id: str ``` Every field must include docstrings specifying the attributes with a description because this is used for generated documentation. Every collector should have one root/environment node. When a collector emits multiple resource nodes, include `environmentid` in the base node properties and set it to the OpenGraph ID of that root/environment node. ## Node Class The node class should extend `Node` from `openhound.core.models.entries_dataclass` and set `self.id` in `__post_init__`. ```python from dataclasses import dataclass, field from openhound.core.models.entries_dataclass import Node as BaseNode @dataclass class EXNode(BaseNode): properties: EXNodeProperties kinds: list[str] id: str = field(init=False) def __post_init__(self): self.id = self.properties.node_id ``` ## Node ID Rules Prefer the platform's native opaque/global node ID when available and only when the native ID is consistent and unique. Do not use raw integer primary keys as OpenGraph node IDs. They are not stable enough across systems and can collide across extensions. If the platform does not expose a suitable ID, derive one from stable properties with `BaseNode.guid(...)`: ```python def __post_init__(self): self.id = self.guid(self.properties.tenant_id, self.properties.slug) ``` Use reproducible values only. Do not include timestamps, random values, pagination offsets or mutable display names unless they are the best stable identifier the service provides. ## Edge Properties The extension edge property class should extend `EdgeProperties`. ```python from dataclasses import dataclass from openhound.core.models.entries_dataclass import EdgeProperties @dataclass class EXEdgeProperties(EdgeProperties): reason: str | None = None ``` Add shared fields only when they apply broadly across extension edges. Entity-specific relationship fields can stay closer to the emitting model when appropriate. ## Graph Schema Rules - Keep `graph.py` generic to the extension, not specific to one entity. - Avoid importing model classes into `graph.py`. - Avoid importing `app` into `graph.py`. - Do not define kind strings in `graph.py`; use `kinds/nodes.py` and `kinds/edges.py`. - Do not add compatibility aliases unless there is persisted data, shipped behavior, external consumers, or an explicit requirement. ## Checklist - Prefix classes match the service prefix. - Base node properties extend the OpenHound base properties class. - Node class extends the OpenHound base node class. - `id` is assigned in `__post_init__`. - Node ID is stable and string-compatible. - Base node properties include `environmentid` when the collector emits a root/environment node. - Every graph property contains docstrings with an "Attributes" section describing each field. - Read `references/validate-extension.md` before finishing. ## references/plan-collector.md # OpenHound Plan Collector Use this reference before implementing a new collector or making broad collector changes. The output should be a concise design brief that maps the target service into OpenHound resources, graph nodes, edges, and follow-up implementation references. Do not implement collection, models, lookup logic, or metadata from this reference. Use it to decide what should be built and in what order. ## Inputs To Gather - Target service name, slug, and intended short graph prefix. - Authentication method, required credentials, and non-secret configuration values. - API base URL, versioning model, pagination style, rate limits, and retry behavior. - Primary resources to collect, such as users, groups, devices, roles, policies, repositories, or memberships. - Sample API responses or schema references for each resource. - Stable identifiers for each resource. - Relationships between resources and whether they can be emitted directly or need lookup/preproc. - Any sensitive fields that should not be collected or emitted. - Expected commands or environments used to validate the collector. If required information is missing, ask focused questions before designing implementation details. ## Planning Checklist ### 1. Service Identity Define the collector identity: - Source name and source_kind used by `OpenHound("", source_kind=)`. - Python package name, normally `openhound_`. - Short uppercase graph prefix, usually two to four characters. - Extension metadata values for `extension.yaml`. ### 2. Credentials And Configuration List required secrets and parameters: - DLT secret names expected under `[sources.source.]`. - Environment variable equivalents where useful. - Non-secret parameters such as tenant, organization, region, or API host. - Whether a dedicated `auth.py` module is likely needed. ### 3. Resources To Collect Create a resource inventory: | Resource | API endpoint | DLT table | Model | Notes | |---|---|---|---|---| | users | `/users` | `users` | `User` | Example top-level resource. | For each resource, decide whether it should be a top-level `@app.resource` or a nested `@app.transformer`. ### 4. Graph Shape Create a graph inventory: | Node | Kind constant | Source resource | Stable ID | Notes | |---|---|---|---|---| | User | `USER` | `users` | Native user ID | Example node. | Include one root/environment node for the collected environment. Other emitted nodes should set `environmentid` to that node's OpenGraph ID. | Edge | Start | End | Source resource | Lookup needed | |---|---|---|---|---| | `MEMBER_OF` | User | Group | memberships | No | Prefer native opaque/global IDs when available. Do not plan raw integer primary keys as OpenGraph node IDs unless they are converted into stable, collision-resistant IDs. ### 5. Lookup And Preproc Needs Identify where direct conversion is not enough: - Relationships requiring joins across collected tables. - Display or context fields that need enrichment from another table. - Derived tables that should be created in `transforms.py`. - Lookup methods that should be cached in `lookup.py`. If no cross-table resolution is needed, explicitly say preproc/lookup is not required. ### 6. Implementation Order Map the plan to references: 1. `references/graph-schema.md` for graph base types and ID strategy. 2. `references/register-extension.md` for phase registration and metadata. 3. `references/source-collection.md` for API resources and transformers. 4. `references/add-asset.md` for each model, node, edge, and kind constant. 5. `references/preproc-lookup.md` only when cross-table lookup is required. 6. `references/validate-extension.md` before finishing. ## Output Format Produce a short collector design brief: ```markdown # Collector Plan: ## Assumptions - ## Credentials And Parameters - ## Resources | Resource | API endpoint | DLT table | Model | Type | |---|---|---|---|---| ## Graph | Node | Kind | Stable ID | Source | |---|---|---|---| | Edge | Start | End | Source | Lookup needed | |---|---|---|---|---| ## Preproc And Lookup - ## Implementation Sequence 1. ``` Keep the brief practical. Avoid speculative resources, edges, or abstractions that are not supported by the target service requirements or API data. ## references/preproc-lookup.md # OpenHound Preproc And Lookup Use this reference when relationships or node properties require data from multiple collected tables. ## Purpose `preproc` loads selected raw JSONL tables into DuckDB, runs optional SQL transforms and produces lookup data used during `convert` through `self._lookup`. Use this path when a model needs to resolve information that is not present in its own raw row. ## Files Usually Touched - `src//transforms.py` - `src//lookup.py` - `src//main.py` - `src//models/.py` - `src//source.py` if new raw tables must be collected ## Transforms `transforms.py` should contain plain DuckDB SQL functions. Keep each transform focused and call them from a top-level `transforms` function. ```python import duckdb def create_joined_tables(con: duckdb.DuckDBPyConnection, schema: str = "myservice") -> None: con.execute( f""" CREATE OR REPLACE TABLE {schema}.asset_groups AS SELECT asset_id, group_id FROM {schema}.asset_memberships """ ) def transforms(con: duckdb.DuckDBPyConnection, schema: str = "myservice") -> None: create_joined_tables(con, schema) ``` Use parameter binding for values. Schema and table names are often interpolated because DuckDB does not bind identifiers; keep those values internal and trusted. Use DuckDB JSON operators for JSON object fields loaded from raw resources: ```sql SELECT metadata->>'name' AS name FROM myservice.assets ``` ## Preproc Registration The `preproc` function in `main.py` returns a mapping of DuckDB table name to JSONL table name. Only listed tables are loaded into the lookup DB. ```python @app.preproc(transformer=transforms) def preproc(ctx: PreProcContext) -> dict[str, str]: return { "assets": "assets", "asset_memberships": "asset_memberships", } ``` If no SQL transforms are needed, the transformer can be omitted. If lookup methods depend on transformed tables, ensure the transformer is registered. If `lookup.py` reads a table created by `transforms.py`, use `@app.preproc(transformer=transforms)`. ## Lookup Manager `lookup.py` should define one extension lookup class extending `LookupManager`. Use `@lru_cache` for repeated lookups. ```python from functools import lru_cache from openhound.core.lookup import LookupManager class EXLookup(LookupManager): def __init__(self, client: DuckDBPyConnection, schema: str = "myservice"): super().__init__(client, schema) self.schema = schema self.client = client @lru_cache def group_id_for(self, name: str) -> str | None: return self._find_single_object( f"SELECT node_id FROM {self.schema}.groups WHERE name = ?", [name], ) @lru_cache def all_assets_domain(self, domain: str) -> list: return self._find_all_objects( f"SELECT * FROM {self.schema}.assets WHERE domain = ?", [domain], ) ``` Prefer small lookup methods with clear names. The `_find_single_object` method will return `None` if no results are found. ## Convert Registration Register the lookup class on `@app.convert(...)`: ```python @app.convert(lookup=EXLookup) def convert(ctx: ConvertContext) -> DltSource: from .source import source as myservice_source return myservice_source(), {} ``` The lookup is injected into each `BaseAsset` as `self._lookup` during convert. ## Model Usage Use lookup methods from `as_node` or `edges` when needed: ```python @property def edges(self): group_id = self._lookup.group_id_for(self.group_name) if group_id is None: return yield Edge( kind=ek.MEMBER_OF, start=EdgePath(value=self.as_node.id, match_by="id"), end=EdgePath(value=group_id, match_by="id"), ) ``` Document in the collector README when users must run `preproc` before `convert`. ## Rules - Do not call `self._lookup` unless `convert` is registered with the lookup class. - Do not assume raw tables are available in DuckDB unless they are returned by `preproc`. - Keep transform output table names stable; lookup methods may depend on them. - Avoid loading unnecessary raw tables into preproc. - Cache deterministic lookup methods with `@lru_cache` or `@cache`. ## Checklist - Required raw tables are collected by `source.py` and match with the defined resource/transformer name. - Required raw tables are included in `main.py`'s `preproc` mapping. - SQL transforms create any derived tables used by lookup methods. - `@app.preproc(transformer=transforms)` is used when transforms are required. - `@app.preproc(transformer=transforms)` is used when lookup reads transformed tables. - `lookup.py` contains cached lookup methods. - `@app.convert(lookup=Lookup)` is registered. - Models handle missing lookup results. - Read `references/validate-extension.md` before finishing. ## references/register-extension.md # OpenHound Register Extension Use this reference when editing `main.py`, `extension.yaml`, package entry points, or extension identity metadata. ## Phase Registration `src//main.py` owns the single `OpenHound` app instance and registers all phases. ```python from dlt.extract.source import DltSource from openhound.core.app import OpenHound from openhound.core.collect import CollectContext from openhound.core.convert import ConvertContext from openhound.core.preproc import PreProcContext from .lookup import EXLookup from .transforms import transforms app = OpenHound("myservice", help="OpenGraph collector for MyService") @app.collect() def collect(ctx: CollectContext) -> DltSource: from .source import source as myservice_source return myservice_source() @app.preproc(transformer=transforms) def preproc(ctx: PreProcContext) -> dict[str, str]: return { "assets": "assets", } @app.convert(lookup=EXLookup) def convert(ctx: ConvertContext) -> DltSource: from .source import source as myservice_source return myservice_source(), {} ``` Import the DLT source function inside phase functions to avoid import cycles with models that import `app` from `main.py`. ## App Rules - Define exactly one `app = OpenHound(...)` instance. - Keep the app in `src//main.py`. - Models should import `app` from `openhound_.main`. - Do not create local app instances in model, source, graph, lookup, or transform modules. ## Collect Phase The collect phase should return the DLT source from `source.py`. ```python @app.collect() def collect(ctx: CollectContext) -> DltSource: from .source import source as myservice_source return myservice_source() ``` ## Preproc Phase The preproc phase maps DuckDB table names to JSONL table names. Only listed tables are loaded into the lookup DB. Use a transformer only when `transforms.py` contains SQL transforms: ```python @app.preproc(transformer=transforms) def preproc(ctx: PreProcContext) -> dict[str, str]: return {"assets": "assets"} ``` The left side is the DuckDB table name used in lookup methods. The right side is the resource name as defined in the `source.py` resource/transformer definition. Keep these aligned with the actual source output and lookup usage. ## Convert Phase The convert phase returns the DLT source and an extras dictionary. The extras dictionary can pass static values to assets during graph conversion through `self._extras`. ```python @app.convert(lookup=EXLookup) def convert(ctx: ConvertContext) -> DltSource: from .source import source as myservice_source return myservice_source(), {} ``` Omit `lookup=...` only when no model uses `self._lookup`. ## Extension Metadata `extension.yaml` declares extension identity, credentials and parameters. This file is only used for metadata and does not affect runtime behavior, but credential and parameter names should stay aligned with `source.py` secrets parameters. ```yaml name: myservice version: 0.1.0 type: local credentials: - name: token description: API token required: true parameters: - name: org description: Organisation slug required: true ``` ## Package Entry Point `pyproject.toml` should expose the app through the `openhound.sources` entry point group. ```toml [project.entry-points."openhound.sources"] myservice = "openhound_myservice.main:app" ``` Optional: For the cookiecutter template, preserve template variables when editing generated paths and names. ## Checklist - `main.py` has one `OpenHound` app instance. - All three `collect`, `preproc` (optional) and `convert` decorators are attached to that app instance. - Lazy source imports happen inside phase functions where needed to avoid cycles and improve startup performance. - Preproc table map includes every raw table needed by lookup if required. - Convert registers lookup when models use `self._lookup`. - `extension.yaml` matches source credentials and parameters. - `pyproject.toml` entry point targets `openhound_.main:app`. - Read `references/validate-extension.md` before finishing. ## references/source-collection.md # OpenHound Source Collection Use this reference when editing `src//source.py` or changing when/how upstream API data is collected. ## Source Pattern `source.py` wires API collection into OpenHound/DLT. It usually contains: - A `SourceContext` dataclass for the authenticated client and shared state. - `@app.resource(...)` functions defining how resources are collected from the API. - `@app.transformer(...)` functions for nested collection seeded by parent `@app.resource` resources. - An `@app.source(...)` function that declares credentials, builds context and returns resources/transformers to be processed. ## Authentication Credentials should come from DLT secrets under `[sources.source.]` in `.dlt/secrets.toml` and should be declared with `dlt.secrets.value` parameters on the source function. Example secrets file: ```toml [sources.source.myservice] token = "xxx" host = "https://api.myservice.com" org_name = "my-org" ``` Environment variables can also provide the same values, for example `SOURCES__SOURCE__MYSERVICE__TOKEN`. If authentication is complex, add a dedicated `auth.py` module. ## Source Context Use a context object to keep resource functions simple and consistent: ```python from dataclasses import dataclass from dlt.sources.rest_api import RESTClient @dataclass class SourceContext: client: RESTClient ``` Add shared identifiers, tenant/org names, rate limit helpers, or authenticated clients to this dataclass when needed. ## Rest API Client Important: Prefer using `dlt.sources.rest_api.RESTClient` for API collection. This client has built-in support for pagination, retries, rate limit handling and logging. If the API has specific needs, extend `RESTClient` with a custom client class in a dedicated `client.py` module. ## Resources And Transformers Use `@app.resource(name=..., columns=)` for top-level collection. The `columns` model should reference the `BaseAsset` Pydantic class that validates yielded rows. Use `@app.transformer(name=..., columns=)` for nested collection that depends on each parent item. ```python @app.resource(name="assets", parallelized=True, columns=Asset) def assets(ctx: SourceContext): for item in ctx.client.paginate("/assets"): yield item @app.transformer(name="asset_users", parallelized=True, columns=AssetUser) def asset_users(asset, ctx: SourceContext): for item in ctx.client.paginate(f"/assets/{asset.id}/users"): yield item ``` ## DLT Source Function Wrap the resources in an `@app.source(...)` function. Build the context once, create parent resources once, and use DLT's pipe operator for transformers. ```python import dlt from dlt.sources.rest_api import HeaderLinkPaginator, RESTClient from dlt.sources.rest_api.auth import BearerTokenAuth @app.source(name="myservice", max_table_nesting=0) def source(token=dlt.secrets.value, host=dlt.secrets.value): ctx = SourceContext( client=RESTClient( base_url=host, auth=BearerTokenAuth(token=token), paginator=HeaderLinkPaginator(), ) ) assets_resource = assets(ctx) return ( assets_resource, assets_resource | asset_users(ctx), ) ``` Use `max_table_nesting=0` unless there is a concrete reason to let DLT infer nested tables. ## Collection Rules - Yield raw dictionaries or objects shaped to match the `columns` model. - Keep API pagination inside resource and transformer functions. - Keep node/edge conversion logic out of `source.py`. Conversion belongs in model classes. - Do not make resource functions reach into DuckDB lookup data. - Avoid collecting fields that are not needed for graph conversion, lookup, metadata, or debugging. - Do not collect or emit secrets, tokens, credentials, private keys, or credential-equivalent material. - Name resources after the raw table they produce, using stable plural table names where practical. ## Checklist - Credentials are declared with `dlt.secrets.value`. - Source context contains authenticated clients and shared state. - Each resource or transformer has the correct `columns=`. - The Pydantic model should be saved under `models/` and exported from `models/__init__.py`. - Parent resources reused by transformers are assigned to variables before piping. - All resources/transformers are returned from the source function. - Read `references/validate-extension.md` before finishing. ## references/validate-extension.md # OpenHound Validate Extension Use this reference before finishing changes to an OpenHound collector. ## Structural Checks Review these before finishing: - Kind strings are defined only in `kinds/nodes.py` and `kinds/edges.py`. - Model files import kind constants instead of hardcoding strings. - Node IDs are stable strings and not raw integer primary keys. - The collector defines and emits a root/environment node. - Every emitted node sets `environmentid` to the root/environment node ID. - Every OpenGraph property dataclass field is documented in the class docstring's `Attributes` section. - Every node-bearing asset declares `NodeDef(properties=...)`. - `EdgeDef(...)` declarations match edges actually yielded by the same asset class. - Edges to existing nodes use `ConditionalEdgePath` when property-based resolution is more reliable than constructing or guessing an ID. - Edge properties use `yield` or `yield from` unless a list is explicitly justified. - Models using `self._lookup` have `@app.convert(lookup=...)` registered. - Tables required by lookup methods are included in the `preproc` map or created by transforms. - `source.py` credentials are declared with `dlt.secrets.value`. - `extension.yaml` credentials and parameters match the source function inputs. - `models/__init__.py` exports newly added models. ## Validation Commands Use an isolated uv virtual environment outside the repository so validation does not modify the user's local `.venv`: ```bash export UV_PROJECT_ENVIRONMENT=/tmp/openhound--venv uv run pytest uv run ruff check src/ uv run mypy src/ ``` Run the checks that are available for the generated collector. If a command cannot run because dependencies, credentials, generated template variables, or external services are unavailable, report the skipped check and reason. ## Common Anti-Patterns | Do not | Prefer | |---|---| | Hardcode `"EX_Asset"` in a model | Import `nk.ASSET` or `ek.RELATIONSHIP` from `kinds/`. | | Use `id: int` as the OpenGraph node ID | Assign `self.id` from a stable string property or `BaseNode.guid(...)`. | | Call `self._lookup` without preproc data | Register lookup and load or transform the required tables. | | Add dataclass fields without descriptions | Add an `Attributes` entry describing each dataclass field. | | Create multiple `OpenHound` app instances | Keep one app in `main.py`. | | Declare edges on a different asset than the emitter | Put `EdgeDef(...)` on the emitting asset. | | Emit nodes without `environmentid` | Set `environmentid` to the root/environment node ID. | | Guess another node's ID manually | Use `ConditionalEdgePath` with stable `PropertyMatch` constraints. | ## Search Checks Use AST or search checks where practical: - No enum-style kind classes or kinds using fixed strings remain in model code. - No `NodeDef(...)` is missing `properties=...`. - Node property instantiations include required base fields such as `environmentid`. ## Final Response Guidance When reporting completion, include: - What nodes/edges are added or modified. - What changes are made to the collection pipeline. - Which validation commands ran or were skipped. - Any other relevant changes. - Any remaining risks or follow-up work.