Skip to main content

Plugin discovery

Discovery is the process by which the registry locates plugins on the file system. The implementation rests on the file-based model declared in ADR-0006: a plugin is a folder with a dagstack.toml. Everything else is an implementation detail.

Directory layout​

my-app/
├── plugins/
│   ├── openai/
│   │   ├── dagstack.toml
│   │   └── plugin.py
│   ├── qdrant/
│   │   ├── dagstack.toml
│   │   └── plugin.py
│   └── echo/
│       ├── dagstack.toml
│       └── plugin.py
└── main.py

Every folder containing a dagstack.toml becomes one plugin. Nested plugin folders are not supported — a plugin cannot live inside another plugin.

Calling discovery​

import asyncio
import logging
from dagstack.plugin_system import PluginContext, PluginRegistry


async def main() -> None:
    registry = PluginRegistry()
    registry.discover("plugins/")

    ctx = PluginContext(
        config={},
        logger=logging.getLogger("app"),
        registry=registry,
    )
    await registry.setup_all(ctx)


asyncio.run(main())

Relative paths are resolved against the process's current working directory. In production we recommend passing an absolute path or a path relative to a known application root.

Discovery rules​

  1. The walk is recursive from the given folder downwards. Depth is unlimited.
  2. Every folder is checked for dagstack.toml. If the file is present, the folder becomes a plugin and its contents are not scanned further (nested plugins are forbidden).
  3. Folders without dagstack.toml are traversed transparently. This lets you group plugins into thematic subfolders (plugins/data-sources/, plugins/llm/).
  4. Plugins sharing the same kind+name pair raise AmbiguousPlugin. The name must be unique within a kind.

Ignored paths​

By default discovery skips system and housekeeping folders:

__pycache__/
node_modules/
.git/
.venv/
venv/
.mypy_cache/
.pytest_cache/
.ruff_cache/
.tox/
dist/
build/

The full list lives in the DEFAULT_IGNORE constant. To add your own rules pass the ignore parameter:

from dagstack.plugin_system import PluginRegistry
from dagstack.plugin_system import DEFAULT_IGNORE

registry = PluginRegistry()
registry.discover(
    "plugins/",
    ignore=[*DEFAULT_IGNORE, "experimental/*", "*.draft"],
)

Patterns support glob syntax (*, **, ?).

Discovering multiple roots​

If the application combines plugins from several independent folders (for example, core plugins next to main.py and user plugins in ~/.config/my-app/plugins/), run discover against each root and merge the results:

:::warning merge is planned for Phase 2 Today the Python binding accepts a single discovery root per PluginRegistry. Combining several roots — and the matching merge / override semantics described below — is planned for Phase 2. For now, lay your plugins out under one discovery root and call registry.discover("plugins/") once. :::

When merge ships in Phase 2, conflicts at the kind+name level surface as AmbiguousPlugin. For the "user overrides a system plugin" case the explicit override mechanism (described below) is the planned path.

Overriding plugins (Phase 2)​

Sometimes you need to swap a plugin out for testing or local development. The Phase 2 release will add an explicit override flow that takes a second registry and lets entries from it replace those of the first without raising AmbiguousPlugin. Until then, point a separate PluginRegistry() at the fixture folder and use it directly in the test:

:::warning override is planned for Phase 2 Today the Python binding does not provide an explicit override flow — discovery is single-rooted. To swap a plugin out for tests, point a test-only registry at a fixture folder (registry.discover("tests/fixtures/plugins/")) and use it directly. The conflict-aware override API lands together with merge in Phase 2. :::

When override ships, conflicts will not be treated as errors — entries from the second registry replace entries from the first.

Performance​

For large directories (≥1000 plugins) the walk takes noticeable time. Recommendations:

  • Run discover once at application start-up and cache the result.
  • Use ignore to skip folders known to contain no plugins (tests, docs).
  • If the plugins are known up front, pass an explicit list of paths instead of a recursive walk.