Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Autotests/mock/test_workflow_plugin_mock.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
DEMO_MESSAGE = "This is a test workflow demonstration"

RESEARCH_WORKFLOW = "research-workflow"
RESEARCH_DIR = "/PeTTa/repos/Omega/memory/workflow_space/research"
RESEARCH_DIR = "/PeTTa/repos/Omega/memory/workflow/space/research"
RESEARCH_NAME = "qa-research-autotest"


Expand Down
16 changes: 10 additions & 6 deletions config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -157,13 +157,17 @@ asione_model: "asi1-ultra"
# Plugins

# Workflow
# Path to the directory which contains available workflows. Default value is
# <project_root>/plugins/workflow/instructions. See plugins/workflow/README.md
# for details.
#pluginWorkflowInstructionsDir: ./plugins/workflow/instructions
# Path to the directory which contains built-in workflows shipped with Omega.
# Default value is <project_root>/plugins/workflow/instructions. See
# plugins/workflow/README.md for details.
pluginWorkflowInstructionsDir: ./repos/Omega/plugins/workflow/instructions
# Path to the directory which contains custom workflows. A custom workflow
# overrides the built-in one with the same name. Default value is
# <memoryDirectory>/workflow/skills
pluginWorkflowCustomInstructionsDir: ./repos/Omega/memory/workflow/skills
# Path to the directory to keep workflow working files when workflow is active.
# Default value is <memoryDirectory>/workflow_space
#pluginWorkflowMemoryDir: ./memory/workflow_space
# Default value is <memoryDirectory>/workflow/space
pluginWorkflowMemoryDir: ./repos/Omega/memory/workflow/space

# OpenClaw
openClawEnabled: disabled
Expand Down
2 changes: 1 addition & 1 deletion docs/reference-channels.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Minimal JSON chat adapter over a WebSocket connection. Selected with `commchanne
- `stop_websocket()` — stop the listener thread and close the socket.
- Requires the `websockets` Python package.
- When `WS_TOKEN` is set it is sent as an `Authorization: Bearer <token>` header. Unlike the IRC/Telegram/Slack adapters there is no one-time `auth <secret>` gate — trust is established by the endpoint URL and bearer token.
- Supports immediate `/memory-export history|ltm|both` commands when memory export is enabled and `WS_TOKEN` is configured. The export handler uses a SHA-256-derived connection principal and never exposes the bearer token. Protect the endpoint with `wss://` and server-side access controls because WebSocket does not have the per-user ownership gate used by the other channels.
- Supports immediate `/memory-export history|ltm|skills|all` commands when memory export is enabled and `WS_TOKEN` is configured. The export handler uses a SHA-256-derived connection principal and never exposes the bearer token. Protect the endpoint with `wss://` and server-side access controls because WebSocket does not have the per-user ownership gate used by the other channels.
- Reconnects automatically with exponential backoff (1s → 30s, ±20% jitter) and is safe to start once at process startup.

### Frame protocol
Expand Down
19 changes: 14 additions & 5 deletions docs/reference-memory-portability.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ token so the credential itself is never used as an identifier:
```text
/memory-export history
/memory-export ltm
/memory-export both
/memory-export skills
/memory-export all
```

Completion is delivered through the active channel and includes the filename,
Expand Down Expand Up @@ -75,10 +76,15 @@ manifest.json
history/history.metta
vector/collections.json
vector/records.jsonl
skills/...
space/...
```

History is the conversation trace. LTM is logical user-memory records from
ChromaDB. Prompts, credentials, logs, skills, and other operational state are
ChromaDB. Skills are custom workflows from `pluginWorkflowCustomInstructionsDir`
together with workflow working files from `pluginWorkflowMemoryDir`; only
regular files are exported, symbolic links and empty directories are skipped.
Built-in workflows, prompts, credentials, logs, and other operational state are
not exported. SHA-256 detects corruption, not archive authorship.

## Import
Expand Down Expand Up @@ -106,9 +112,12 @@ scripts/omega start -d "$image" -p OpenAI -t telegram \
```

`overwrite` replaces the selected components after validation and rollback
preparation. `append` preserves existing history and adds imported LTM records
under new IDs. Select a single component with `--only-history` or
`--only-vector`. Without either option, both components are imported.
preparation. For skills it replaces both the custom skills and the workflow
space directories, even when the archive contains no skills. `append` preserves
existing history and adds imported LTM records under new IDs, and copies imported
skills and space files over the existing ones with the same names. Select a
single component with `--only-history`, `--only-vector`, or `--only-skills`.
Without these options, all components present in the archive are imported.

The importer validates archive paths, checksums, manifest metadata, record
counts, and embedding compatibility before changing live memory. It runs before
Expand Down
1 change: 1 addition & 0 deletions entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ elif operation == "import":
mode=os.environ.get("MEMORY_IMPORT_MODE", "overwrite"),
include_history=os.environ.get("MEMORY_IMPORT_NO_HISTORY") != "1",
include_vectors=os.environ.get("MEMORY_IMPORT_NO_VECTOR") != "1",
include_skills=os.environ.get("MEMORY_IMPORT_NO_SKILLS") != "1",
)
else:
raise ValueError(f"Unsupported memory portability operation: {operation!r}")'
Expand Down
25 changes: 20 additions & 5 deletions plugins/workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,19 @@ For example:
Start an agent and ask it to use the workflow. For the test workflow above
send: `Demonstrate workflow plugin`

New workflows put into `pluginWorkflowInstructionsDir` while the agent is
## Built-in and custom workflows

Workflows are looked up in two directories:
- built-in workflows shipped with Omega are kept in
`pluginWorkflowInstructionsDir` and are updated together with Omega;
- custom workflows are kept in `pluginWorkflowCustomInstructionsDir` which is
placed in the memory directory by default, so they survive container
restarts and image updates. The directory is created on start if missing.

A custom workflow overrides the built-in one with the same name. To change a
built-in workflow copy it into the custom directory and edit the copy.

New workflows put into `pluginWorkflowCustomInstructionsDir` while the agent is
running become available after the agent calls the `workflow-refresh-list`
skill. Ask the agent to refresh the list of workflows, or just ask it to load
the new workflow by name.
Expand All @@ -69,11 +81,14 @@ the new workflow by name.

Wofkflow plugin Omega configuration parameters:
- `pluginWorkflowInstructionsDir` - path to the directory which contains
available workflows. Default value is `<project
root>/plugins/workflow/instructions`
built-in workflows. Default value is `<project_root>/plugins/workflow/instructions`
- `pluginWorkflowCustomInstructionsDir` - path to the directory which contains
custom workflows. Default value is `<memoryDirectory>/workflow/skills`
- `pluginWorkflowMemoryDir` - path to the directory to keep workflow working
files when workflow is active. Default value is `<project
root>/memory/workflow_space`
files when workflow is active. Default value is `<memoryDirectory>/workflow/space`

Custom and working directories are created on start together with missing
parent directories.

One can set this parameters passing them as a command line arguments. For
example:
Expand Down
53 changes: 42 additions & 11 deletions plugins/workflow/workflow.metta
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
!(import! &self (library lib_spaces))

(= (pluginWorkflowInstructionsDir) (empty))
(= (pluginWorkflowCustomInstructionsDir) (empty))
(= (pluginWorkflowMemoryDir) (empty))

(= (loadOmegaPlugin)
Expand All @@ -9,10 +10,15 @@
(configure pluginWorkflowInstructionsDir (joinPath ((projectRootDirectory) "plugins" "workflow" "instructions")))
(if (not (exists-directory (pluginWorkflowInstructionsDir)))
(Error (_loadOmegaPlugin) (strings-concat ("Instructions directory: " (pluginWorkflowInstructionsDir) " doesn't exist")))
(progn
(configure pluginWorkflowMemoryDir (joinPath ((memoryDirectory) workflow_space)))
(progn
(configure pluginWorkflowCustomInstructionsDir (joinPath ((memoryDirectory) "workflow" "skills")))
(let $customrc (make-directory-path (pluginWorkflowCustomInstructionsDir)) ())
(if (not (exists-directory (pluginWorkflowCustomInstructionsDir)))
(log ERROR "workflow-plugin" (strings-concat ("Could not create custom instructions directory: " (pluginWorkflowCustomInstructionsDir)", only built-in workflows are available")))
())
(configure pluginWorkflowMemoryDir (joinPath ((memoryDirectory) "workflow" "space")))
(log INFO "workflow-plugin" (strings-concat ("Creating working directory: " (pluginWorkflowMemoryDir))))
(let $rc (make-directory (pluginWorkflowMemoryDir)) ())
(let $rc (make-directory-path (pluginWorkflowMemoryDir)) ())
(if (exists-directory (pluginWorkflowMemoryDir))
(progn
(log INFO "workflow-plugin" "Adding plugin's skills")
Expand Down Expand Up @@ -45,12 +51,19 @@
; False))

(= (instructionsDir) (get-existing-dir (pluginWorkflowInstructionsDir)))

;; custom workflow overrides built-in one with the same name
(= (workflow-dir $name)
(if (exists-file (joinPath ((pluginWorkflowCustomInstructionsDir) $name "SKILL.md")))
(pluginWorkflowCustomInstructionsDir)
(instructionsDir)))

(= (workflow-load-instructions $name)
(let $active (get-state &workflow_active_name)
(if (== $active "")
(if (not (valid-file-or-folder-name $name))
(Error (quote (workflow-load-instructions $name)) (strings-concat ("ERROR: invalid workflow name: " $name)))
(let $path (joinPath ((instructionsDir) $name "SKILL.md"))
(let $path (joinPath ((workflow-dir $name) $name "SKILL.md"))
(if (not (exists-file $path))
(Error (quote (workflow-load-instructions $name)) (strings-concat ("ERROR: skill file not found: " $path ". Try to call workflow-refresh-list")))
(let $content (read-file $path)
Expand Down Expand Up @@ -83,7 +96,7 @@

; to load skills descriptions which are defined only for current workflow
(= (add-workflow-skills $name)
(let $path (joinPath ((instructionsDir) $name "skill"))
(let $path (joinPath ((workflow-dir $name) $name "skill"))
(progn
(import! &workflow_skills $path)
(collapse (match &workflow_skills (skill $func $desc $args) (add-skill $func $desc $args)))
Expand All @@ -102,11 +115,29 @@
(add-prompt-extension workflow_active_instructions $content)))

;;========================== Functions to load workflow description =======================
;; returns workflow name and its description
;; returns workflow name and its description, custom workflows override built-in ones with the same name
(= (workflow-list-available)
(let* ( ($base (instructionsDir))
($dirs (directory-dirs $base)))
(collect-description $dirs $base)))
($builtin (collect-description (filter-shadowed (directory-dirs $base)) $base ()))
($custom (pluginWorkflowCustomInstructionsDir)))
(if (exists-directory $custom)
(collect-description (directory-dirs $custom) $custom $builtin)
$builtin)))

;; drops built-in workflows overridden by custom ones with the same name
(= (filter-shadowed ()) ())
(= (filter-shadowed (cons $first $tail))
(let $rest (filter-shadowed $tail)
(if (exists-file (joinPath ((pluginWorkflowCustomInstructionsDir) $first "SKILL.md")))
$rest
(cons-atom $first $rest))))

;; re-reads available workflows into the prompt
(= (workflow-refresh-list)
(let $text (join (newline) (cons-atom "Available workflows: " (workflow-list-available)))
(progn
(remove-prompt-extension workflows-available)
(add-prompt-extension workflows-available $text))))

;; re-reads available workflows into the prompt
(= (workflow-refresh-list)
Expand All @@ -116,10 +147,10 @@
(add-prompt-extension workflows-available $text))))

;; collects workflow description from description.txt
(= (collect-description () $base) ())
(= (collect-description (cons $first $tail) $base)
(= (collect-description () $base $acc) $acc)
(= (collect-description (cons $first $tail) $base $acc)
(let* (($path (joinPath ($base $first "description.txt")))
($rest (collect-description $tail $base)))
($rest (collect-description $tail $base $acc)))
(if (exists-file $path)
(cons-atom (strings-concat ($first ": " (read-file $path))) $rest)
$rest)))
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ chromadb==1.5.9
openai==2.38.0
transformers==5.8.0
sentence-transformers==5.5.1
import-kb==0.2.5
import-kb==0.2.6
py-landlock==0.1.1
pyyaml==6.0.3
ddgs==9.14.4
Expand Down
22 changes: 15 additions & 7 deletions scripts/omega
Original file line number Diff line number Diff line change
Expand Up @@ -673,6 +673,7 @@ help() {
echo -e "\t--memory-mode overwrite|append import mode (default: overwrite)"
echo -e "\t--only-history restore history only"
echo -e "\t--only-vector restore long-term memory only"
echo -e "\t--only-skills restore custom skills and workflow space only"
}

require_option_value() {
Expand Down Expand Up @@ -704,6 +705,7 @@ options() {
memory_import_mode="overwrite"
memory_import_only_history=0
memory_import_only_vector=0
memory_import_only_skills=0
memory_export_enabled=0

if [[ "$#" -eq 0 ]]; then
Expand Down Expand Up @@ -767,6 +769,10 @@ options() {
memory_import_only_vector=1
shift
;;
--only-skills)
memory_import_only_skills=1
shift
;;
--version|-v) version; exit 0;;
--help|-h) help; exit 0;;
*)
Expand Down Expand Up @@ -854,13 +860,12 @@ options() {
validate_memory_archive_name "${memory_import_file}" "--memory-import" || return 1
fi

if [[ "${memory_import_only_history}" == "1" && "${memory_import_only_vector}" == "1" ]]; then
echo "--only-history and --only-vector cannot be combined" >&2
if (( memory_import_only_history + memory_import_only_vector + memory_import_only_skills > 1 )); then
echo "--only-history, --only-vector and --only-skills cannot be combined" >&2
return 1
fi
if [[ -z "${memory_import_file}" &&
( "${memory_import_only_history}" == "1" ||
"${memory_import_only_vector}" == "1" ) ]]; then
if [[ -z "${memory_import_file}" ]] &&
(( memory_import_only_history + memory_import_only_vector + memory_import_only_skills > 0 )); then
echo "Import component flags require --memory-import" >&2
return 1
fi
Expand Down Expand Up @@ -1140,10 +1145,13 @@ if sys.argv[1]:
-e "MEMORY_IMPORT_MODE=${memory_import_mode}"
)
if [[ "${memory_import_only_history}" == "1" ]]; then
memory_import_env+=(-e MEMORY_IMPORT_NO_VECTOR=1)
memory_import_env+=(-e MEMORY_IMPORT_NO_VECTOR=1 -e MEMORY_IMPORT_NO_SKILLS=1)
fi
if [[ "${memory_import_only_vector}" == "1" ]]; then
memory_import_env+=(-e MEMORY_IMPORT_NO_HISTORY=1)
memory_import_env+=(-e MEMORY_IMPORT_NO_HISTORY=1 -e MEMORY_IMPORT_NO_SKILLS=1)
fi
if [[ "${memory_import_only_skills}" == "1" ]]; then
memory_import_env+=(-e MEMORY_IMPORT_NO_HISTORY=1 -e MEMORY_IMPORT_NO_VECTOR=1)
fi
fi

Expand Down
10 changes: 8 additions & 2 deletions src/memory_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ def _resolve_chroma_path() -> Path:
return Path(str(configured)).expanduser().resolve()


def _resolve_config_path(key: str) -> Path:
return Path(str(config_get_by_key(key))).expanduser().resolve()


def _cloud_embedder(provider, model):
def embed(texts):
from import_knowledge.import_knowledge import embed_batch, init_embeddings
Expand Down Expand Up @@ -67,6 +71,8 @@ def create_memory_store():
memory_dir=_resolve_memory_dir(),
chroma_path=_resolve_chroma_path(),
collection_name="memories",
skills_dir=_resolve_config_path("pluginWorkflowCustomInstructionsDir"),
space_dir=_resolve_config_path("pluginWorkflowMemoryDir"),
**_embedding_options(),
)

Expand All @@ -89,7 +95,7 @@ def _get_transfer():
return _transfer


_VALID_COMPONENTS = ("history", "ltm", "both")
_VALID_COMPONENTS = ("history", "ltm", "skills", "all")


def is_export_enabled() -> bool:
Expand Down Expand Up @@ -135,7 +141,7 @@ def handle_export_command(

return (
"Unknown /memory-export command. "
"Use: /memory-export history|ltm|both"
"Use: /memory-export history|ltm|skills|all"
)


Expand Down
5 changes: 5 additions & 0 deletions src/utils.metta
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
!(import_prolog_function sread)
!(import_prolog_function string_concat)
!(import_prolog_function make_directory)
!(import_prolog_function make_directory_path)

(= (sleep $t)
(progn (translatePredicate (sleep $t))
Expand All @@ -29,6 +30,10 @@
(let $rc (catch (((translatePredicate (make_directory $path)))))
(return-on-error $rc True)))

(= (make-directory-path $path)
(let $rc (catch (((translatePredicate (make_directory_path $path)))))
(return-on-error $rc True)))

(= (string-replace $str $a $b)
(atomic_list_concat (split_string $str $a "") $b))

Expand Down
Loading
Loading