Skip to content

Commit 08a1bd1

Browse files
tjbcktjbck
committed
doc: readme
1 parent 24ee149 commit 08a1bd1

1 file changed

Lines changed: 50 additions & 157 deletions

File tree

‎README.md‎

Lines changed: 50 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -1,207 +1,101 @@
11
# 📚 oikb
22

3-
A CLI tool that syncs content to [Open WebUI](https://github.com/open-webui/open-webui) Knowledge Bases. 30+ connectors for local files, Git repos, cloud storage, wikis, ticketing, messaging, CRM, and more.
3+
Keep your [Open WebUI](https://github.com/open-webui/open-webui) Knowledge Bases in sync. Point it at a local directory, a GitHub repo, a Confluence space, an S3 bucket, or any of 30+ supported sources. Only new and modified files are uploaded via incremental SHA-256 diffing.
44

55
> [!IMPORTANT]
6-
> Requires **Open WebUI 0.9.6+** (uses the Knowledge Base sync API).
6+
> Requires **Open WebUI 0.9.6+**
77
8-
## Why oikb?
8+
## Quick Start
99

10-
Manually uploading files to Knowledge Bases is tedious. oikb automates it with incremental SHA-256 diffing, so only new and modified files are uploaded. Directory structure is mirrored automatically. Run it once, run it in CI, or leave it watching for changes.
11-
12-
## Connectors
13-
14-
oikb connects to all your tools. One CLI, incremental sync, automatic directory mirroring.
15-
16-
### Code Repositories
17-
18-
| Source | Syntax | Auth |
19-
|---|---|---|
20-
| **GitHub** | `github:owner/repo` | `GITHUB_TOKEN` |
21-
| **GitLab** | `gitlab:owner/repo` | `GITLAB_TOKEN` |
22-
| **Bitbucket** | `bitbucket:owner/repo` | `BITBUCKET_USER` + `BITBUCKET_APP_PASSWORD` |
23-
24-
### Cloud Storage
25-
26-
| Source | Syntax | Auth |
27-
|---|---|---|
28-
| **Local directory** | `./docs` | - |
29-
| **Amazon S3** | `s3://bucket/prefix` | AWS credentials |
30-
| **Google Cloud Storage** | `gs://bucket/prefix` | `GOOGLE_APPLICATION_CREDENTIALS` |
31-
| **Azure Blob Storage** | `az://container/prefix` | `AZURE_STORAGE_CONNECTION_STRING` |
32-
| **Dropbox** | `dropbox:/path` | `DROPBOX_TOKEN` |
33-
| **Cloudflare R2** | `r2://bucket/prefix` | `R2_ACCOUNT_ID` + R2 keys |
34-
35-
### Knowledge Bases & Wikis
36-
37-
| Source | Syntax | Auth |
38-
|---|---|---|
39-
| **Confluence** | `confluence:SPACEKEY` | `CONFLUENCE_URL` + `CONFLUENCE_TOKEN` |
40-
| **Notion** | `notion:<database-id>` | `NOTION_TOKEN` |
41-
| **BookStack** | `bookstack:` | `BOOKSTACK_URL` + `BOOKSTACK_TOKEN_*` |
42-
| **Discourse** | `discourse:category` | `DISCOURSE_URL` + `DISCOURSE_API_KEY` |
43-
| **GitBook** | `gitbook:<space-id>` | `GITBOOK_TOKEN` |
44-
| **Guru** | `guru:` | `GURU_USER` + `GURU_TOKEN` |
45-
46-
### Ticketing & Task Management
47-
48-
| Source | Syntax | Auth |
49-
|---|---|---|
50-
| **Jira** | `jira:PROJECT` | `JIRA_URL` + `JIRA_TOKEN` |
51-
| **Linear** | `linear:<team-id>` | `LINEAR_TOKEN` |
52-
| **Asana** | `asana:<project-id>` | `ASANA_TOKEN` |
53-
| **ClickUp** | `clickup:<space-id>` | `CLICKUP_TOKEN` |
54-
| **Zendesk** | `zendesk:<subdomain>` | `ZENDESK_SUBDOMAIN` + `ZENDESK_TOKEN` |
55-
| **Freshdesk** | `freshdesk:<domain>` | `FRESHDESK_DOMAIN` + `FRESHDESK_TOKEN` |
56-
| **Airtable** | `airtable:<base-id>` | `AIRTABLE_TOKEN` |
57-
58-
### Messaging
59-
60-
| Source | Syntax | Auth |
61-
|---|---|---|
62-
| **Slack** | `slack:<channel-id>` | `SLACK_TOKEN` |
63-
| **Discord** | `discord:<channel-id>` | `DISCORD_TOKEN` |
64-
| **Microsoft Teams** | `teams:<team-id>/<channel-id>` | `TEAMS_*` credentials |
65-
| **Gmail** | `gmail:user@domain.com` | `GOOGLE_APPLICATION_CREDENTIALS` |
66-
67-
### Sales & CRM
68-
69-
| Source | Syntax | Auth |
70-
|---|---|---|
71-
| **Salesforce** | `salesforce:` | `SALESFORCE_URL` + `SALESFORCE_TOKEN` |
72-
| **HubSpot** | `hubspot:` | `HUBSPOT_TOKEN` |
73-
74-
### Other
10+
```bash
11+
pip install oikb
7512

76-
| Source | Syntax | Auth |
77-
|---|---|---|
78-
| **Website / Sitemap** | `web:https://docs.example.com` | - |
79-
| **Google Drive** | `gdrive:<folder-id>` | `GOOGLE_APPLICATION_CREDENTIALS` |
80-
| **SharePoint** | `sharepoint:site/library` | `SHAREPOINT_*` credentials |
13+
export OPEN_WEBUI_URL=http://localhost:3000
14+
export OPEN_WEBUI_API_KEY=sk-your-api-key
8115

82-
Most connectors use `httpx` (included). Some require an extra:
16+
# Sync a directory to a Knowledge Base
17+
oikb sync ./docs --kb-id your-kb-id
8318

84-
```bash
85-
pip install oikb[gdrive] # Google Drive
86-
pip install oikb[gmail] # Gmail
87-
pip install oikb[s3] # Amazon S3, Cloudflare R2
88-
pip install oikb[gcs] # Google Cloud Storage
89-
pip install oikb[azure] # Azure Blob Storage
90-
pip install oikb[dropbox] # Dropbox
91-
pip install oikb[web] # Web crawler
92-
pip install oikb[all] # Everything
19+
# Or watch for changes and auto-sync continuously
20+
oikb watch ./docs --kb-id your-kb-id
9321
```
9422

95-
## Getting Started
96-
97-
### Docker (recommended)
23+
Or with Docker:
9824

9925
```bash
10026
docker run --rm \
10127
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
10228
-e OPEN_WEBUI_API_KEY=sk-your-key \
10329
-v ./docs:/data \
104-
ghcr.io/open-webui/oikb sync /data --kb-id your-kb-id
105-
```
106-
107-
### pip
108-
109-
```bash
110-
pip install oikb
111-
```
112-
113-
```bash
114-
export OPEN_WEBUI_URL=http://localhost:3000
115-
export OPEN_WEBUI_API_KEY=sk-your-api-key
116-
117-
oikb sync ./docs --kb-id your-kb-id
30+
ghcr.io/open-webui/oikb watch /data --kb-id your-kb-id
11831
```
11932

120-
> [!TIP]
121-
> No config file needed. Env vars are enough. You can also use `oikb config set url ...` and `oikb config set token ...` to save them to `~/.config/oikb/config.yaml`.
122-
12333
## Commands
12434

35+
| Command | Description |
36+
|---|---|
37+
| `oikb sync <source>` | Incremental sync to a Knowledge Base |
38+
| `oikb watch <source>` | Watch for changes and auto-sync |
39+
| `oikb diff <source>` | Preview what a sync would do |
40+
| `oikb ls` | List files in a Knowledge Base |
41+
| `oikb status` | Show KB info and file count |
42+
| `oikb reset` | Delete all files in a Knowledge Base |
43+
| `oikb config` | Manage saved URL and API key |
44+
45+
## 30+ Connectors
46+
47+
Beyond local directories, oikb can sync from remote sources using the same `oikb sync <source>` interface.
48+
49+
| Category | Sources |
50+
|---|---|
51+
| **Code Repos** | GitHub, GitLab, Bitbucket |
52+
| **Cloud Storage** | S3, GCS, Azure Blob, Dropbox, R2, Google Drive, SharePoint |
53+
| **Wikis & KBs** | Confluence, Notion, BookStack, Discourse, GitBook, Guru |
54+
| **Ticketing** | Jira, Linear, Zendesk, Freshdesk, Asana, ClickUp, Airtable |
55+
| **Messaging** | Slack, Discord, Microsoft Teams, Gmail |
56+
| **Sales & CRM** | Salesforce, HubSpot |
57+
| **Web** | Website / Sitemap crawler |
58+
12559
```bash
126-
# Sync from any connector
127-
oikb sync ./docs --kb-id your-kb-id
12860
oikb sync github:owner/repo --kb-id your-kb-id
12961
oikb sync confluence:ENG --kb-id your-kb-id
130-
oikb sync notion:abc123 --kb-id your-kb-id
131-
oikb sync slack:C0123ABC --kb-id your-kb-id
132-
oikb sync jira:PROJ --kb-id your-kb-id
13362
oikb sync s3://bucket/prefix --kb-id your-kb-id
134-
oikb sync web:https://docs.example.com --kb-id your-kb-id
135-
136-
# Preview changes without uploading
137-
oikb diff ./docs --kb-id your-kb-id
138-
139-
# Watch for changes and auto-sync
140-
oikb watch ./docs --kb-id your-kb-id
141-
142-
# List files / show info / reset
143-
oikb ls --kb-id your-kb-id
144-
oikb status --kb-id your-kb-id
145-
oikb reset --kb-id your-kb-id
63+
oikb sync slack:C0123ABC --kb-id your-kb-id
14664
```
14765

66+
Some connectors need an optional extra: `pip install oikb[gdrive]`, `pip install oikb[s3]`, or `pip install oikb[all]` for everything.
67+
14868
## Configuration
14969

150-
Settings are resolved in this order (highest priority wins):
70+
Resolved in order (highest priority wins):
15171

15272
1. **CLI flags** (`--url`, `--token`)
15373
2. **Environment variables** (`OPEN_WEBUI_URL`, `OPEN_WEBUI_API_KEY`)
15474
3. **Config file** (`~/.config/oikb/config.yaml`)
15575

156-
## Declarative Config (`.oikb.yaml`)
76+
### Multi-source (`.oikb.yaml`)
15777

158-
For recurring multi-source syncs:
78+
Define multiple sources in a single config file:
15979

16080
```yaml
16181
sync:
16282
- source: ./docs
16383
kb: project-docs
164-
16584
- source: github:owner/wiki
16685
kb: team-wiki
16786
branch: main
168-
16987
- source: confluence:ENG
17088
kb: eng-handbook
171-
172-
- source: slack:C0123ABC
173-
kb: team-updates
174-
175-
- source: jira:PROJ
176-
kb: project-issues
17789
```
17890
17991
```bash
18092
oikb sync # Sync all entries
18193
oikb sync --name docs # Sync a specific entry
18294
```
18395

184-
## Docker
185-
186-
Published at `ghcr.io/open-webui/oikb`.
187-
188-
```bash
189-
# One-shot sync
190-
docker run --rm \
191-
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
192-
-e OPEN_WEBUI_API_KEY=sk-your-key \
193-
-v ./docs:/data \
194-
ghcr.io/open-webui/oikb sync /data --kb-id your-kb-id
195-
196-
# Watch mode
197-
docker run --rm -d \
198-
-e OPEN_WEBUI_URL=http://host.docker.internal:3000 \
199-
-e OPEN_WEBUI_API_KEY=sk-your-key \
200-
-v ./docs:/data \
201-
ghcr.io/open-webui/oikb watch /data --kb-id your-kb-id
202-
```
96+
## Docker Compose
20397

204-
### Docker Compose
98+
Run as a sidecar alongside Open WebUI:
20599

206100
```yaml
207101
services:
@@ -223,7 +117,7 @@ services:
223117
restart: unless-stopped
224118
```
225119
226-
### GitHub Actions
120+
## GitHub Actions
227121
228122
```yaml
229123
- name: Sync docs to Open WebUI
@@ -237,11 +131,10 @@ services:
237131
238132
## How It Works
239133
240-
1. Scan source (local dir, GitHub, Confluence, S3, etc.), compute checksums
241-
2. Send manifest to Open WebUI's `/sync/diff` endpoint
242-
3. Server diffs against stored file hashes
243-
4. Delete stale files, create missing directories
244-
5. Upload only new and modified files
134+
1. Scan source, compute checksums
135+
2. Send manifest to Open WebUI `/sync/diff`
136+
3. Delete stale files, create missing directories
137+
4. Upload only new and modified files
245138

246139
## License
247140

0 commit comments

Comments
 (0)