cozystack · myasnikovdaniil · Aug 12, 2026
@@ -433,7 +433,7 @@ cilium:
       enabled: true
 ```
 
-See [Enabling Hubble](https://docs.cilium.io/en/stable/observability/hubble/) for full configuration details.
+See [Enabling Hubble for network observability](/docs/next/networking/hubble/) for the metrics list this needs, the Grafana dashboards Cozystack ships for it, and troubleshooting. The upstream [Cilium Hubble documentation](https://docs.cilium.io/en/stable/observability/hubble/) covers the remaining knobs.
 
 ## Traffic Flow Summary
 

@@ -0,0 +1,96 @@
+---
+title: "Enabling Hubble for Network Observability"
+linkTitle: "Hubble"
+description: "Turn on Cilium's Hubble observability stack, and read the flow, DNS and L7 metrics through the platform Grafana dashboards Cozystack ships for it."
+weight: 50
+---
+
+Hubble is the network and security observability layer built on top of Cilium. It gives visibility into the communication and behaviour of services in the cluster — flow logs, DNS queries, and L7 request metrics.
+
+Hubble is **disabled by default** in Cozystack to keep resource usage down. This page covers turning it on and reading the results. For where Hubble sits in the data plane, see [Networking architecture](/docs/next/networking/architecture/#observability-with-hubble).
+
+## Prerequisites
+
+- A Cozystack cluster running Cilium as the CNI (the default).
+- The [Monitoring](/docs/next/operations/services/monitoring/) hub deployed, for Grafana access and metric storage.
+
+## Enable Hubble
+
+Enable Hubble, Relay and the UI in the Cilium configuration, and turn on the metrics you want exported:
+
+```yaml
+cilium:
+  hubble:
+    enabled: true
+    relay:
+      enabled: true
+    ui:
+      enabled: true
+    metrics:
+      enabled:
+        - dns
+        - drop
+        - tcp
+        - flow
+        - port-distribution
+        - icmp
+        - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
+```
+
+The `metrics.enabled` list is what makes the dashboards below work — without it Hubble runs but exports nothing for Grafana to draw. The `httpV2` entry in particular must keep its `labelsContext`, because the L7 HTTP dashboard groups by source and destination workload and cannot do so if those labels are absent.
+
+### Components
+
+Enabling Hubble brings up:
+
+- **Hubble Relay** — aggregates flow data from all Cilium agents.
+- **Hubble UI** — web interface for exploring network flows.
+- **Hubble Metrics** — Prometheus metrics for network observability.
+
+## Grafana dashboards
+
+Cozystack ships four Hubble dashboards, delivered in the `hubble` folder of the platform Grafana:
+
+| Dashboard | Description |
+|-----------|-------------|
+| **Overview** | General Hubble metrics including processing statistics |
+| **DNS Namespace** | DNS query and response metrics by namespace |
+| **L7 HTTP Metrics** | HTTP layer 7 metrics by workload |
+| **Network Overview** | Network flow overview by namespace |
+
+These are infrastructure dashboards, so they are provisioned only for the platform-level Monitoring release — the one in `tenant-root` or `cozy-monitoring`. A tenant's own Grafana does not receive them; tenants see their own application dashboards instead.
+
+To reach them, open Grafana through the monitoring hub, browse to the `hubble` folder in the dashboard browser, and pick a dashboard.
+
+## Metrics
+
+Hubble exposes the following, all queryable directly in Grafana:
+
+- `hubble_flows_processed_total` — total number of flows processed
+- `hubble_dns_queries_total` — DNS queries by type
+- `hubble_dns_responses_total` — DNS responses by status
+- `hubble_drop_total` — dropped packets by reason
+- `hubble_tcp_flags_total` — TCP connections by flag
+- `hubble_http_requests_total` — HTTP requests by method and status
+
+## Troubleshooting
+
+Check that Relay and the UI are running:
+
+```bash
+kubectl get pods -n cozy-cilium -l k8s-app=hubble-relay
+kubectl get pods -n cozy-cilium -l k8s-app=hubble-ui
+```
+
+Verify the metrics endpoint is serving:
+
+```bash
+kubectl port-forward -n cozy-cilium svc/hubble-metrics 9965:9965
+curl http://localhost:9965/metrics
+```
+
+Confirm the scrape target exists — if the dashboards are empty but the endpoint above returns data, this is usually the missing link:
+
+```bash
+kubectl get servicemonitor -n cozy-cilium
+```
@@ -0,0 +1,130 @@
+---
+title: "Attaching a Virtual Machine to an External VLAN"
+linkTitle: "VM External VLAN"
+description: "Bridge a virtual machine onto a physically-routed VLAN so it shares a broadcast domain with external hardware, and why macvlan cannot work for this."
+weight: 35
+---
+
+This page describes how to attach a Cozystack virtual machine (the [`vm-instance`](/docs/next/virtualization/vm-instance/) application) directly to an external, physically-routed VLAN — the layer-2 segment a VM needs when it must appear on the same broadcast domain as external hardware (a licensing appliance, a storage box, a gateway managed outside the cluster), with an address from that VLAN's subnet rather than from the cluster overlay.
+
+The default Cozystack VM networking is overlay-only (the pod network, plus optional KubeOVN [VPC subnets](/docs/next/networking/vpc/)). Bridging a VM onto a real VLAN is a different pattern and has one non-obvious constraint: **it works with a Linux bridge and the `bridge` CNI plugin, and it does not work with `macvlan`.** The rest of this guide explains why and gives a working recipe.
+
+## Why `bridge` and not `macvlan`
+
+KubeVirt attaches a VM interface to a secondary network using **bridge binding** by default (this is what the `vm-instance` chart emits for every network in `.spec.networks`). With bridge binding the guest's own MAC address is placed on the wire — the launcher pod does not masquerade or translate it.
+
+A `macvlan` attachment is incompatible with that model. `macvlan` demultiplexes inbound frames strictly by the MAC address of the macvlan child interface. Because KubeVirt puts the *guest's* MAC on the wire — not the macvlan child's — replies from the gateway or other hosts arrive at the parent interface addressed to the guest MAC, do not match any macvlan child, and are silently dropped before they ever reach the VM. The symptom is a guest that can transmit (ARP requests and pings leave, visible in `tcpdump` on the parent interface) but never receives a reply (its neighbor entry for the gateway stays `FAILED`). As a secondary consequence, the host cannot reach macvlan children through the parent interface either, so a host-side service on the parent IP is unreachable from the VMs.
+
+A **Linux bridge** does not have this limitation: it forwards by learned MAC on all bridged ports, so the guest MAC is reachable, and the host can carry an address on the bridge itself to talk to the VMs. Attach the VLAN sub-interface to a bridge and point a `bridge`-type NetworkAttachmentDefinition at it.
+
+## Overview
+
+Three pieces cooperate:
+
+1. A **Linux bridge on each node** that enslaves the tagged VLAN sub-interface. This is node-level networking — it is configured by your node provisioning (netplan / Talos machine config / systemd-networkd), not by a Cozystack chart.
+2. A **`NetworkAttachmentDefinition`** of type `bridge` referencing that bridge, created in the VM's tenant namespace.
+3. The **`vm-instance`** application referencing the NetworkAttachmentDefinition by name in `.networks`, with the guest's static address supplied through cloud-init.
+
+## Prerequisites
+
+- The `multus` package is enabled (it provides the `NetworkAttachmentDefinition` CRD and the secondary-network plumbing).
+- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. The `multus` package puts it there itself on every platform; see [the multus package README](https://github.com/cozystack/cozystack/blob/main/packages/system/multus/README.md) for what it stages and the opt-out, and read it before upgrading a cluster whose `/opt/cni/bin` you provision yourself. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NetworkAttachmentDefinition fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`.
+- There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM.
+
+## 1. Linux bridge on the node
+
+Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach).
+
+This example uses netplan on an Ubuntu/Debian node; the VLAN id and subnet are illustrative (`203.0.113.0/24`, VLAN 100, gateway `203.0.113.1`). Adapt to your uplink naming and to Talos or `systemd-networkd` if that is your provisioning:
+
+```yaml
+network:
+  version: 2
+  vlans:
+    # Tagged VLAN sub-interface, no address of its own — enslaved to the bridge.
+    uplink.100:
+      id: 100
+      link: uplink
+  bridges:
+    br100:
+      interfaces:
+        - uplink.100
+      # Optional host presence on the VLAN. Keep the node's default route on
+      # its management interface — do not add a default route here.
+      addresses:
+        - 203.0.113.2/24
+```
+
+Notes:
+
+- The node's **default route must stay on the management interface.** The bridge address (if any) is only for on-VLAN reachability, not a second default gateway.
+- `netplan apply` cannot move an interface into a bridge while a consumer still holds it (for example a `virt-launcher` pod using a previous `macvlan` attachment). Remove the consumer first (delete the VMI so the launcher releases the interface), then reconfigure.
+- After a reboot, `systemd-networkd` may briefly report the bridge "routable" while the link is not yet actually up. If your VMs need the VLAN immediately at boot, gate their start on a reachability check, or re-run `netplan apply` until the gateway answers.
+
+## 2. NetworkAttachmentDefinition
+
+Create a `bridge`-type NetworkAttachmentDefinition in the tenant namespace that will host the VM. The `vm-instance` chart resolves a network by name **in the VM's own namespace**, so one copy must exist in every tenant namespace that runs VMs on this VLAN.
+
+```yaml
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: vlan100
+  namespace: tenant-example
+spec:
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "type": "bridge",
+      "bridge": "br100",
+      "ipam": {}
+    }
+```
+
+- `bridge` must match the bridge name from step 1 (`br100` here).
+- `ipam: {}` — no cluster-side address assignment; the guest configures its address itself (step 3).
+
+## 3. Attach the VM and assign a static address
+
+Reference the NetworkAttachmentDefinition by name in the `vm-instance` values. Because the chart does not support `networkData`, the static address goes into cloud-init `userData` (`cloudInit`), written by the guest at first boot:
+
+```yaml
+# vm-instance values
+instanceType: u1.medium
+instanceProfile: ubuntu
+disks:
+  - name: example-system
+networks:
+  - name: vlan100
+cloudInit: |
+  #cloud-config
+  write_files:
+    - path: /etc/netplan/60-vlan100.yaml
+      permissions: "0600"
+      content: |
+        network:
+          version: 2
+          ethernets:
+            # Match the second NIC (the pod-network NIC is the first). Use the
+            # interface that comes up without a DHCP lease.
+            enp2s0:
+              addresses:
+                - 203.0.113.10/24
+  runcmd:
+    - netplan apply
+```
+
+The VM ends up with two interfaces: the always-present **pod-network** NIC (`default`, used for cluster-internal traffic and for the `vm-instance` external-access features) and the **VLAN** NIC. The `/24` address above brings up only the connected route for the VLAN subnet — it adds no default route, so the guest's egress stays wherever you want it (typically the pod NIC). If the VLAN is meant to be the guest's default gateway instead, add a default route under the VLAN NIC and remove it from the pod NIC.
+
+## Gotchas
+
+- **VMs are dual-homed.** The `vm-instance` chart always adds the pod-network NIC in addition to any `networks` you declare; there is no single-homed (VLAN-only) option today. Address the VLAN NIC inside the guest and leave the pod NIC to the cluster.
+- **No `networkData`.** The chart wires cloud-init through `userData` only, so in-guest static configuration (netplan `write_files` plus `netplan apply`, as above) is the way to assign the VLAN address.
+- **MAC changes on VM re-creation.** KubeVirt generates a fresh guest MAC each time the VM object is re-created, and `vm-instance` exposes no way to pin it, so re-creating a VM changes its MAC. The upstream gateway then holds a stale ARP entry for the old MAC for a few minutes, so "gateway unreachable" immediately after re-creating a VM is expected — wait for the ARP entry to age out (roughly five minutes) rather than treating it as a fault.
+- **Host-to-VM traffic.** If the host must talk to the VMs (a proxy, a health check), give the bridge a host address on the VLAN (step 1) — traffic through a bare VLAN sub-interface to bridge-attached guests will not work the way `macvlan` users expect.
+
+## See also
+
+- [Attaching GPUs to virtual machines](/docs/next/virtualization/gpu/) — passing NVIDIA GPUs and vGPU profiles into the same VMs.
+- [Networking architecture](/docs/next/networking/architecture/) — how the default overlay data plane is put together.
+- KubeVirt user guide, [Interfaces and Networks](https://kubevirt.io/user-guide/network/interfaces_and_networks/) — bridge binding versus other binding methods.