# Anomaly Detecting Research

A research-oriented anomaly detection pipeline for identifying malicious or harmful model behavior using latent-space analysis and a multi-vector variant of Representation Engineering (RepE).

This project extracts hidden-state representations from a language model, compares them against benign and malicious reference manifolds, and flags prompts whose activations align most strongly with harmful behavior.

## Why this project exists

Large language models can produce harmful outputs even when the model itself is not obviously failing in a conventional way. This project explores a detection strategy based on model internals instead of relying only on explicit textual outputs.

The core idea is:

- benign prompts occupy a relatively stable region in activation space
- malicious prompts may cluster into multiple harmful sub-manifolds
- a detector can score whether a new prompt is anomalous relative to benign behavior

Rather than treating malicious intent as a single global direction, this implementation models it as multiple harmful subspaces using K-means clustering.

## Method overview

The detector follows this flow:

1. Load a benign dataset and a malicious dataset.
2. Pass prompt texts through a pretrained causal LM.
3. Extract a normalized hidden-state vector from a chosen intermediate layer.
4. Save the resulting latent vectors in `data/<model_id>_latent_vectors.npz`.
5. Use a calibration subset to estimate:
   - a benign center
   - multiple malicious centroids via K-means clustering
6. Construct harmful direction vectors from each malicious centroid relative to the benign baseline.
7. Score held-out prompts by their maximum alignment with any harmful direction.
8. Use a threshold derived from benign scores to detect anomalies.

This makes the method a latent-space anomaly detector based on internal model representations rather than a standard text classifier.

## Repository structure

- `download_datasets.py` — downloads and stores the malicious and benign datasets locally.
- `extract_vectors.py` — loads the model, extracts latent vectors for each prompt, and writes the processed feature set.
- `extract_all_layers.py` — runs a forward pass for each prompt and stores normalized hidden-state vectors for every layer, producing a per-layer representation stack.
- `analyze_all_layers.py` — sweeps the hidden layers, computes a layer-wise ROC-AUC using a simple RepE score, and saves a plot of layer performance.
- `early_exit_detection.py` — calibrates a harmful-direction threshold using a chosen layer, then evaluates benign prompts, base malicious attacks, GCG suffixes, and roleplay wrappers against that threshold.
- `multi_vector_repe.py` — runs the multi-vector RepE detector and reports evaluation metrics.
- `prev_detection_scripts/` — earlier baseline implementations, including a single-vector RepE detector and a Mahalanobis-based anomaly detector.
- `lib/common.py` — shared utility functions for model loading, hardware device detection, tokenization, dataset normalization, and error CSV export.
- `lib/evaluation_utils.py` — evaluation status classification utilities.
- `config/config.yml` — configuration specifying the active model and layer indices.
- `data/` — cached datasets, generated latent vectors, and exported CSV results.
- `requirements.txt` — Python dependencies for the pipeline.

## Datasets

The project uses a mixture of benign and malicious sources:

- HarmBench for malicious prompts
- AdvBench for additional harmful behavior examples
- UltraChat for benign conversational data

The processed representation cache is saved to:

- `data/<model_id>_latent_vectors.npz`

## Attribution and licenses

This project uses the following third-party models and datasets. Please review
their original terms before downloading, using, or redistributing them.

### Models

- [Meta Llama 3.2 3B Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct), developed by Meta. It is distributed under the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE) and is subject to Meta's [Acceptable Use Policy](https://www.llama.com/llama3_2/use-policy/).
- [Qwen2.5 3B Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct), developed by the Qwen Team. The model card lists the `qwen-research` license and provides the [Qwen2.5 citation](https://qwenlm.github.io/blog/qwen2.5/).
- [Google Gemma 2 2B](https://huggingface.co/google/gemma-2-2b), developed by Google. It is distributed under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms).

### Datasets

- [HarmBench](https://huggingface.co/datasets/walledai/HarmBench), released by Walled AI and the HarmBench authors under the MIT license. Please cite [Mazeika et al., 2024](https://arxiv.org/abs/2402.04249).
- [AdvBench](https://huggingface.co/datasets/walledai/AdvBench), distributed under the MIT license. This repository downloads its `harmful_behaviors.csv` from the [LLM Attacks repository](https://github.com/llm-attacks/llm-attacks/blob/main/data/advbench/harmful_behaviors.csv). Please cite [Zou et al., 2023](https://arxiv.org/abs/2307.15043).
- [UltraChat 200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k), curated by Hugging Face H4. The dataset card requests citation of [Ding et al., 2023](https://arxiv.org/abs/2305.14233).

The model and dataset licenses are separate from this repository's license and
may include additional access or attribution requirements.

## Setup

Create a Python environment and install dependencies:

```bash
pip install -r requirements.txt
```

## Running the pipeline

### 1) Download datasets

```bash
python download_datasets.py
```

### 2) Extract latent vectors

```bash
python extract_vectors.py
```

This script loads the model, extracts one hidden-state vector per prompt for the selected layer, and saves them to the data directory.

### 3) Run the anomaly detector

```bash
python multi_vector_repe.py
```

This script:

- loads cached latent vectors
- splits data into calibration and test sets
- clusters malicious calibration samples into K malicious sub-manifolds
- evaluates the detector on held-out prompts
- prints ROC-AUC and attack detection statistics
- exports misclassified prompts (false alarms and missed attacks) to a timestamped CSV in `data/<model_id>_multi_vector_repe_results_<timestamp>.csv` for error analysis

## Expected outputs

The detector reports summary metrics to the console:

- ROC-AUC score
- decision threshold based on the benign distribution (e.g., 99th percentile)
- benign pass rate
- false alarms (`False Positive`)
- attacks detected (`True Positive`)
- attacks missed (`False Negative`)

The current implementation uses a calibration set of 100 benign and 100 malicious samples, with the remainder reserved for evaluation.

### Error analysis CSV export

At the end of evaluation, `multi_vector_repe.py`, `early_exit_detection.py`, and baseline detectors automatically filter and export misclassified prompts to a CSV file in `data/`, prefixed by the active model ID (e.g. `data/<model_id>_<script_results>_<timestamp>.csv`):

- **Only misclassifications are saved**: filters specifically for `False Alarm (False Positive)` (benign prompts incorrectly flagged as malicious) and `Missed Attack (False Negative)` (malicious prompts that evaded detection). Correctly classified prompts are omitted to keep error triage focused.
- **Sorted by score**: rows are sorted descending by detector score (`RepE_Harm_Score` or anomaly score) so the highest-confidence misclassifications appear first.
- **Columns included**: `Ground_Truth`, `Test_Set`, `RepE_Harm_Score`, `Prediction`, `Prompt_Text`, and `Result_Status`.

### 4) Optional: extract all-layer latent states

```bash
python extract_all_layers.py
```

This script performs the same model pass but keeps the normalized hidden state from every transformer layer, saving a 3D array of shape `(num_prompts, num_layers, hidden_dim)` for both benign and malicious prompts to `data/<model_id>_all_layers.npz`.

This is useful when you want to inspect which layer is most informative before committing to a single chosen layer for the full detector.

### 4a) Optional: analyze layer-by-layer detection quality

```bash
python analyze_all_layers.py
```

This script loads the all-layer cache, computes a simple RepE score at each layer, and plots ROC-AUC as a function of layer depth. It saves a chart to `data/<model_id>_layer_trajectory.png` and helps identify the strongest hidden layer for downstream analysis.

### 4b) Optional: run early-exit detection tests

```bash
python early_exit_detection.py
```

This script calibrates a harmful-direction threshold from the selected layer, then evaluates the early-exit detector across benign prompts (UltraChat), base malicious attacks (HarmBench), and adversarial prompt variants such as GCG-style suffix attacks and roleplay/developer-mode jailbreak wrappers. It is useful for checking whether a discovered layer remains robust under stronger attack patterns beyond the normal evaluation split.

Like the main detector, it automatically exports misclassified prompts (false alarms and missed attacks across base attacks, GCG suffixes, and roleplay wrappers) to `data/<model_id>_early_exit_detection_results_<timestamp>.csv`.


## Model configuration

Models and their designated evaluation layers (`layer_idx` and `auc_layer`) are managed via `config/config.yml`. Preconfigured models include:

- `google/gemma-2-2b` (`layer_idx: 14`, `auc_layer: 20`)
- `meta-llama/Llama-3.2-3B-Instruct` (`layer_idx: 14`, `auc_layer: 12`)
- `Qwen/Qwen2.5-3B-Instruct` (`layer_idx: 18`, `auc_layer: 21`)

To switch the active model, set `model.name` under `model:` in `config/config.yml`.

## Notes

- This is a research-focused pipeline for model safety and anomaly detection.
- Inference can be computationally expensive, so a GPU is recommended.
- The layer index used for vector extraction is configurable in `config/config.yml`.
- The project is intended for experimentation and analysis of malicious latent structure rather than deployment in production systems.

## Related ideas

This work is inspired by the broader literature on:

- representation engineering (RepE)
- latent-space anomaly detection
- activation steering and safety probing
- multi-manifold modeling of harmful behavior

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for the full text.

The MIT license applies to the code in this repository. Third-party models and datasets used by this project remain subject to their own individual licenses and terms.

## Quick summary

This repo implements several latent-space detection methods for identifying harmful behavior, including a multi-vector RepE detector, layer-wise anomaly sweeps, and adversarial stress tests that probe whether a chosen harmful direction remains effective against stronger prompt attacks.