SVNN

This repository contains the code to replicate the experiments in the paper Sparse Covariance Neural Networks (link).

Description

Sparse coVariance Neural Networks (SVNNs) are graph convolutional networks that operate on sparse estimates of the covariance matrix. Using sparse estimates improves their robustness to spurious correlations and their computational efficiency, resulting in better downstream performance and faster execution times.

Requirements

• Python 3.10.18
• pip install -r requirements.txt

Datasets

The datasets are automatically preprocessed from the raw files the first time they are used (see data/data_loading.py for the details on the preprocessing steps). The following files should be downloaded and placed at the path data/datasets:

ADNI

From the website https://ida.loni.usc.edu/, download the files ADSP_PHC_T1_FS_DATADIC and ADSP_PHC_T1_FS and place them in the folder data/datasets/adni

MHEALTH

From the website https://archive.ics.uci.edu/dataset/319/mhealth+dataset, download the folder MHEALTHDATASET and place it at the path data/datasets/MHEALTHDATASET

Realdisp

From the website https://archive.ics.uci.edu/dataset/305/realdisp+activity+recognition+dataset, download the folder realdisp+activity+recognition+dataset and place it at the path data/datasets/realdisp+activity+recognition+dataset

Experiments

The experiments in the paper can be replicated by using the files train.py (for S-VNNs and VNNs), train_neigh_sampling.py (for NeighSampl+GCN) and train_pca.py (for PCA and MLP baselines). The hyperparameters can be specified via command line. The file run_exp.py contains examples of commands for the various experiments. Below are the main arguments to be specified for an experiment.

Generic

• dset: selects the dataset. Can be adni1, adni2, mhealth, realdisp.

S-VNNs and VNNs hyperparameters

• cov_type: selects the covariance sparsification. Can be standard for VNNs (no sparsification) or hard_thr, soft_thr, ACV, RCV for S-VNNs.
• p: parameter for RCV S-VNNs. If used with hard or soft thresholding, the threshold is set to achieve the desired sparsification level.
• hidden_sizes: comma-separated list with sizes of VNN hidden layers (e.g., 32,32 for 2 layers of hidden size 32).
• hidden_mlp_sizes: comma-separated list with sizes of readout MLP hidden layers (e.g., 32,32 for 2 layers of hidden size 32). The first dimension should correspond to the last dimension of hidden_sizes.
• K: order of S-VNNs filters
• use_batch_norm: whether to implement batch normalization (True or False)
• use_layer_norm: whether to implement layer normalization (True or False)
• trace_norm: whether to normalize the covariance by its trace for numerical stability (True or False)
• sparse_tensor: whether to implement the covariance as a sparse tensor (True or False)

Training parameters

• nEpochs: number of training epochs
• patience: number of epochs after which training is stopped if there is no improvement on validation set
• batchSize: batch size
• lr: learning rate
• dropout: dropout rate (0.0 for no dropout)
• iterations: number of iterations per experiment

Baselines

• pca_type: for MLP and PCA experiments: None for MLP on raw data, standard, sparse, kernel for PCA variants
• n_components: number of PCA components