Same speaker in multiple files

[ClayShoaf]

Is there any way to identify the same speaker in a different file? How about adding a label?

For example, let's say I have James who is speaking in audio1.wav and audio2.wav. Pyannote identifies him as SPEAKER_07 in audio1.wav. Is there any way to force pyannote to recocnize him as SPEAKER_07 when I run it on audio2.wav?

[hbredin]

Not out of the box, no.

One straightforward solution would be to concatenate audio1.wav and audio2.wav into a longer audio1+2.wav file -- but that won't scale well with more audio files.

A more involved solution would be to

• run diarization on each file separately
• extract one speaker embedding for each speaker in each file
• do some kind of clustering on top of those embeddings to match speakers across files

[PhantomSpike]

Hey,

I am having the same issue.

@ClayShoaf were you able to find a solution? Did anything work for you?

Do you want to maybe have a quick chat about this on DM or something?

[ClayShoaf]

I was not able to figure it out and I'm working on other stuff now. Sorry

[bghorvath]

Hey,

I am having the same issue.

@ClayShoaf were you able to find a solution? Did anything work for you?

Do you want to maybe have a quick chat about this on DM or something?

You can extract the embeddings using this module: https://huggingface.co/pyannote/embedding

[kevdawg94]

This is an old thread but something of interest to me if anyone has come up with an easy solution. Will explore the suggestion by @bghorvath.

[alunkingusw]

Here is my gist that demonstrates my solution to the problem using diarization to extract the speakers, then average the embeddings and compare with a known speaker embedding to output similarity scores.
https://gist.github.com/alunkingusw/4275a26ba79cefd49f5a2e91d91c4da2

[SrinivasKalagotla]

Just take a 10 seconds clip of James and add it to the audio2 file. Now perform SD on the file and you will get James as Speaker 0.

[ClayShoaf]

This would probably be the easiest solution for my purposes, for now, but it's not exaclty elegant, nor is it a great solution for working with a lot of files.

[alunkingusw]

I am working on this for some research. I'm creating an API to manage transcriptions of groups who have multiple meetings. It will be used in an educational setting for a supervisor to manage the meetings of groups completing a project.

The audio for each meeting is uploaded and I am using whisper and pyannote to transcribe the meetings.
The intention is that I can extract an embedding for each speaker and use it on subsequent meetings that are uploaded. The interface allows names to be applied to the speakers.

Here's my gist for my basic solution
https://gist.github.com/alunkingusw/4275a26ba79cefd49f5a2e91d91c4da2

[arcticloud]

Right — not out of the box, because speaker labels are only consistent within a single diarization run. To carry a speaker's identity across files you have to match on embeddings, not on the labels. Concrete recipe:

from pyannote.audio import Pipeline, Inference, Model
from pyannote.core import Segment
import numpy as np
from sklearn.cluster import AgglomerativeClustering

dia = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=TOKEN)
emb = Inference(Model.from_pretrained("pyannote/embedding", use_auth_token=TOKEN),
                window="whole")

voiceprints, index = [], []          # one averaged embedding per (file, local speaker)
for path in ["audio1.wav", "audio2.wav"]:
    diar = dia(path)
    for spk in diar.labels():
        vecs = [emb.crop(path, seg) for seg in diar.label_timeline(spk)
                if seg.duration > 0.5]
        if vecs:
            voiceprints.append(np.mean(vecs, axis=0))
            index.append((path, spk))

# cluster the voiceprints across ALL files -> stable global IDs
labels = AgglomerativeClustering(n_clusters=None, distance_threshold=0.7,
                                 metric="cosine", linkage="average").fit_predict(voiceprints)
global_id = {index[i]: f"SPEAKER_{labels[i]:02d}" for i in range(len(index))}

Now global_id[("audio2.wav", "SPEAKER_07")] returns the same global ID James got in file 1. Tune distance_threshold on your own data (lower = stricter matching). If you already have a clean reference clip for each known person, embed that once and assign each detected speaker by nearest cosine distance instead of clustering — that's the "known speakers" variant of the same idea.