Blind moving speaker extraction using X-vectors


Abstract: We propose a novel approach for semi-supervised extraction of a moving audio source of interest (SOI) applicable in reverberant and noisy environments. The blind part of the method is based on independent vector extraction (IVE) and uses the recently proposed constant separating vector (CSV) mixing model. This model allows for changes of mixing parameters within the processed interval of the mixture, which potentially leads to higher accuracy of SOI estimation. The supervised part of the method concerns a pilot signal which is related to the SOI and ensures the convergence of the blind method towards the SOI. The pilot is based on robust detection of frames where SOI is dominant via speaker embeddings called X-vectors. Robustness of the detection is achieved through augmentation of the data for the supervised training of the X-vectors. The pilot-supported extraction yields significantly better performance compared to its unsupervised counterpart identifying SOI solely using the initialization.

Examples in different acoustic environments:

The listener should hear that the extracted speaker is sometimes vanishing, depending on the performance of the given method. This is mainly caused by the speaker’s movements and by the limited ability of blind source extraction algorithms to keep convergence to the desired speaker, in particular, in the online block-by-block processing regime (which is essential for practical deployment). The CSV mixing model brings new flexibility to this approach compared to the conventional block-by-block method where standard static (time-invariant) mixing model is assumed on the blocks.

T60 = 100 ms, all methods with block length = 200 ms

original stereo mixture
conventional block-by-block unsupervised blind extraction
conventional block-by-block blind extraction piloted by oracle pilot
conventional block-by-block blind extraction piloted by X-vectors
proposed CSV-based unsupervised blind extraction

proposed CSV-based blind extraction piloted by oracle pilot
proposed CSV-based blind extraction piloted by X-vectors
proposed CSV-based blind extraction piloted by X-vectors when the block length is 800 ms; you can hear that the speaker is sometimes vanishing when getting from the focus of the algorithm – this is caused by the length of block, which is too long to capture the fast movements of the speaker

T60 = 300 ms, all methods with block length = 200 ms

original stereo mixture
conventional block-by-block unsupervised blind extraction
conventional block-by-block blind extraction piloted by oracle pilot
conventional block-by-block blind extraction piloted by X-vectors
proposed CSV-based unsupervised blind extraction

proposed CSV-based blind extraction piloted by oracle pilot
proposed CSV-based blind extraction piloted by X-vectors

T60 = 600 ms, all methods with block length = 200 ms

original stereo mixture
conventional block-by-block unsupervised blind extraction
conventional block-by-block blind extraction piloted by oracle pilot
conventional block-by-block blind extraction piloted by X-vectors
proposed CSV-based unsupervised blind extraction

proposed CSV-based blind extraction piloted by oracle pilot
proposed CSV-based blind extraction piloted by X-vectors