Training new models

This page describes how to train vectors with the models that are currently implemented in VSMlib.


Word2vec is arguably the most popular word embedding model. We provide implementation of extended word2vec model, which can be trained on linear and dependency-based contexts, with bound and unbound context representations.

Additionally we provide an implementation which considers characters rather than words to be the minimal units. This enables it to take advantage of morphological information: as far as a word-level models such as word2vec is concerned, “walk” and “walking” are completely unrelated, except through similarities in their distributions.

To train word2vec embeddings vsmlib can be envoked via the command line interface:

>>> python3 -m vsmlib.embeddings.train_word2vec

The command line parameters are as

--dimensions size of embeddings
--context_type context type [linear’ or ‘deps’], for deps context, the annotated corpus is required
 context representation [‘bound’ or ‘unbound’]
--window window size’)
--model base model type [‘skipgram’ or ‘cbow’]
 number of negative samples
--out_type output model type [“hsm”: hierarchical softmax, “ns”: negative sampling, “original”: no approximation]
--subword specify if subword-level approach should be used [“none”, “rnn”]
--batchsize learning minibatch size
--gpu GPU ID (negative value indicates CPU)
--epochs number of epochs to learn
 max word length (only used for char-level subword)
--path_vocab path to the vocabulary
--path_corpus path to the corpus
--path_out path to save embeddings
--test run in test mode
--verbose verbose mode

Alternatively, word2vec training can be done though vsmlib python API.

>>> vsmlib.embeddings.train_word2vec.train(args)

The arguments are argparse.namespace identical to command line arguments. Instance of ModelDense is returned.

Realted papers: original w2v, Bofang, Mnih, subword.

 title = {Efficient Estimation of Word Representations in Vector Space},
 urldate = {2015-12-03},
 booktitle = {Proceedings of International Conference on Learning Representations (ICLR)},
 author = {Mikolov, Tomas and Chen, Kai and Corrado, Greg and Dean, Jeffrey},
 year = {2013}}
 title={Investigating Different Syntactic Context Types and Context Representations for Learning Word Embeddings},
 author={Bofang Li and Tao Liu and Zhe Zhao and Buzhou Tang and Aleksandr Drozd and Anna Rogers and Xiaoyong Du},