N-Grams

NGrams.ngrams — Function.

ngrams(tokens, n; add_bos=true, bos=BOS, add_eos=true, eos=EOS)

Iterate over a sequence of n-grams (of length n) from tokens.

NGrams.unigrams — Function.

unigrams(tokens)

Return a sequence of unigrams from tokens.

NGrams.bigrams — Function.

bigrams(xs; bos="*BOS*", eos="*EOS*")

Iterate over a sequence of bigrams from xs.

NGrams.trigrams — Function.

trigrams(xs; bos="*BOS*", eos="*EOS*")

Iterate over a sequence of trigrams from xs.