LanguageModelingDatasets.jl
API
LanguageModelingDatasets.read_tokens — Functionread_tokens(corpus)todo
LanguageModelingDatasets.test_tokens — Functiontest_tokens(corpus)todo
LanguageModelingDatasets.dev_tokens — Functiondev_tokens(corpus)todo
Missing docstring for test_tokens. Check Documenter's build log for details.
LanguageModelingDatasets.test_sentences — Functiontest_sentences(corpus)todo
LanguageModelingDatasets.dev_sentences — Functiondev_sentences(corpus)todo
Missing docstring for test_sentences. Check Documenter's build log for details.
Datasets
Billion Word Benchmark
Missing docstring for BillionWordBenchmark. Check Documenter's build log for details.
WikiText
LanguageModelingDatasets.WikiText2 — TypeWikiText2v1WikiText2v1 corpus for word-level language modeling. See https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/.
LanguageModelingDatasets.WikiText2Raw — TypeWikiText2RawV1WikiText2RawV1 corpus for character-level language modeling. See https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/.
LanguageModelingDatasets.WikiText103 — TypeWikiText103v1WikiText103v1 corpus for word-level language modeling. See https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/.
LanguageModelingDatasets.WikiText103Raw — TypeWikiText103RawV1WikiText103RawV1 corpus for character-level language modeling. See https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/.
enwiki8
Missing docstring for enwiki8. Check Documenter's build log for details.