Software Framework for Topic Modelling with Large Corpora
Large corpora are ubiquitous in today’s world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). In this paper, we identify a gap in existing implementations of many of the popular algorithms, which is their scalability and ease of use. We descr...
Main Author: | |
---|---|
Other Authors: | |
Format: | Text |
Language: | English |
Published: |
2010
|
Subjects: | |
Online Access: | http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.695.4595 http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf |
id |
ftciteseerx:oai:CiteSeerX.psu:10.1.1.695.4595 |
---|---|
record_format |
openpolar |
spelling |
ftciteseerx:oai:CiteSeerX.psu:10.1.1.695.4595 2023-05-15T16:01:52+02:00 Software Framework for Topic Modelling with Large Corpora Petr Sojka The Pennsylvania State University CiteSeerX Archives 2010 application/pdf http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.695.4595 http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf en eng http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.695.4595 http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf Metadata may be used without restrictions as long as the oai identifier remains attached to it. http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf text 2010 ftciteseerx 2016-01-08T18:36:46Z Large corpora are ubiquitous in today’s world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). In this paper, we identify a gap in existing implementations of many of the popular algorithms, which is their scalability and ease of use. We describe a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion. Within this framework, we implement several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation, in a way that makes them completely independent of the training corpus size. Particular emphasis is placed on straightforward and intuitive framework design, so that modifications and extensions of the methods and/or their application by interested practitioners are effortless. We demonstrate the usefulness of our approach on a real-world scenario of computing document similarities within an existing digital library DML-CZ. 1. Text DML Unknown |
institution |
Open Polar |
collection |
Unknown |
op_collection_id |
ftciteseerx |
language |
English |
description |
Large corpora are ubiquitous in today’s world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). In this paper, we identify a gap in existing implementations of many of the popular algorithms, which is their scalability and ease of use. We describe a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion. Within this framework, we implement several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation, in a way that makes them completely independent of the training corpus size. Particular emphasis is placed on straightforward and intuitive framework design, so that modifications and extensions of the methods and/or their application by interested practitioners are effortless. We demonstrate the usefulness of our approach on a real-world scenario of computing document similarities within an existing digital library DML-CZ. 1. |
author2 |
The Pennsylvania State University CiteSeerX Archives |
format |
Text |
author |
Petr Sojka |
spellingShingle |
Petr Sojka Software Framework for Topic Modelling with Large Corpora |
author_facet |
Petr Sojka |
author_sort |
Petr Sojka |
title |
Software Framework for Topic Modelling with Large Corpora |
title_short |
Software Framework for Topic Modelling with Large Corpora |
title_full |
Software Framework for Topic Modelling with Large Corpora |
title_fullStr |
Software Framework for Topic Modelling with Large Corpora |
title_full_unstemmed |
Software Framework for Topic Modelling with Large Corpora |
title_sort |
software framework for topic modelling with large corpora |
publishDate |
2010 |
url |
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.695.4595 http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf |
genre |
DML |
genre_facet |
DML |
op_source |
http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf |
op_relation |
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.695.4595 http://www.fi.muni.cz/usr/sojka/papers/lrec2010-rehurek-sojka.pdf |
op_rights |
Metadata may be used without restrictions as long as the oai identifier remains attached to it. |
_version_ |
1766397566390370304 |