qflow: a fast customer-oriented NetFlow database for accounting and data retention
Internet service providers in Iceland must manage large databases of network flow data in order to charge customers and comply with data retention laws. The databases need to efficiently handle large volumes of data, often billions or trillions of records, and they must support fast queries of traff...
Main Author: | |
---|---|
Other Authors: | |
Format: | Thesis |
Language: | English |
Published: |
2014
|
Subjects: | |
Online Access: | http://hdl.handle.net/1946/19868 |
id |
ftskemman:oai:skemman.is:1946/19868 |
---|---|
record_format |
openpolar |
spelling |
ftskemman:oai:skemman.is:1946/19868 2023-05-15T16:49:53+02:00 qflow: a fast customer-oriented NetFlow database for accounting and data retention Hallgrímur H. Gunnarsson 1983- Háskóli Íslands 2014-09 application/pdf http://hdl.handle.net/1946/19868 en eng http://hdl.handle.net/1946/19868 Tölvunarfræði Gagnasafnskerfi Gagnagrunnar Thesis Master's 2014 ftskemman 2022-12-11T06:51:56Z Internet service providers in Iceland must manage large databases of network flow data in order to charge customers and comply with data retention laws. The databases need to efficiently handle large volumes of data, often billions or trillions of records, and they must support fast queries of traffic volume per customer over time and extraction of raw flow data for given customers. Popular open-source tools for storing flow data, such as nfdump and flow-tools, are backed by flat binary files. They do not provide any type of indexing or summaries of customer traffic. As a result, flow queries for a given customer need to linearly scan through all the flow records in a given time period. We present a high-performance customer-oriented flow database that provides fast customer queries and compressed flow storage. The database is backed by indexed flow tablets that allow for fast extraction of customer flows and traffic volume per customer. Thesis Iceland Skemman (Iceland) |
institution |
Open Polar |
collection |
Skemman (Iceland) |
op_collection_id |
ftskemman |
language |
English |
topic |
Tölvunarfræði Gagnasafnskerfi Gagnagrunnar |
spellingShingle |
Tölvunarfræði Gagnasafnskerfi Gagnagrunnar Hallgrímur H. Gunnarsson 1983- qflow: a fast customer-oriented NetFlow database for accounting and data retention |
topic_facet |
Tölvunarfræði Gagnasafnskerfi Gagnagrunnar |
description |
Internet service providers in Iceland must manage large databases of network flow data in order to charge customers and comply with data retention laws. The databases need to efficiently handle large volumes of data, often billions or trillions of records, and they must support fast queries of traffic volume per customer over time and extraction of raw flow data for given customers. Popular open-source tools for storing flow data, such as nfdump and flow-tools, are backed by flat binary files. They do not provide any type of indexing or summaries of customer traffic. As a result, flow queries for a given customer need to linearly scan through all the flow records in a given time period. We present a high-performance customer-oriented flow database that provides fast customer queries and compressed flow storage. The database is backed by indexed flow tablets that allow for fast extraction of customer flows and traffic volume per customer. |
author2 |
Háskóli Íslands |
format |
Thesis |
author |
Hallgrímur H. Gunnarsson 1983- |
author_facet |
Hallgrímur H. Gunnarsson 1983- |
author_sort |
Hallgrímur H. Gunnarsson 1983- |
title |
qflow: a fast customer-oriented NetFlow database for accounting and data retention |
title_short |
qflow: a fast customer-oriented NetFlow database for accounting and data retention |
title_full |
qflow: a fast customer-oriented NetFlow database for accounting and data retention |
title_fullStr |
qflow: a fast customer-oriented NetFlow database for accounting and data retention |
title_full_unstemmed |
qflow: a fast customer-oriented NetFlow database for accounting and data retention |
title_sort |
qflow: a fast customer-oriented netflow database for accounting and data retention |
publishDate |
2014 |
url |
http://hdl.handle.net/1946/19868 |
genre |
Iceland |
genre_facet |
Iceland |
op_relation |
http://hdl.handle.net/1946/19868 |
_version_ |
1766040059870445568 |