How real are observed trends in small correlated datasets?
The eye may perceive a significant trend in plotted time-series data, but if the model errors of nearby data points are correlated, the trend may be an illusion. We examine generalized least-squares (GLS) estimation, finding that error correlation may be underestimated in highly correlated small dat...
Published in: | Royal Society Open Science |
---|---|
Main Authors: | , , |
Format: | Article in Journal/Newspaper |
Language: | English |
Published: |
The Royal Society
2019
|
Subjects: | |
Online Access: | https://doi.org/10.1098/rsos.181089 https://doaj.org/article/ebc6c140849349f69bc6fd5eaeefb5c0 |
id |
ftdoajarticles:oai:doaj.org/article:ebc6c140849349f69bc6fd5eaeefb5c0 |
---|---|
record_format |
openpolar |
spelling |
ftdoajarticles:oai:doaj.org/article:ebc6c140849349f69bc6fd5eaeefb5c0 2023-05-15T13:58:11+02:00 How real are observed trends in small correlated datasets? S. J. Salamon H. J. Hansen D. Abbott 2019-03-01T00:00:00Z https://doi.org/10.1098/rsos.181089 https://doaj.org/article/ebc6c140849349f69bc6fd5eaeefb5c0 EN eng The Royal Society https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.181089 https://doaj.org/toc/2054-5703 2054-5703 doi:10.1098/rsos.181089 https://doaj.org/article/ebc6c140849349f69bc6fd5eaeefb5c0 Royal Society Open Science, Vol 6, Iss 3 (2019) regression analysis autoregressive processes prediction methods supervised learning trend estimation statistical significance Science Q article 2019 ftdoajarticles https://doi.org/10.1098/rsos.181089 2022-12-31T14:23:44Z The eye may perceive a significant trend in plotted time-series data, but if the model errors of nearby data points are correlated, the trend may be an illusion. We examine generalized least-squares (GLS) estimation, finding that error correlation may be underestimated in highly correlated small datasets by conventional techniques. This risks indicating a significant trend when there is none. A new correlation estimate based on the Durbin–Watson statistic is developed, leading to an improved estimate of autoregression with highly correlated data, thus reducing this risk. These techniques are generalized to randomly located data points in space, through the new concept of the nearest new neighbour path. We describe tests on the validity of the GLS schemes, allowing verification of the models employed. Examples illustrating our method include a 40-year record of atmospheric carbon dioxide, and Antarctic ice core data. While more conservative than existing techniques, our new GLS estimate finds a statistically significant increase in background carbon dioxide concentration, with an accelerating trend. We conclude with an example of a worldwide empirical climate model for radio propagation studies, to illustrate dealing with spatial correlation in unevenly distributed data points over the surface of the Earth. The method is generally applicable, not only to climate-related data, but to many other kinds of problems (e.g. biological, medical and geological data), where there are unequally (or randomly) spaced observations in temporally or spatially distributed datasets. Article in Journal/Newspaper Antarc* Antarctic ice core Directory of Open Access Journals: DOAJ Articles Antarctic Royal Society Open Science 6 3 181089 |
institution |
Open Polar |
collection |
Directory of Open Access Journals: DOAJ Articles |
op_collection_id |
ftdoajarticles |
language |
English |
topic |
regression analysis autoregressive processes prediction methods supervised learning trend estimation statistical significance Science Q |
spellingShingle |
regression analysis autoregressive processes prediction methods supervised learning trend estimation statistical significance Science Q S. J. Salamon H. J. Hansen D. Abbott How real are observed trends in small correlated datasets? |
topic_facet |
regression analysis autoregressive processes prediction methods supervised learning trend estimation statistical significance Science Q |
description |
The eye may perceive a significant trend in plotted time-series data, but if the model errors of nearby data points are correlated, the trend may be an illusion. We examine generalized least-squares (GLS) estimation, finding that error correlation may be underestimated in highly correlated small datasets by conventional techniques. This risks indicating a significant trend when there is none. A new correlation estimate based on the Durbin–Watson statistic is developed, leading to an improved estimate of autoregression with highly correlated data, thus reducing this risk. These techniques are generalized to randomly located data points in space, through the new concept of the nearest new neighbour path. We describe tests on the validity of the GLS schemes, allowing verification of the models employed. Examples illustrating our method include a 40-year record of atmospheric carbon dioxide, and Antarctic ice core data. While more conservative than existing techniques, our new GLS estimate finds a statistically significant increase in background carbon dioxide concentration, with an accelerating trend. We conclude with an example of a worldwide empirical climate model for radio propagation studies, to illustrate dealing with spatial correlation in unevenly distributed data points over the surface of the Earth. The method is generally applicable, not only to climate-related data, but to many other kinds of problems (e.g. biological, medical and geological data), where there are unequally (or randomly) spaced observations in temporally or spatially distributed datasets. |
format |
Article in Journal/Newspaper |
author |
S. J. Salamon H. J. Hansen D. Abbott |
author_facet |
S. J. Salamon H. J. Hansen D. Abbott |
author_sort |
S. J. Salamon |
title |
How real are observed trends in small correlated datasets? |
title_short |
How real are observed trends in small correlated datasets? |
title_full |
How real are observed trends in small correlated datasets? |
title_fullStr |
How real are observed trends in small correlated datasets? |
title_full_unstemmed |
How real are observed trends in small correlated datasets? |
title_sort |
how real are observed trends in small correlated datasets? |
publisher |
The Royal Society |
publishDate |
2019 |
url |
https://doi.org/10.1098/rsos.181089 https://doaj.org/article/ebc6c140849349f69bc6fd5eaeefb5c0 |
geographic |
Antarctic |
geographic_facet |
Antarctic |
genre |
Antarc* Antarctic ice core |
genre_facet |
Antarc* Antarctic ice core |
op_source |
Royal Society Open Science, Vol 6, Iss 3 (2019) |
op_relation |
https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.181089 https://doaj.org/toc/2054-5703 2054-5703 doi:10.1098/rsos.181089 https://doaj.org/article/ebc6c140849349f69bc6fd5eaeefb5c0 |
op_doi |
https://doi.org/10.1098/rsos.181089 |
container_title |
Royal Society Open Science |
container_volume |
6 |
container_issue |
3 |
container_start_page |
181089 |
_version_ |
1766266291257081856 |