Online citations, reference lists, and bibliographies.
← Back to Search

Smooth Estimates Of Multiple Quantiles In Dynamically Varying Data Streams

H. Hammer, A. Yazidi
Published 2019 · Mathematics, Computer Science

Cite This
Download PDF
Analyze on Scholarcy
In this paper, we investigate the problem of estimating multiple quantiles when samples are received online (data stream). We assume a dynamical system, i.e., the distribution of the samples from the data stream changes with time. A major challenge of using incremental quantile estimators to track multiple quantiles is that we are not guaranteed that the monotone property of quantiles will be satisfied, i.e, an estimate of a lower quantile might erroneously overpass that of a higher quantile estimate. Surprisingly, we have only found two papers in the literature that attempt to counter these challenges, namely the works of Cao et al. (Proceedings of the first ACM workshop on mobile internet through cellular networks, ACM, 2009 ) and Hammer and Yazidi (Proceedings of the 30th international conference on industrial engineering and other applications of applied intelligent systems (IEA/AIE), France, Springer, 2017 ) where the latter is a preliminary version of the work in this paper. Furthermore, the state-of-the-art incremental quantile estimator called deterministic update-based multiplicative incremental quantile estimator (DUMIQE), due to Yazidi and Hammer (IEEE Trans Cybernet, 2017 ), fails to guarantee the monotone property when estimating multiple quantiles. A challenge with the solutions, in Cao et al. ( 2009 ) and Hammer and Yazidi ( 2017 ), is that even though the estimates satisfy the monotone property of quantiles, the estimates can be highly irregular relative to each other which usually is unrealistic from a practical point of view. In this paper, we suggest to generate the quantile estimates by inserting the quantile probabilities (e.g., $$0.1, 0.2, \ldots , 0.9$$ 0.1 , 0.2 , … , 0.9 ) into a monotonically increasing and infinitely smooth function (can be differentiated infinitely many times). The function is incrementally updated from the data stream. The monotonicity and smoothness of the function ensure that both the monotone property and regularity requirement of the quantile estimates are satisfied. The experimental results show that the method performs very well and estimates multiple quantiles more precisely than the original DUMIQE (Yazidi and Hammer 2017 ), and the approaches reported in Hammer and Yazidi ( 2017 ) and Cao et al. ( 2009 ).
This paper references
A Space-Efficient Recursive Procedure for Estimating a Quantile of an Unknown Distribution
L. Tierney (1983)
Frugal Streaming for Estimating Quantiles: One (or two) memory suffices
Qiang Ma (2014)
Data skeletons: simultaneous estimation of multiple quantiles for massive streaming datasets with applications to density estimation
J. McDermott (2007)
Quantile estimation from grouped data: the cell midpoint
B. W. Schmeiser (1977)
Medians and beyond: new aggregation techniques for sensor networks
Nisheeth Shrivastava (2004)
Incremental quantile estimation for massive tracking
F. Chen (2000)
The P2 algorithm for dynamic calculation of quantiles and histograms without storing observations
R. Jain (1985)
Two Maximum Entropy-Based Algorithms for Running Quantile Estimation in Nonstationary Data Streams
Ognjen Arandjelovic (2015)
Tracking Quantiles of Network Data Streams with Dynamic Operations
J. Cao (2010)
Multiplicative Update Methods for Incremental Quantile Estimation
A. Yazidi (2019)
Incremental tracking of multiple quantiles for network monitoring in cellular networks
J. Cao (2009)
Incremental quantile estimation
Katharina Tschumitschew (2010)
Incremental Quantiles Estimators for Tracking Multiple Quantiles
H. Hammer (2017)
Monitoring Networked Applications With Incremental Quantile Estimation
J. M. Chambers (2006)
Quantiles over data streams: experimental comparisons, new analyses, and further improvements
Ge Luo (2016)

This paper is referenced by
Semantic Scholar Logo Some data provided by SemanticScholar