Structural Breaks

All functions · nseries package

DickeyFuller

func (s Series) DickeyFuller(n int) Series

DickeyFuller returns the rolling Dickey-Fuller t-statistic of the lagged level in a regression of each change on a constant and the previous value over the last n lag pairs.

At bar t the n lag pairs are the regression rows i = t-n+1 .. t. Row i regresses the change s[i] - s[i-1] on a constant and the lagged level s[i-1], so a window of n pairs reads the n + 1 values s[t-n .. t]. With b the coefficient of the lagged level, Sxx the centred sum of squares of the lagged level and SSR the residual sum of squares, the statistic is t = b sqrt((n-2) Sxx/SSR), the multiplicative form of b/se with se = sqrt(SSR/(n-2)/Sxx). The statistic has n - 2 degrees of freedom: n rows less two coefficients. The coefficient b is OUTheta(n), bit for bit, and b + 1 is the slope of the level on its lag, so the statistic is also the t-statistic of that slope tested against 1. Its sign is the sign of OUTheta(n) wherever neither is 0.

DickeyFuller is built on the shared co-moment kernel behind OUTheta and SScore, with the residuals formed explicitly rather than by subtracting co-moments, which cancels near a perfect fit. The source is Dickey and Fuller (1979), Journal of the American Statistical Association 74, pages 427 to 431. The value at bar t is, in exact arithmetic, the test statistic of statsmodels adfuller(window, maxlag=0, regression=‘c’, autolag=None) on the n + 1 values s[t-n .. t]; the two agree to rounding on ordinary windows, while statsmodels, which fits the uncentred design, loses digits where the level is large against the moves (a level of 1e9 with unit moves) and this method does not. There are no lagged differences and no trend: the augmented and trend forms are not built. AFML picks the order d of FracDiff as the smallest d whose output passes its ADF test, which has one lagged difference, so this method does not reproduce that choice.

Pass log prices, c.Log().DickeyFuller(n). On raw prices a rejection of the unit root would imply stationary prices of finite variance, so return volatility would have to shrink as prices rise, while on log prices a rescaling of the price only shifts the series, which the constant absorbs (AFML, section 17.4.2.1, on raw and log prices). The method takes no logarithm and uses the receiver as given. Levels are legitimate input too (a spread, a yield): there is then no dependence on a logarithm and no NaN for non-positive values.

The warm-up reads 0, not NaN: bars 0 to n-1 are 0 and the first valid index is n. Because 0 is also a legitimate value, mask the warm-up where it matters with DickeyFuller(n).SetN(n, math.NaN()). An n below 3 (three pairs leave one residual degree of freedom) or of at least len(s) gives len(s) zeros, and the minimum length of a series with a valid bar is n + 1.

A NaN or an infinity at bar p gives NaN from the later of p and n to the earlier of p+n and len(s)-1, exactly the windows s[t-n .. t] that hold it; later bars recover. The method forms no difference of input values, so a finite pair whose difference would overflow is not a NaN case. The kernel scales each window by a power of two first, so there is no limit on the magnitudes, upper or lower, but there is one on the dynamic range of a window s[t-n .. t], the ratio of the largest to the smallest nonzero difference between two of its values: finite inputs give finite, accurate outputs while that ratio stays below about 2^1000 (about 1e301). Beyond it the slope, rebuilt from the kernel’s scaled units, can overflow while the power of two that scales the root underflows, with no special rule, so the result can be NaN or an infinity: the series 0, 2^-700, 0, 2^400 gives NaN from DickeyFuller(3) at bar 3, where the statistic is about -0.577.

A window whose residual sum of squares is at most 1e-24 times the centred sum of squares of the levels it regresses (a perfect fit, or flat levels) gives 0, and so does a flat lagged level. Just outside that guard the value can be huge, far beyond any critical value, which marks a nearly deterministic window rather than a signal: a path converging geometrically to a level is a perfect fit up to the rounding of the stored values. While the largest move of a window exceeds about 1e6 times that rounding, the statistic here is very large, above 1e5, while SupADF gives 0; below about 1e5 times that rounding, SupADF gives nearly the same values as this method, and they shrink with the moves to ordinary sizes as the moves reach the rounding, where both methods read the noise.

The value at bar t reads only s[t-n .. t], so the method is causal, does not repaint and does not depend on where the history starts: a suffix s[k:] gives at bar t-k the full series’ value at bar t, for every t >= k + n, and a prefix gives the first values of the full series’ result, bit for bit. Scaling the series by a power of two, 2^k, for every k that keeps the values normal and finite, or negating it, leaves the result unchanged bit for bit (negation leaves every ADF-type statistic with a constant unchanged), and adding a constant changes it only by rounding. Inside SupADF’s narrower domain, SupADF(n, n) agrees with it to rounding, not bit for bit, except near a perfect fit, where the two guards differ. The bits are the same on every platform.

The period-4 pattern +1, +1, -1, -1 (repeating, in any phase, also scaled by a power of two) with n a multiple of 4 gives -sqrt(n-2) exactly at every valid bar, because the lagged level and the level both have mean 0 over whole periods, so their co-moment is 0 and b = -1. The powers of two s[i] = 2^-i are a perfect fit and give 0, as do a flat series and a strictly alternating two-price series.

In the bubble literature (see SupADF) the test is right-tailed: large positive values reject the unit root in favour of explosiveness. In the classical unit-root test the alternative is stationarity and the test is left-tailed: large negative values reject. Critical values depend on n, and none is quoted here.

The cost is O(n) per valid bar, one kernel fit and one residual pass. A call makes two allocations, the result and the kernel, for every length with a valid bar; an invalid n makes one, the result, and an empty series none. The result is a new series; the method reads s only, so concurrent calls on a shared input are safe.

Verification: two clean-room references implement the contract in testdata/gle257-wp32 on its 27 input series. A Python reference computes each window’s statistic exactly, in rational arithmetic with the square root to 50 digits (mpmath), and gives an error bound per bar; a Julia reference fits each window by direct least squares in binary64. The Julia fit and this method both lie within the Python bound at every compared bar (the knife edges of the guard and the bars beyond the domain are not compared), and match its zeros and NaN bars exactly. The identity with OUTheta, the period-4 anchor and the scaling and negation identities hold as stated, and the outputs are the same bytes natively (arm64) and as amd64 code. The adfuller of statsmodels is a cross-check only: it agrees to rounding on ordinary windows, but its uncentred design loses digits where the level is large against the moves.

SupADF

func (s Series) SupADF(minPairs, maxPairs int) Series

SupADF returns, at each bar, the largest Dickey-Fuller t-statistic over the windows of minPairs to maxPairs lag pairs that end at that bar, the bounded backward supremum ADF sequence.

A window of m lag pairs ending at bar t reads the m + 1 values s[t-m .. t] and has the statistic of DickeyFuller, t = b sqrt((m-2) Sxx/SSR), with b the coefficient of the lagged level in the regression of the change on a constant and the lagged level. Each window has m - 2 degrees of freedom: m rows less two coefficients. SupADF takes the largest of these statistics over m = minPairs .. maxPairs; the windows are visited from the shortest, the first defined value is kept on a tie, and the winning window is not returned (no argmax). A window with a flat lagged level or a perfect fit is undefined and is skipped, and the bar reads 0 when no window is defined. When minPairs and maxPairs both equal n there is one window, and inside the domain below SupADF agrees with DickeyFuller(n) to rounding, not bit for bit, except near a perfect fit, because the supremum accumulates each window in O(1) per row and its guard differs from that of DickeyFuller.

In the terms of Phillips, Shi and Yu (2015), PSY below, this is the backward SADF (BSADF) sequence, bounded to the last maxPairs pairs. It is AFML’s SADF_t (Advances in Financial Machine Learning, chapter 17, section 17.4.2), which follows Phillips, Wu and Yu (2011), PWY below, International Economic Review 52(1), with the start points t0 running from t - maxPairs to t - minPairs as price indices. AFML’s get_bsadf returns the key ‘gsadf’ for what PSY call the BSADF sequence. PWY’s SADF (a fixed start and an expanding end) and PSY’s GSADF (the supremum of BSADF over the end points) depend on where the history starts and are not methods. A rolling GSADF is the recipe SupADF(a, b).Highest(k), valid from bar b + k - 1; Highest carries a NaN for k bars, so a NaN of SupADF at bar j makes the recipe NaN from bar j to j+k-1.

minPairs and maxPairs count regression rows (lag pairs), so a window of m pairs uses m + 1 values. This is the unit of exuber’s minw and of AFML’s minSL at lag 0; psymonitor’s swindow0 counts prices, one more.

Pass log prices, c.Log().SupADF(minPairs, maxPairs). On raw prices a rejection of the unit root would imply stationary prices of finite variance, so return volatility would have to shrink as prices rise, while on log prices a rescaling of the price only shifts the series, which the constant absorbs (AFML, section 17.4.2.1, on raw and log prices). The method takes no logarithm and uses the receiver as given; levels (a spread, a yield) are legitimate input.

The statistic is right-tailed: large positive values mark explosive behaviour. It is unchanged by negation, so it responds to explosive falls as to rises.

A minPairs below 3, a maxPairs below minPairs, or a maxPairs of at least len(s) gives len(s) zeros; minPairs equal to maxPairs is valid and gives one window. The minimum length of a series with a valid bar is maxPairs + 1.

The warm-up reads 0, not NaN: bars 0 to maxPairs-1 are 0 and the first valid index is maxPairs. A partial supremum over the early bars, from minPairs, would depend on where the history starts and is not given. Because 0 is also a legitimate value, mask the warm-up where it matters with SupADF(minPairs, maxPairs).SetN(maxPairs, math.NaN()).

Row i of the regression is bad when s[i] - s[i-1] is not finite, which covers a non-finite s[i-1] or s[i] and a finite pair whose difference overflows. A bad row among the rows t-maxPairs+1 .. t gives NaN at bar t, so a shorter clean window does not win past a NaN. A non-finite s[p] makes rows p and p+1 bad and gives NaN from the later of p and maxPairs to the earlier of p+maxPairs and len(s)-1. A finite pair whose difference overflows makes row p bad only, with NaN from the later of p and maxPairs to the earlier of p+maxPairs-1 and len(s)-1. The bars p-1 and p+maxPairs, which read one huge value without a bad row, lie beyond the domain below and can be NaN or finite. Two adjacent non-finite values, at p and p+1, make three rows bad, with NaN up to the earlier of p+1+maxPairs and len(s)-1. Warm-up bars stay 0 even where the input is bad, and bars whose windows do not read the value are unchanged.

Finite inputs whose magnitudes (and so window ranges) stay below about 1e150 give finite outputs; beyond that the arithmetic decides, with no special rule. SupADF also has a lower end: it multiplies unscaled differences, so where some value of s[t-maxPairs+1 .. t] differs from s[t-maxPairs] by a nonzero amount below about 2^-500 its accuracy is not assured. DickeyFuller scales each window by a power of two first and does not need that limit. SupADF is also accurate only while the dynamic range of each window s[t-maxPairs .. t], the ratio of the largest to the smallest nonzero difference between two of its values, stays below about 2^500 (about 3e150), against about 2^1000 for DickeyFuller. Beyond it the ratio Sxx/SSR under the square root becomes subnormal and then 0, again with no special rule, so a bar can read 0 or a wrong value: the series 0, 2^-300, 0, 2^300 gives 0 from SupADF(3, 3) at bar 3, where the statistic is about -0.577.

A window is undefined when its lagged level is flat or when its residual sum of squares, formed as Sdd - b Sxd, is at most 1e-10 times Sdd, the centred sum of squares of the changes (a perfect fit, a flat change, or a negative value from rounding). This is far looser than the 1e-24 of DickeyFuller, whose guard is relative to the levels and uses explicit residuals. The subtraction form has a rounding floor that grows with the window length and lies far above that of explicit residuals, so a threshold as tight as 1e-24 would let a perfect fit through as an enormous statistic, and a strictly alternating two-price window (a bid-ask bounce on illiquid ticks) is exactly such a fit. The two guards differ, so a window can lie inside one and outside the other.

The windows at bar t share one anchor, the lagged level s[t-1] of the last row, and the rows are added from t backwards, one Welford step each, so after m rows the accumulator holds the window of the m pairs ending at t. The value of a window therefore does not depend on minPairs, maxPairs or the bars before s[t-maxPairs], and these identities hold bit for bit. For any j from minPairs to maxPairs-1, SupADF(minPairs, maxPairs) at a bar t >= maxPairs is the larger of SupADF(minPairs, j) and SupADF(j+1, maxPairs), and NaN where either is NaN, provided each part has at least one defined window. Where the narrower range has a defined window, the value does not fall as maxPairs grows and does not rise as minPairs grows. A part in which every window is undefined (a stretch of strictly alternating prices, say) reads 0 and contributes nothing to the whole, so a longer range that reaches back past the stretch can be negative where the shorter one reads 0.

The value at bar t reads only s[t-maxPairs .. t]. It is window-local and start-invariant: a suffix s[k:] gives at bar t-k the full series’ value at bar t, for every t >= k + maxPairs; a prefix gives the first values of the full series’ result; and the method is causal and does not repaint. Scaling the series by a power of two, 2^k, for every k that keeps the values and the products of their differences normal and finite (the domain above), or negating it, leaves the result unchanged bit for bit, and adding a constant changes it only by rounding. The period-4 pattern +1, +1, -1, -1, with minPairs and maxPairs both equal to a multiple n of 4, gives values within 1e-12 of -sqrt(n-2). The powers of two s[i] = 2^-i, a flat series and a strictly alternating two-price series have only undefined or perfect-fit windows, which SupADF skips, so they give 0.

No critical values are shipped: they depend on minPairs, maxPairs and the null. To simulate them, generate many series, each the cumulative sum of maxPairs + 1 independent N(0, 1) draws (the shortest series with a valid bar; the starting level does not matter, because the constant absorbs it), take the last value of SupADF(minPairs, maxPairs) for each series and read the percentile you need, for example the 95th. The Monte Carlo critical values of exuber and the bootstrap of psymonitor are the published tools. For minPairs, the minimum-window rule of Phillips, Shi and Yu, floor((0.01 + 1.8/sqrt(T))*T) with T the sample length, is guidance: T = 252 gives 31, so SupADF(31, 252) on daily log prices (exuber’s psy_minw computes the same rule).

The cost is O(maxPairs) per bar: maxPairs accumulator steps and up to maxPairs - minPairs + 1 square roots, which is 252 steps a bar at maxPairs = 252, instead of a fresh fit of every window. A call allocates only the result, a new series; the method reads s only, so concurrent calls on a shared input are safe. The bits are the same on every platform, because every product that meets an addition is rounded on its own. Not built: lagged differences (lag 0 only), trends, the QADF and CADF views, and PWY’s SADF and the GSADF as methods.

Verification: the two clean-room references of DickeyFuller (testdata/gle257-wp32) cover SupADF too. The Julia reference also transcribes line by line the binary64 program the contract pins for SupADF, and this method equals that transcription at every cell, bit for bit, NaN matching NaN. The transcription, the Julia direct least squares and this method all lie within the Python reference’s error bound at every compared bar (the knife edges of the guard and the bars beyond the domain at either end are not compared). The decomposition, the period-4 anchor and the scaling and negation identities hold as stated, and the outputs are the same bytes natively (arm64) and as amd64 code.