Fractional Differencing

FracDiff

func (s Series) FracDiff(d float64, n int) Series

FracDiff returns the fixed-width fractional difference of order d of s, the truncated binomial expansion of the operator (1-B)^d, B the backshift operator, applied to s over a window of n weights, so that out[t] is the sum of w_k s[t-k] for k = 0 to n-1. It is the fixed-width window form of López de Prado, Advances in Financial Machine Learning (2018), here “AFML”, section 5.5.2 and Snippet 5.3, on the operator of Hosking (1981) and of Peters, Fractal Market Analysis (1994), chapter 13.

The result is s.FIRSum(FracDiffWeights(d, n)) bit for bit: each product is rounded on its own, the terms are summed oldest first from +0 and 0 is added at the end, so a zero result is +0 and every platform gives the same bits.

n is the number of weights, covering lags 0 to n-1: it is the window of the Python package fracdiff and AFML’s “width + 1”, since Snippet 5.3 counts lags. AFML’s and mlfinpy’s threshold form is s.FracDiff(d, FracDiffWidth(d, tau)), which is AFML’s fracDiff_FFD to rounding on gap-free finite data; FracDiffWidth’s 0 is an invalid n here, so a failure of the threshold rule shows as len(s) zeros.

Any finite d is accepted. d = 0 gives the input plus 0 from bar n-1, the other weights being -0. d = 1 is the first difference, Delta(1): with n = 2 it matches over the whole series except for the sign of a zero, and with n > 2 it matches from bar n-1 on finite data. d = 2 with n = 3 is the second difference, s.Delta(1).Delta(1) from bar 2, to rounding. A negative d is truncated fractional integration: d = -1 gives the n-bar window sum, Sum(n) from bar n-1 to rounding, and d = -2 the weights 1, 2, 3 and so on.

The method is linear in s and takes no logarithm: pass log prices yourself, as in Log().FracDiff(d, n), for example c.Log().FracDiff(d, n), which is what AFML and Peters intend, while Kaabar, Deep Learning for Finance (2024), applies fracdiff to price levels.

A constant input c gives cw_{n-1}(d-1), that is c times FracDiffWeights(d-1, n)[0], to rounding, and that is 0 only for an integer d from 1 to n-1, so the output on log prices is “level plus noise” and not a zero-mean return. There is no flat-window rule: a flat window of x gives xw_{n-1}(d-1) to rounding.

The warm-up reads 0, not NaN: bars 0 to n-2 are 0 and the first valid index is n-1. Trim it with Window(-(n-1)) for n >= 2, since Window(0) is empty, or mask it with SetN(n-1, math.NaN()). n = 1 gives each value plus 0. n < 1, n > len(s), a non-finite d and overflowing weights give len(s) zeros, and n > len(s) is checked before any weights are built, so a huge n allocates only the zeros.

Non-finite values follow IEEE arithmetic and stay in the windows that hold them: a NaN gives NaN, and an infinity gives an infinity with the sign of its weighted term, or NaN when the window’s weighted terms are infinities of both signs or the infinity meets a weight of exactly 0, 0*Inf being NaN, as with d = 0 or an integer d, where an infinite value gives NaN even at lags whose weight is 0. Finite values can overflow to an infinity. AFML’s forward-fill and skipping of non-finite bars is a pandas convenience and forward-fill is not reproduced.

Each valid bar costs O(n) and depends only on its own window, so a prefix of s gives a prefix of the result, and the weights cost O(n) memory besides the result, a new series of len(s) values; s is not modified.

AFML picks the smallest d whose output passes an ADF test (Snippet 5.4: statsmodels adfuller, one lag, a constant, tau = 0.01), and nseries has no ADF test with lagged differences (DickeyFuller is the form without lags). Peters’s d = H - 0.5 (his equation 13.10) is theory for the return process, so on log prices the implied order would be H + 0.5, and Hurst is not wired in.

FracDiffWeights

func FracDiffWeights(d float64, n int) []float64

FracDiffWeights returns the n weights of the fixed-width fractional difference of order d, oldest first: index n-1 holds w_0 = 1, the current bar’s weight, and index 0 holds w_{n-1}, the oldest bar’s weight, so that s.FIRSum(FracDiffWeights(d, n)) is s.FracDiff(d, n).

In closed form w_k = (-1)^k C(d, k), the product of (i - d)/(i + 1) for i = 0 to k-1. They are built instead by the recursion of Snippet 5.1 of López de Prado’s Advances in Financial Machine Learning (2018, “AFML” below and in FracDiff), with w_0 = 1 and, for k = 1 to n-1,

w[k] = -w[k-1] / float64(k) * (d - float64(k) + 1)

which divides before it multiplies (mlfinpy multiplies first) and forms the last factor as (d - float64(k)) + 1. Nothing is added to a product, so no compiler fuses the recursion and every platform gives the same weights. Because the last factor is rounded, w_1 is -d only to rounding: FracDiffWeights(0.1, 2) is [-0.09999999999999998 1], and a positive d up to 2^-54, about 5.6e-17, or a negative d down to -2^-53, about -1.1e-16, gives w_1 = -0. d = 0 gives 1 and then -0s, so FracDiffWeights(0, 4) is [-0 -0 -0 1]. A positive integer d gives the signed binomial row to rounding, exactly for d up to 28, and then exact zeros: FracDiffWeights(1, 4) is [0 0 -1 1] with +0s and FracDiffWeights(2, 5) is [-0 -0 1 -2 1]. From d = 29 a quotient such as 406/3 rounds, so FracDiffWeights(29, 30) holds w_3 = -3654.0000000000005 and ends in w_29 = -1.0000000000000007, while the zeros after k = d stay exact. d = 0.5 gives the dyadic weights -Catalan(k-1)/2^(2k-1), exact through k = 31. d = -1 gives 1 to rounding, exactly 1 through k = 48 with w_49 = 0.9999999999999999, and d = -2 gives exactly 1, 2, 3 and so on, so FracDiffWeights(-2, 6) is [6 5 4 3 2 1]. In exact arithmetic the weights sum to w_{n-1}(d-1), the last weight of order d-1.

The result is nil when n < 1 and when d is NaN or an infinity, in both cases before anything is allocated, and nil when any w_k for k = 1 to n-1 is not finite, at the first such weight. Any finite d is otherwise valid, negative and large values included. Overflow happens only at large |d| or large n. A positive d above about 1029.33 overflows once n reaches the hump of the weights near k = d/2, so FracDiffWeights(1030, 500) is finite and FracDiffWeights(1030, 501) is nil, while the integer d = 1029 peaks at 1.43e308 and is finite at every n, reaching an exact zero at k = 1030. For d < -1 the weights grow like k^(-d-1), so d = -1030 overflows from n = 304 and d = -100 from n = 48832, while d = -3 would need n near 2e154.

The slice is newly allocated on every call, and a finite d with a huge n allocates n values, so n must be allocatable, as for the package’s other weight builders such as ALMAWeights and HannWeights.

FracDiffWidth

func FracDiffWidth(d, tau float64) int

FracDiffWidth returns the number of weights that the threshold rule of AFML (see FracDiff) keeps for the fractional order d and the threshold tau, counted as mlfinpy’s get_weights_ffd counts it: the first k >= 1 at which the absolute value of w_k falls below tau, so that the weights 0 to k-1 are kept and a weight equal to tau is kept. It is the n of s.FracDiff(d, FracDiffWidth(d, tau)), which is AFML’s fracDiff_FFD(series, d, thres=tau) to rounding on gap-free finite data. mlfinpy multiplies before it divides, so its count can differ from this one where a weight lies within rounding of tau.

The domain is 0 < tau < 1, since w_0 = 1 is always kept. The result is 0 for any other tau, including 0, -0, negative values, 1, values above 1, NaN and the infinities, for a NaN or infinite d, for a weight that is not finite, and for a threshold that no weight up to k = fracDiffMaxWidth, inclusive, falls below; the cap is 1 << 20, that is 1,048,576. A crossing at exactly k = fracDiffMaxWidth returns fracDiffMaxWidth and one at fracDiffMaxWidth + 1 returns 0. FracDiff reads 0 as an invalid n and so gives zeros, which makes the failure visible, while mlfinpy instead truncates the width to the series length.

A tau above the absolute value of w_1 gives 1, as does d = 0 and, since w_1 is -d only to rounding, a d whose absolute value is below tau by more than that rounding. Equality keeps, so FracDiffWidth(0.5, 0.0625) is 4 and FracDiffWidth(0.5, 0.5) is 2. FracDiffWidth(0.4, 1e-5) is 1458, the window behind AFML’s Figure 5.4 (d = 0.4, tau = 1e-5), a count AFML does not print, and the largest count at tau = 1e-5 over d >= 0 is 4093, near d = 0.11, with 4076 at d = 0.1, 4078 at d = 0.12 and 929 at d = 0.01. For d in (0, 1) the largest counts per threshold are 12 at 1e-2, 74 at 1e-3, 528 at 1e-4, 4093 at 1e-5, 33,244 at 1e-6 and 279,095 at 1e-7, and for d >= 1 no count exceeds 1032 (an integer d gives d + 1, up to 1030 at d = 1029, and overflow ends the range just above), so the cap covers every tau >= 1e-7 for d >= 0, while tau = 1e-8 at d = 0.1 would need 2,174,855 weights and gives 0. An integer d from 0 to 1029 gives d + 1, and 1030 and above overflow and give 0. FracDiffWidth(0.9, 1e-3) is 12, mlfinpy’s own test. For d in (-1, 0) the weights decay like k^(-d-1), so d = -0.1 needs 29,411 at tau = 1e-5, while d = -0.5 would need about 3e9 and gives 0. For d <= -1 the result is 0 because the weights do not fall below tau, and d = -1’s weights stay within 1.5e-13 of 1 up to the cap, so only a tau that close to 1 is crossed: FracDiffWidth(-1, 0.9999999999999999) is 51 and FracDiffWidth(-1, 1 - 1e-14) is 15064. mlfinpy’s order keeps those weights at exactly 1, so it finds no crossing there.

FracDiffWidth allocates nothing and costs at most 1 << 20 steps.

Delta / Change Directional Movement