Maths

Sections: Math · Clamp · Gaussian Distribution · Random

Math

Squash transforms

A squash maps an indicator onto a bounded scale, so that rare extreme readings cannot dominate a model and every input shares the same range. nseries has no squash method: each is a short composition of existing methods.

Normal-CDF squash (TSSB’s compression): V = 100*Phi(c*x) - 50, in (-50, 50), with Phi the standard normal CDF. Write it with Erf, as x.Mul(c / math.Sqrt2).Erf().Mul(50). That equals 100*Phi(c*x) - 50 to rounding (within 2e-14 on the -50 to 50 scale) and is odd bit for bit with Go’s own erf (negating x negates the result; all platforms but s390x, whose Erf is assembly). It gives -50 and 50 for -Inf and +Inf and NaN for NaN, and reaches exactly -50 or 50 once |c*x| exceeds about 8.4. Its slope at 0 is 100*c/sqrt(2*pi), 9.97 at c = 0.25. Computing 100*Phi - 50 through a CDF instead cancels near 0. TSSB applies c = 0.25 to X/IQR (its SCALE adjustment) and c = 0.5 to (X - median)/IQR (NORMALIZE), with the median and the interquartile range taken over a trailing window. Those adjustments take the trailing median x.Quantile(n, 0.5) and the interquartile range x.Quantile(n, 0.75).Sub(x.Quantile(n, 0.25)) (see Quantile, under Relative: Channels); their full recipes are left to a later work package.
Hyperbolic tangent: x.Div(s).Tanh(), in (-1, 1), reaching exactly -1 or 1 once |x/s| exceeds about 19.1. FisherInverse is the same function: x.FisherInverse() equals x.Tanh() bit for bit.
Logistic minus one half: x.Div(2).Tanh().Div(2), in (-0.5, 0.5), reaching exactly -0.5 or 0.5 once |x| exceeds about 38.1, since 1/(1 + exp(-x)) - 1/2 = tanh(x/2)/2 (with a prescale s, x.Div(2*s)). It agrees with that direct form to within 2e-16, and is the more accurate of the two near 0, where subtracting one half cancels.
Empirical CDF: x.RankPosition(n), in [0, 1], centred by x.RankPosition(n).Sub(0.5).SetN(n-1, 0) (mask the warm-up after centring, or its zeros become -0.5). It is j/(n-1), with j the number of window values below the current one, so ties take the lowest rank; Masters’s #{y <= x}/N - 0.5 counts ties inclusively and, without ties, is (j+1)/n - 0.5.
Rank-Gaussianisation: the rolling rank as a standard normal score:

j := x.RankPosition(n).Mul(float64(n - 1)).Round() // the rank, 0 to n-1, recovered exactly
p := j.Add(0.5).Div(float64(n))                     // in [0.5/n, 1 - 0.5/n]
z := p.Mul(2).Sub(1).Erfinv().Mul(math.Sqrt2).SetN(n-1, 0)

Round repairs the product, which can miss the integer rank by one or two ulps either way (one below at n = 50 and 99, one above at n = 6, two at n = 106). The offset of one half keeps p off 0 and 1, so z is finite wherever the window holds only finite values, at most Phi^-1(1 - 0.5/n) in size to rounding. SetN masks bars 0 to n-2: RankPosition’s warm-up zeros would otherwise become Phi^-1(0.5/n) (-1.96 at n = 20), not 0.

A full-sample empirical CDF, ranked against the whole history at once, uses future data at every bar but the last; of the empirical CDFs, only the rolling forms are causal.

Abs

func (s Series) Abs() Series

Abs returns the absolute value of each value in the series.

Special cases are:

Abs(±Inf) = +Inf
Abs(NaN) = NaN

Acos

func (s Series) Acos() Series

Acos returns the arccosine, in radians, of each value in the series.

Special case is:

Acos(x) = NaN if x < -1 or x > 1

Acosh

func (s Series) Acosh() Series

Acosh returns the inverse hyperbolic cosine of each value in the series.

Special cases are:

Acosh(+Inf) = +Inf
Acosh(x) = NaN if x < 1
Acosh(NaN) = NaN

Asin

func (s Series) Asin() Series

Asin returns the arcsine, in radians, of each value in the series.

Special cases are:

Asin(±0) = ±0
Asin(x) = NaN if x < -1 or x > 1

Asinh

func (s Series) Asinh() Series

Asinh returns the inverse hyperbolic sine of each value in the series.

Special cases are:

Asinh(±0) = ±0
Asinh(±Inf) = ±Inf
Asinh(NaN) = NaN

Atan

func (s Series) Atan() Series

Atan returns the arctangent, in radians, of each value in the series.

Special cases are:

Atan(±0) = ±0
Atan(±Inf) = ±Pi/2

Atan2

func (s Series) Atan2(y float64) Series

Atan2 returns the arc tangent of y/x, using the signs of the two to determine the quadrant of the return value.

Special cases are (in order):

Atan2(y, NaN) = NaN
Atan2(NaN, x) = NaN
Atan2(+0, x>=0) = +0
Atan2(-0, x>=0) = -0
Atan2(+0, x<=-0) = +Pi
Atan2(-0, x<=-0) = -Pi
Atan2(y>0, 0) = +Pi/2
Atan2(y<0, 0) = -Pi/2
Atan2(+Inf, +Inf) = +Pi/4
Atan2(-Inf, +Inf) = -Pi/4
Atan2(+Inf, -Inf) = 3Pi/4
Atan2(-Inf, -Inf) = -3Pi/4
Atan2(y, +Inf) = 0
Atan2(y>0, -Inf) = +Pi
Atan2(y<0, -Inf) = -Pi
Atan2(+Inf, x) = +Pi/2
Atan2(-Inf, x) = -Pi/2

Atanh

func (s Series) Atanh() Series

Atanh returns the inverse hyperbolic tangent of each value in the series.

Special cases are:

Atanh(1) = +Inf
Atanh(±0) = ±0
Atanh(-1) = -Inf
Atanh(x) = NaN if x < -1 or x > 1
Atanh(NaN) = NaN

Cbrt

func (s Series) Cbrt() Series

Cbrt returns the cube root of each value in the series.

Special cases are:

Cbrt(±0) = ±0
Cbrt(±Inf) = ±Inf
Cbrt(NaN) = NaN

Ceil

func (s Series) Ceil() Series

Ceil returns the least integer value greater than or equal to x.

Special cases are:

Ceil(±0) = ±0
Ceil(±Inf) = ±Inf
Ceil(NaN) = NaN

Copysign

func (s Series) Copysign(sign float64) Series

Copysign returns a value with the magnitude of f and the sign of sign.

Cos

func (s Series) Cos() Series

Cos returns the cosine of the radian argument x.

Special cases are:

Cos(±Inf) = NaN
Cos(NaN) = NaN

Cosh

func (s Series) Cosh() Series

Cosh returns the hyperbolic cosine of each value in the series.

Special cases are:

Cosh(±0) = 1
Cosh(±Inf) = +Inf
Cosh(NaN) = NaN

Dim

func (s Series) Dim(y float64) Series

Dim returns the maximum of x-y or 0.

Special cases are:

Dim(+Inf, +Inf) = NaN
Dim(-Inf, -Inf) = NaN
Dim(x, NaN) = Dim(NaN, x) = NaN

Erf

func (s Series) Erf() Series

Erf returns the error function of each value in the series.

Special cases are:

Erf(+Inf) = 1
Erf(-Inf) = -1
Erf(NaN) = NaN

Erfc

func (s Series) Erfc() Series

Erfc returns the complementary error function of each value in the series.

Special cases are:

Erfc(+Inf) = 0
Erfc(-Inf) = 2
Erfc(NaN) = NaN

Erfcinv

func (s Series) Erfcinv() Series

Erfcinv returns the inverse of Erfc(x).

Special cases are:

Erfcinv(0) = +Inf
Erfcinv(2) = -Inf
Erfcinv(x) = NaN if x < 0 or x > 2
Erfcinv(NaN) = NaN

Erfinv

func (s Series) Erfinv() Series

Erfinv returns the inverse error function of each value in the series.

Special cases are:

Erfinv(1) = +Inf
Erfinv(-1) = -Inf
Erfinv(x) = NaN if x < -1 or x > 1
Erfinv(NaN) = NaN

Exp

func (s Series) Exp() Series

Exp returns e**x, the base-e exponential of each value in the series.

Special cases are:

Exp(+Inf) = +Inf
Exp(NaN) = NaN

Very large values overflow to 0 or +Inf. Very small values underflow to 1.

Exp2

func (s Series) Exp2() Series

Exp2 returns 2**x, the base-2 exponential of each value in the series.

Special cases are the same as Exp.

Expm1

func (s Series) Expm1() Series

Expm1 returns e**x - 1, the base-e exponential of x minus 1. It is more accurate than Exp(x) - 1 when x is near zero.

Special cases are:

Expm1(+Inf) = +Inf
Expm1(-Inf) = -1
Expm1(NaN) = NaN

Very large values overflow to -1 or +Inf.

Floor

func (s Series) Floor() Series

Floor returns the greatest integer value less than or equal to x.

Special cases are:

Floor(±0) = ±0
Floor(±Inf) = ±Inf
Floor(NaN) = NaN

FMA

func (s Series) FMA(y, z float64) Series

FMA returns x * y + z, computed with only one rounding. (That is, FMA returns the fused multiply-add of x, y, and z.)

Gamma

func (s Series) Gamma() Series

Gamma returns the Gamma function of each value in the series.

Special cases are:

Gamma(+Inf) = +Inf
Gamma(+0) = +Inf
Gamma(-0) = -Inf
Gamma(x) = NaN for integer x < 0
Gamma(-Inf) = NaN
Gamma(NaN) = NaN

Hypot

func (s Series) Hypot(y float64) Series

Hypot returns Sqrt(xx + yy), taking care to avoid unnecessary overflow and underflow.

Special cases are:

Hypot(±Inf, y) = +Inf
Hypot(y, ±Inf) = +Inf
Hypot(NaN, y) = NaN
Hypot(px, NaN) = NaN

J0

func (s Series) J0() Series

J0 returns the order-zero Bessel function of the first kind.

Special cases are:

J0(±Inf) = 0
J0(0) = 1
J0(NaN) = NaN

J1

func (s Series) J1() Series

J1 returns the order-one Bessel function of the first kind.

Special cases are:

J1(±Inf) = 0
J1(NaN) = NaN

Jn

func (s Series) Jn(n int) Series

Jn returns the order-n Bessel function of the first kind.

Special cases are:

Jn(n, ±Inf) = 0
Jn(n, NaN) = NaN

Ldexp

func (s Series) Ldexp(exp int) Series

Ldexp is the inverse of Frexp. It returns frac × 2**exp.

Special cases are:

Ldexp(±0, exp) = ±0
Ldexp(±Inf, exp) = ±Inf
Ldexp(NaN, exp) = NaN

Log

func (s Series) Log() Series

Log returns the natural logarithm of each value in the series. Unlike math.Log, which is wrong on amd64 for positive subnormal values because Go’s assembly implementation does not normalise them, it is accurate for them on every architecture; every other value gives exactly math.Log’s result.

Special cases are:

Log(+Inf) = +Inf
Log(0) = -Inf
Log(x < 0) = NaN
Log(NaN) = NaN

Log10

func (s Series) Log10() Series

Log10 returns the decimal logarithm of each value in the series. Like Log, it is accurate for positive subnormal values on every architecture, where math.Log10 inherits math.Log’s amd64 defect; every other value gives exactly math.Log10’s result. The special cases are the same as for Log.

Log1p

func (s Series) Log1p() Series

Log1p returns the natural logarithm of 1 plus its argument x. It is more accurate than Log(1 + x) when x is near zero.

Special cases are:

Log1p(+Inf) = +Inf
Log1p(±0) = ±0
Log1p(-1) = -Inf
Log1p(x < -1) = NaN
Log1p(NaN) = NaN

Log2

func (s Series) Log2() Series

Log2 returns the binary logarithm of each value in the series. The special cases are the same as for Log.

Logb

func (s Series) Logb() Series

Logb returns the binary exponent of each value in the series.

Special cases are:

Logb(±Inf) = +Inf
Logb(0) = -Inf
Logb(NaN) = NaN

Max

func (s Series) Max(y float64) Series

Max returns the larger of x or y.

Special cases are:

Max(x, +Inf) = Max(+Inf, x) = +Inf
Max(x, NaN) = Max(NaN, x) = NaN
Max(+0, ±0) = Max(±0, +0) = +0
Max(-0, -0) = -0

Note that this differs from the built-in function max when called with NaN and +Inf.

Maximum pairs the receiver with another series bar by bar, with the built-in max, so a NaN beside +Inf gives NaN there.

Maximum

func (s Series) Maximum(y Series) Series

Maximum returns, bar by bar, the greater of the receiver and y, right-aligned at the most recent bar, a NaN operand giving NaN as the built-in max does.

Each overlapping bar uses built-in max. Either NaN operand gives NaN, including beside +Inf or -Inf. Max(y float64) uses math.Max instead, which gives +Inf when paired with NaN. For signed zeros, max(-0, +0) is +0 in either operand order.

As for Add, y is right-aligned at the most recent bar: positions before the overlap are zero. A longer y contributes only its trailing len(s) values; an empty or nil y gives len(s) zeros. The result has the receiver’s length and fresh storage, including an empty, non-nil result for an empty or nil receiver. Series are paired by position only; see “Time alignment” in the manual. There is no warm-up: the first valid index is the first bar of the overlap, or 0 for equal lengths.

For equal lengths, with close c and open o, c.Maximum(o) is the top of the real body, and h.Sub(c.Maximum(o)) is the upper shadow. With high h and low l, h.Maximum(c.Shift(1)) is the true high: from bar 1 on it equals h.TrueHigh(c) bit for bit, while TrueHigh keeps h[0] on bar 0. h.Maximum(c.Shift(1)).Sub(l.Minimum(c.Shift(1))) equals c.TrueRange(h, l, c) bit for bit from bar 1 on. At bar 0, Shift supplies +0 for the missing previous close, whereas TrueRange uses h[0] - l[0].

The composition s.Add(y).Add(s.Sub(y).Abs()).Div(2) is not exact: rounding and overflow can change its result. This method takes one pass and does not modify either input.

Min

func (s Series) Min(y float64) Series

Min returns the smaller of x or y.

Special cases are:

Min(x, -Inf) = Min(-Inf, x) = -Inf
Min(x, NaN) = Min(NaN, x) = NaN
Min(-0, ±0) = Min(±0, -0) = -0

Note that this differs from the built-in function min when called with NaN and -Inf.

Minimum pairs the receiver with another series bar by bar, with the built-in min, so a NaN beside -Inf gives NaN there.

Minimum

func (s Series) Minimum(y Series) Series

Minimum returns, bar by bar, the lesser of the receiver and y, right-aligned at the most recent bar, a NaN operand giving NaN as the built-in min does.

Each overlapping bar uses built-in min. Either NaN operand gives NaN, including beside +Inf or -Inf. Min(y float64) uses math.Min instead, which gives -Inf when paired with NaN. For signed zeros, min(-0, +0) is -0 in either operand order.

For equal lengths, with close c and open o, c.Minimum(o) is the bottom of the real body, and c.Minimum(o).Sub(l) is the lower shadow. With high h and low l, l.Minimum(c.Shift(1)) is the true low: from bar 1 on it equals l.TrueLow(c) bit for bit, while TrueLow keeps l[0] on bar 0. h.Maximum(c.Shift(1)).Sub(l.Minimum(c.Shift(1))) equals c.TrueRange(h, l, c) bit for bit from bar 1 on. At bar 0, Shift supplies +0 for the missing previous close, whereas TrueRange uses h[0] - l[0].

The composition s.Add(y).Sub(s.Sub(y).Abs()).Div(2) is not exact: rounding and overflow can change its result. This method takes one pass and does not modify either input.

Mod

func (s Series) Mod(y float64) Series

Mod returns the floating-point remainder of x/y. The magnitude of the result is less than y and its sign agrees with that of each value in the series.

Special cases are:

Mod(±Inf, y) = NaN
Mod(NaN, y) = NaN
Mod(x, 0) = NaN
Mod(x, ±Inf) = x
Mod(x, NaN) = NaN

Pow

func (s Series) Pow(y float64) Series

Pow returns x**y, the base-x exponential of y. For a positive subnormal x and a finite y that is neither an integer nor ±0.5, math.Pow goes through math.Log and is wrong on amd64; Pow is accurate there on every architecture, and gives exactly math.Pow’s result everywhere else.

Special cases are (in order):

Pow(x, ±0) = 1 for any x
Pow(1, y) = 1 for any y
Pow(x, 1) = x for any x
Pow(NaN, y) = NaN
Pow(x, NaN) = NaN
Pow(±0, y) = ±Inf for y an odd integer < 0
Pow(±0, -Inf) = +Inf
Pow(±0, +Inf) = +0
Pow(±0, y) = +Inf for finite y < 0 and not an odd integer
Pow(±0, y) = ±0 for y an odd integer > 0
Pow(±0, y) = +0 for finite y > 0 and not an odd integer
Pow(-1, ±Inf) = 1
Pow(x, +Inf) = +Inf for |x| > 1
Pow(x, -Inf) = +0 for |x| > 1
Pow(x, +Inf) = +0 for |x| < 1
Pow(x, -Inf) = +Inf for |x| < 1
Pow(+Inf, y) = +Inf for y > 0
Pow(+Inf, y) = +0 for y < 0
Pow(-Inf, y) = Pow(-0, -y)
Pow(x, y) = NaN for finite x < 0 and finite non-integer y

Remainder

func (s Series) Remainder(y float64) Series

Remainder returns the IEEE 754 floating-point remainder of x/y.

Special cases are:

Remainder(±Inf, y) = NaN
Remainder(NaN, y) = NaN
Remainder(x, 0) = NaN
Remainder(x, ±Inf) = x
Remainder(x, NaN) = NaN

Round

func (s Series) Round() Series

Round returns the nearest integer, rounding half away from zero.

Special cases are:

Round(±0) = ±0
Round(±Inf) = ±Inf
Round(NaN) = NaN

RoundToEven

func (s Series) RoundToEven() Series

RoundToEven returns the nearest integer, rounding ties to even.

Special cases are:

RoundToEven(±0) = ±0
RoundToEven(±Inf) = ±Inf
RoundToEven(NaN) = NaN

Sin

func (s Series) Sin() Series

Sin returns the sine of the radian argument x.

Special cases are:

Sin(±0) = ±0
Sin(±Inf) = NaN
Sin(NaN) = NaN

Sinh

func (s Series) Sinh() Series

Sinh returns the hyperbolic sine of each value in the series.

Special cases are:

Sinh(±0) = ±0
Sinh(±Inf) = ±Inf
Sinh(NaN) = NaN

Sqrt

func (s Series) Sqrt() Series

Sqrt returns the square root of each value in the series.

Special cases are:

Sqrt(+Inf) = +Inf
Sqrt(±0) = ±0
Sqrt(x < 0) = NaN
Sqrt(NaN) = NaN

Squared

func (s Series) Squared() Series

Squared returns the squared of each value in the series.

Tan

func (s Series) Tan() Series

Tan returns the tangent of the radian argument x.

Special cases are:

Tan(±0) = ±0
Tan(±Inf) = NaN
Tan(NaN) = NaN

Tanh

func (s Series) Tanh() Series

Tanh returns the hyperbolic tangent of each value in the series.

Special cases are:

Tanh(±0) = ±0
Tanh(±Inf) = ±1
Tanh(NaN) = NaN

Trunc

func (s Series) Trunc() Series

Trunc returns the integer value of each value in the series.

Special cases are:

Trunc(±0) = ±0
Trunc(±Inf) = ±Inf
Trunc(NaN) = NaN

Y0

func (s Series) Y0() Series

Y0 returns the order-zero Bessel function of the second kind. Like Log, it is accurate for positive subnormal values on every architecture, where math.Y0 inherits math.Log’s amd64 defect; every other value gives exactly math.Y0’s result.

Special cases are:

Y0(+Inf) = 0
Y0(0) = -Inf
Y0(x < 0) = NaN
Y0(NaN) = NaN

Y1

func (s Series) Y1() Series

Y1 returns the order-one Bessel function of the second kind.

Special cases are:

Y1(+Inf) = 0
Y1(0) = -Inf
Y1(x < 0) = NaN
Y1(NaN) = NaN

Yn

func (s Series) Yn(n int) Series

Yn returns the order-n Bessel function of the second kind. For n = 0 it is Y0, so it is accurate for positive subnormal values on every architecture.

Special cases are:

Yn(n, +Inf) = 0
Yn(n ≥ 0, 0) = -Inf
Yn(n < 0, 0) = +Inf if n is odd, -Inf if n is even
Yn(n, x < 0) = NaN
Yn(n, NaN) = NaN

Clamp

func (s Series) Clamp(low float64, high float64) Series

Clamp returns a value no lower than the specified low value and no higher than the specified high value. If the value provided is between low and high, it is returned unchanged.

The bounds are inclusive and in the series’ units, and the result is max(low, min(x, high)); a NaN value stays NaN. A NaN bound, or low > high, is invalid and gives all zeros. An infinite bound is valid and clamps one side only, and low == high is valid.

Gaussian Distribution

Gaussian

func (s Series) Gaussian(mean, variance float64) Series

Gaussian implements the Gaussian distribution.

It returns the normal probability density at each value; mean is in the series’ units and variance is in squared units, not a standard deviation.

Random

Rand

func (s Series) Rand(factor float64) Series

Rand modifies each element x by a pseudo-random number in the half-open range [-0.5, 0.5) multiplied by the provided factor. It can be used to generate a random walk of values. TODO – Think about whether to leave the first value unchanged?

RandNorm

func (s Series) RandNorm(n int, factor float64) Series

RandNorm modifies each element x by a normally-distributed pseudo-random amount based on the distribution of the previous n values. If n > 0, the first n-1 values in the series will be unmodified (since we need n values to determine the mean and standard deviation). If n <= 0, we sample the whole range to determine the mean and standard deviation to apply.

factor is a multiplier of the rolling standard deviation, not an absolute amount: each perturbation is a standard normal draw times the n-bar standard deviation times factor.

Series operations Statistics and regression