atoti.array module¶
atoti is designed to handle array data efficiently.
There are multiple ways to load arrays into atoti stores. For instance:
With
atoti.session.Session.read_pandas():>>> df = pd.DataFrame( ... columns=["NumPy array", "Python list"], ... data=[ ... (np.array([1.0, 2.0, 3.0]), [1, 2, 3]), ... (np.array([4.0, 5.0, 6.0]), [4, 5, 6]), ... (np.array([7.0, 8.0, 9.0]), [7, 8, 9]), ... ], ... ) >>> df NumPy array Python list 0 [1.0, 2.0, 3.0] [1, 2, 3] 1 [4.0, 5.0, 6.0] [4, 5, 6] 2 [7.0, 8.0, 9.0] [7, 8, 9] >>> pandas_store = session.read_pandas(df, "DataFrame with arrays") >>> pandas_store.head() NumPy array Python list 0 doubleVector[3]{1.0, ...} intVector[3]{1, ...} 1 doubleVector[3]{4.0, ...} intVector[3]{4, ...} 2 doubleVector[3]{7.0, ...} intVector[3]{7, ...}With
atoti.session.Session.read_csv():>>> pnl_store = session.read_csv( ... f"{RESOURCES}/pnl.csv", ... array_sep=";", ... keys=["Continent", "Country"], ... store_name="PnL", ... ) >>> pnl_store.head() PnL Continent Country Europe France doubleVector[10]{-0.465, ...} UK doubleVector[10]{11.449, ...} America Mexico doubleVector[10]{-10.716, ...} Asia China doubleVector[10]{-1.715, ...} India doubleVector[10]{-18.716, ...}
As for scalar measures, atoti provides the default SUM and MEAN aggregations on array measures.
They are applied element by element:
>>> cube = session.create_cube(pnl_store)
>>> l, m = cube.levels, cube.measures
>>> cube.query(m["PnL.SUM"], m["PnL.MEAN"], levels=l["Continent"])
PnL.SUM PnL.MEAN
Continent
America doubleVector[10]{-10.716, ...} doubleVector[10]{-10.716, ...}
Asia doubleVector[10]{-20.431, ...} doubleVector[10]{-10.2155, ...}
Europe doubleVector[10]{10.984, ...} doubleVector[10]{5.492, ...}
Besides the functions below, arrays support the following operations:
Arithmetic operators:
>>> m["PnL +10"] = m["PnL.SUM"] + 10.0 >>> cube.query(m["PnL +10"]) PnL +10 0 doubleVector[10]{-10.163, ...} >>> m["PnL -10"] = m["PnL.SUM"] - 10.0 >>> cube.query(m["PnL -10"]) PnL -10 0 doubleVector[10]{-30.163, ...} >>> m["PnL x10"] = m["PnL.SUM"] * 10.0 >>> cube.query(m["PnL x10"]) PnL x10 0 doubleVector[10]{-201.63, ...} >>> m["PnL /10"] = m["PnL.SUM"] / 10.0 >>> cube.query(m["PnL /10"]) PnL /10 0 doubleVector[10]{-2.0163, ...}Indexing:
>>> m["First element"] = m["PnL.SUM"][0] >>> cube.query(m["First element"], m["PnL.SUM"]) First element PnL.SUM 0 -20.16 doubleVector[10]{-20.163, ...}This can be used with
atoti.cube.Cube.create_static_parameter_hierarchy()to “slice” the array:>>> cube.create_static_parameter_hierarchy("Index", list(range(0, 10))) >>> m["PnL at index"] = m["PnL.SUM"][l["Index"]] >>> cube.query(m["PnL at index"], levels=l["Index"]) PnL at index Index 0 -20.16 1 -14.97 2 -110.10 3 -76.30 4 .48 5 -57.51 6 -.53 7 -15.49 8 -22.97 9 9.26Non-integer hierarchies can also be created:
>>> from datetime import date, timedelta >>> dates = [ ... date(2020, 1, 1) + timedelta(days=offset) for offset in range(0, 10) ... ] >>> cube.create_static_parameter_hierarchy( ... "Dates", dates, index_measure="Date index" ... ) >>> m["PnL at date"] = m["PnL.SUM"][m["Date index"]] >>> cube.query(m["Date index"], m["PnL at date"], levels=l["Dates"]) Date index PnL at date Dates 2020-01-01 0 -20.16 2020-01-02 1 -14.97 2020-01-03 2 -110.10 2020-01-04 3 -76.30 2020-01-05 4 .48 2020-01-06 5 -57.51 2020-01-07 6 -.53 2020-01-08 7 -15.49 2020-01-09 8 -22.97 2020-01-10 9 9.26It is also possible to provide a list of indices when they need to be of arbitrary order or range:
>>> cube.create_static_parameter_hierarchy( ... "Custom dates", ... dates, ... indices=[9, 8, 7, 6, 5, 0, 1, 2, 3, 4], ... index_measure="Custom date index", ... ) >>> m["PnL at custom date"] = m["PnL.SUM"][m["Custom date index"]] >>> cube.query( ... m["Custom date index"], ... m["PnL at custom date"], ... levels=l["Custom dates"], ... ) Custom date index PnL at custom date Custom dates 2020-01-01 9 9.26 2020-01-02 8 -22.97 2020-01-03 7 -15.49 2020-01-04 6 -.53 2020-01-05 5 -57.51 2020-01-06 0 -20.16 2020-01-07 1 -14.97 2020-01-08 2 -110.10 2020-01-09 3 -76.30 2020-01-10 4 .48Slicing:
>>> m["First 2 elements"] = m["PnL.SUM"][0:2] >>> cube.query(m["First 2 elements"], m["PnL.SUM"]) First 2 elements PnL.SUM 0 doubleVector[2]{-20.163, ...} doubleVector[10]{-20.163, ...}
-
atoti.array.len(measure)¶ Return a measure equal to the number of elements of the passed array measure.
Example
>>> m["Length"] = tt.array.len(m["PnL.SUM"]) >>> cube.query(m["Length"]) Length 0 10
- Return type
-
atoti.array.max(measure)¶ Return a measure equal to the maximum element of the passed array measure.
Example
>>> m["Max"] = tt.array.max(m["PnL.SUM"]) >>> m["Empty max"] = tt.array.max(m["PnL.SUM"][0:0]) >>> cube.query(m["Max"], m["Empty max"]) Max Empty max 0 9.26
- Return type
-
atoti.array.mean(value: ArrayType) → JavaFunctionOperation¶ -
atoti.array.mean(value: Measure) → Measure Return a measure equal to the mean of all the elements of the passed array measure.
Example
>>> m["Mean"] = tt.array.mean(m["PnL.SUM"]) >>> m["Empty mean"] = tt.array.mean(m["PnL.SUM"][0:0]) >>> cube.query(m["Mean"], m["Empty mean"]) Mean Empty mean 0 -30.83 .00
-
atoti.array.min(measure)¶ Return a measure equal to the minimum element of the passed array measure.
Example
>>> m["Min"] = tt.array.min(m["PnL.SUM"]) >>> m["Empty min"] = tt.array.min(m["PnL.SUM"][0:0]) >>> cube.query(m["Min"], m["Empty min"]) Min Empty min 0 -110.10
- Return type
-
atoti.array.n_greatest(measure, n)¶ Return an array measure containing the n greatest elements of the passed array measure.
Example
>>> m["Top 3"] = tt.array.n_greatest(m["PnL.SUM"], n=3) >>> cube.query(m["Top 3"]) Top 3 0 doubleVector[3]{9.259999999999998, ...}
- Return type
-
atoti.array.n_greatest_indices(measure, n)¶ Return an array measure containing the indices of the n greatest elements of the passed array measure.
Example
>>> m["Top 3 indices"] = tt.array.n_greatest_indices(m["PnL.SUM"], n=3) >>> cube.query(m["Top 3 indices"]) Top 3 indices 0 intVector[3]{6, ...}
- Return type
-
atoti.array.n_lowest(measure, n)¶ Return an array measure containing the n lowest elements of the passed array measure.
Example
>>> m["Bottom 3"] = tt.array.n_lowest(m["PnL.SUM"], n=3) >>> cube.query(m["Bottom 3"]) Bottom 3 0 doubleVector[3]{-57.51499999999999, ...}
- Return type
-
atoti.array.n_lowest_indices(measure, n)¶ Return an array measure containing the indices of the n lowest elements of the passed array measure.
Example
>>> m["Bottom 3 indices"] = tt.array.n_lowest_indices(m["PnL.SUM"], n=3) >>> cube.query(m["Bottom 3 indices"]) Bottom 3 indices 0 intVector[3]{2, ...}
- Return type
-
atoti.array.negative_values(measure)¶ Return a measure where all the elements > 0 of the passed array measure are replaced by 0.
- Return type
-
atoti.array.nth_greatest(measure, n)¶ Return a measure equal to the n-th greatest element of the passed array measure.
Example
>>> m["3rd greatest"] = tt.array.nth_greatest(m["PnL.SUM"], n=3) >>> cube.query(m["3rd greatest"]) 3rd greatest 0 -.53
- Return type
-
atoti.array.nth_lowest(measure, n)¶ Return a measure equal to the n-th lowest element of the passed array measure.
Example
>>> m["3rd lowest"] = tt.array.nth_lowest(m["PnL.SUM"], n=3) >>> cube.query(m["3rd lowest"]) 3rd lowest 0 -57.51
- Return type
-
atoti.array.positive_values(measure)¶ Return a measure where all the elements < 0 of the passed array measure are replaced by 0.
- Return type
-
atoti.array.prefix_sum(measure)¶ Return a measure equal to the sum of the previous elements in the passed array measure.
Example
If an array has the following values:
[2.0, 1.0, 0.0, 3.0], the returned array will be:[2.0, 3.0, 3.0, 6.0].- Return type
-
atoti.array.prod(measure)¶ Return a measure equal to the product of all the elements of the passed array measure.
Example
>>> m["Product"] = tt.array.prod(m["PnL.SUM"]) >>> m["Empty product"] = tt.array.prod(m["PnL.SUM"][0:0]) >>> cube.query(m["Product"], m["Empty product"]) Product Empty product 0 122,513,372,194.94 1.00
- Return type
-
atoti.array.quantile(measure, q, *, mode='inc', interpolation='linear')¶ Return a measure equal to the requested quantile of the elements of the passed array measure.
Here is how to obtain the same behavior as these standard quantile calculation methods:
R-1:
mode="centered"andinterpolation="lower"R-2:
mode="centered"andinterpolation="midpoint"R-3:
mode="simple"andinterpolation="nearest"R-4:
mode="simple"andinterpolation="linear"R-5:
mode="centered"andinterpolation="linear"R-6 (similar to Excel’s
PERCENTILE.EXC):mode="exc"andinterpolation="linear"R-7 (similar to Excel’s
PERCENTILE.INC):mode="inc"andinterpolation="linear"R-8 and R-9 are not supported
The formulae given for the calculation of the quantile index assume a 1-based indexing system.
- Parameters
measure (
Measure) – The measure to get the quantile of.q (
Union[float,Measure]) – The quantile to take. Must be between0and1. For instance,0.95is the 95th percentile and0.5is the median.mode (
Literal[‘simple’, ‘centered’, ‘inc’, ‘exc’]) –The method used to calculate the index of the quantile. Available options are, when searching for the q quantile of a vector
X:simple:len(X) * qcentered:len(X) * q + 0.5exc:(len(X) + 1) * qinc:(len(X) - 1) * q + 1
interpolation (
Literal[‘linear’, ‘higher’, ‘lower’, ‘nearest’, ‘midpoint’]) –If the quantile index is not an integer, the interpolation decides what value is returned. The different options are, considering a quantile index
kwithi < k < jfor a sorted vectorX:linear:v = X[i] + (X[j] - X[i]) * (k - i)lowest:v = X[i]highest:v = X[j]nearest:v = X[i]orv = X[j]depending on which ofiorjis closest tokmidpoint:v = (X[i] + X[j]) / 2
- Return type
-
atoti.array.quantile_index(measure, q, *, mode='inc', interpolation='lower')¶ Return a measure equal to the index of requested quantile of the elements of the passed array measure.
- Parameters
measure (
Measure) – The measure to get the quantile of.q (
Union[float,Measure]) – The quantile to take. Must be between0and1. For instance,0.95is the 95th percentile and0.5is the median.mode (
Literal[‘simple’, ‘centered’, ‘inc’, ‘exc’]) –The method used to calculate the index of the quantile. Available options are, when searching for the q quantile of a vector
X:simple:len(X) * qcentered:len(X) * q + 0.5exc:(len(X) + 1) * qinc:(len(X) - 1) * q + 1
interpolation (
Literal[‘higher’, ‘lower’, ‘nearest’]) –If the quantile index is not an integer, the interpolation decides what value is returned. The different options are, considering a quantile index
kwithi < k < jfor the original vectorXand the sorted vectorY:lowest: the index inXofY[i]highest: the index inXofY[j]nearest: the index inXofY[i]orY[j]depending on which ofiorjis closest tok
- Return type
-
atoti.array.sort(measure, *, ascending=True)¶ Return an array measure with the elements of the passed array measure sorted.
Example
>>> m["Sorted ascendingly"] = tt.array.sort(m["PnL.SUM"]) >>> m["Sorted descendingly"] = tt.array.sort(m["PnL.SUM"], ascending=False) >>> cube.query(m["Sorted ascendingly"], m["Sorted descendingly"]) Sorted ascendingly Sorted descendingly 0 doubleVector[10]{-110.09900000000002, ...} doubleVector[10]{9.259999999999998, ...}
- Return type
-
atoti.array.std(measure, *, mode='sample')¶ Return a measure equal to the standard deviation of the elements of the passed array measure.
- Parameters
measure (
Measure) – The measure to get the standard deviation of.mode (
Literal[‘sample’, ‘population’]) –One of the supported modes:
The
samplestandard deviation, similar to Excel’sSTDEV.S, is \(\sqrt{\frac{\sum_{i=0}^{n} (X_i - m)^{2}}{n - 1}}\) wheremis the sample mean andnthe size of the sample. Use this mode if the data represents a sample of the population.The
populationstandard deviation, similar to Excel’sSTDEV.Pis \(\sqrt{\frac{\sum_{i=0}^{n}(X_i - m)^{2}}{n}}\) wheremis the mean of theXielements andnthe size of the population. Use this mode if the data represents the entire population.
- Return type
-
atoti.array.sum(value: ArrayType) → JavaFunctionOperation¶ -
atoti.array.sum(value: Measure) → Measure Return a measure equal to the sum of all the elements of the passed array measure.
Example
>>> m["Sum"] = tt.array.sum(m["PnL.SUM"]) >>> m["Empty sum"] = tt.array.sum(m["PnL.SUM"][0:0]) >>> cube.query(m["Sum"], m["Empty sum"]) Sum Empty sum 0 -308.29 .00
-
atoti.array.var(measure, *, mode='sample')¶ Return a measure equal to the variance of the elements of the passed array measure.
- Parameters
measure (
Measure) – The measure to get the variance of.mode (
Literal[‘sample’, ‘population’]) –One of the supported modes:
The
samplevariance, similar to Excel’sVAR.S, is \(\frac{\sum_{i=0}^{n} (X_i - m)^{2}}{n - 1}\) wheremis the sample mean andnthe size of the sample. Use this mode if the data represents a sample of the population.The
populationvariance, similar to Excel’sVAR.Pis \(\frac{\sum_{i=0}^{n}(X_i - m)^{2}}{n}\) wheremis the mean of theXielements andnthe size of the population. Use this mode if the data represents the entire population.
- Return type