How to select ALL lines that have a value exceeding 3 or -3 in a pandas dataframe?

Question

Be it:

data = pd.DataFrame(np.random.randn(1000, 4))

I found the following to select ALL lines that have a value exceeding 3 or -3:

data[(np.abs(data) >3).any(1)]

I didn’t understand Any’s parameter 1():

Any other value in place of 1, generates errors! Is there any other way?

Answer 1

On the help page of the function informs that the function entries are:

DataFrame.any(self, axis=0, bool_only=None, skipna=True, level=None, **kwargs).  

The parameter the function is taking as value 1 is Axis.

axis : {0 or ‘index’, 1 or ‘columns’, None}, default 0.

That is, the parameter only accepts 0 or 1. If you run the command np.abs(data) >3).any(0) will have as answer below, because it will make a test grouped by column:

0    True
1    True
2    True
3    True
dtype: bool

It will give error, because to filter a frame of 1000 lines would be necessary a list of 1000 elements with True or False values.

If you transpose the Dataframe and apply the search with 0 will work:

>>> data.T[(np.abs(data) >3).any()]
        0         1         2         3         4    ...       995       996       997       998       999
0  1.450311  1.192274  0.221418  0.154094  1.182402  ... -0.114990  0.022501  1.450387 -0.148182  1.176990
1  1.366100 -1.103633 -0.736515 -1.032142  0.324770  ...  0.367205  0.229713  0.913765 -0.495397  0.943025
2 -0.657914  0.732565 -1.034729  1.256725  0.556723  ...  0.105583 -0.357065 -0.907179 -0.709269 -2.395450
3 -0.611006  0.628155  1.463978 -0.261827 -0.964198  ...  0.338030  1.421146 -1.490041  0.030427  0.887882

[4 rows x 1000 columns]