How to transform an array of arrays into a nested array in python

Question

I’m looking to transfer an array of arrays in a nested numpy matrix, i.e.:

array[array[1,2],array[1,2,3],array[8]]

in

array[ [1,2,nan],
       [1,2,2],
       [8,nan,nan] ].

That’s right when I convert to pandas and then to numpy again however, my problem involves 22.4GB of memory, and for the dataframe it n supports, so I wanted something straight in numpy

Answer 1

Dear, I don’t know any commands that do this in a single step. In this case, I believe I have to follow the steps below:

Load the numpy

from numpy import array
import numpy as np

Define the array

arr = array([array([1,2]), array([1,2,3]), array([8])], dtype=object)

Define variables

max_size = 3
num = np.nan

The max_size is the size of the largest array you have. In this example 3.

The variable num will be the number you want to fill in the gaps, in this case the np.nan.

Resize the arrays

new_l = [np.pad(a.astype(np.float32), (0, max_size - a.shape[0]), mode='constant', constant_values=num) for a in arr]

The np.pad will add max_size - a.shape[0] elements to the right of each array. Why to the right? Because of the tuple (0, max_size - a.shape[0]), the 0 says nothing will be inserted to the left.

Note have to convert the types to float so that np.Nan is part of the array. Hence the astype(np.float32)

Create the new array by stacking the arrays

new_arr = np.vstack(new_l)

The result is:

>>> new_arr
array([[ 1.,  2., nan],
       [ 1.,  2.,  3.],
       [ 8., nan, nan]], dtype=float32)

I hope it helps