Ioerror: [Errno 24] Too Many open files

Question

When using the command

python generator video.mp4 2 150 80 10 thumbnails.jpg

I have the following return

Extracting 734 frames
  [####################################]  100%
Frames extracted.
Traceback (most recent call last):
  File "generator", line 90, in <module>
    generate_video_thumbnail(arguments)
  File "generator", line 39, in generate_video_thumbnail
    generate_sprite_from_frames(outputPrefix, columns, size, output)
  File "generator", line 62, in generate_sprite_from_frames
    images = [Image.open(filename) for filename in framesMap]
  File "C:\Python27\lib\site-packages\PIL\Image.py", line 2312, in open
    fp = builtins.open(filename, "rb")
IOError: [Errno 24] Too many open files: 'c:\\users\\gpich\\appdata\\local\\temp\\tmpellfxjf728913fdda8373761ccb51c69fe51d5_00497.png'

When I need to generate smaller numbers of thumbnails it generates smoothly. How do I get around this problem?

The project is on github https://github.com/flavioribeiro/video-thumbnail-generator

below the some functions relevant to the generation

def generate_video_thumbnail(args):
    videoFileClip = VideoFileClip(args['<video>'])
    interval = int(args['<interval>'])
    size = (int(args['<width>']), int(args['<height>']))
    outputPrefix = get_output_prefix()
    generate_frames(videoFileClip, interval, outputPrefix, size)

    columns = int(args['<columns>'])
    output = args['<output>']
    generate_sprite_from_frames(outputPrefix, columns, size, output)


def extract_frame(videoFileClip, moment, outputPrefix, size, frameCount):
    output = outputPrefix + ("%05d.png" % frameCount)
    videoFileClip.save_frame(output, t=int(moment))
    resize_frame(output, size)

def resize_frame(filename, size):
    image = Image.open(filename)
    image = image.resize(size, Image.ANTIALIAS)
    image.save(filename)

def generate_sprite_from_frames(framesPath, columns, size, output):
    framesMap = sorted(glob.glob(framesPath + "*.png"))
    images = [Image.open(filename) for filename in framesMap]
    masterWidth = size[0] * columns
    masterHeight = size[1] * int(math.ceil(float(len(images)) / columns))
    finalImage = Image.new(mode='RGBA', size=(masterWidth, masterHeight), color=(0,0,0,0))
    merge_frames(images, finalImage, columns, size, output)

Answer 1

Really, you are opening all the files for each frame of the video. While doing:

images = [Image.open(filename) for filename in framesMap]

You are generating a list (images) with the Handles of each file returned by glob of the previous line. Hence the error.

As I already mentioned in comment, with a 10 minute video recorded at 30 FPS (frames per second), you can easily reach 18,000 files open!

Change your code to process file by file, so you don’t exceed the limit of Handles of your process.

First, change the function code generate_sprite_from_frames thus:

def generate_sprite_from_frames(framesPath, columns, size, output):
    framesMap = sorted(glob.glob(framesPath + "*.png"))

    masterWidth = size[0] * columns
    masterHeight = size[1] * int(math.ceil(float(len(framesMap)) / columns))

    finalImage = Image.new(mode='RGBA', size=(masterWidth, masterHeight), color=(0,0,0,0))

    for col, filename in enumerate(framesMap):
        with Image.open(filename) as img:                              
            merge_frames_add(finalImage, image, col, size)

You will need to make a change also in function merge_frames so that she can open an existing image of spritesheet with all frames and add the current frame to it. And it can be all in memory: the function merge_frames_add simply copy the current frame into image for the spritesheet image on finalImage in the region of interest (ROI) according to the column in col.

Note that in the above example I intentionally changed the name to merge_frames_add to denote this change.

How will you process within a block width ... as ..., with each interaction the handle the file is closed when it leaves this block and so you will not have any more problems. :)