What are the most files I can have in a folder?

Question

I’m developing a web application to run on linux.

In my project the visitor of my site can send images as many times as he wants. My image upload folder structure will be separated by year/month. Similar to the Wordpress structure for file uploads:

/wp-content/uploads/2014/09/imagem.jpg

The question is:
Is there a limit on the amount of files in a single folder? If for example in a single month I have 1,000,000 images on my website in the folder 2014/09?
Another question:
If there are too many files in a folder, performance is compromised when requesting a file in the folder?

I think it depends on the filing system and sometimes I keep asking myself: How big companies like facebook and google deal with amount of files and folders?

Answer 1

There is a limit, but it is not the operating system that imposes this, but the file system. Check which file system your server uses, and search for your maximum amount of files per folder. In that answer in Soen has several, but not all, so some additional research may be required. I will transcribe the list here, and complement with a few more:

FAT32:

• Maximum number of files: 268,173,300
• Maximum number of files per folder: 2¹⁶ - 1 (65,535)
• Maximum file size: 2 Gib - 1 without LFS, 4 Gib - 1 with

NTFS:

• Maximum number of files: 2³² - 1 (4,294,967,295)
• Maximum file size
  • Implementation: 2⁴⁴ - 2⁶ bytes (16 Tib - 64 Kib)
  • Theoretical: 2⁶⁴ - 2⁶ bytes (16 Eib - 64 Kib)
• Maximum volume size
  • Implementation: 2³² - 1 clusters (256 Tib - 64 Kib)
  • Theoretical: 2⁶⁴ - 1 clusters

ext2:

• Maximum number of files: 10¹⁸
• Maximum number of files per folder: ~1.3 10²⁰ (performance problems starting from about 10,000)
• Maximum file size
  • 16 Gib (1 Kib size blocks)
  • 256 Gib (2 Kib size blocks)
  • 2 Tib (size 4 Kib blocks)
  • 2 Tib (size 8 Kib blocks)
• Maximum volume size
  • 4 Tib (1 Kib size blocks)
  • 8 Tib (size 2 Kib blocks)
  • 16 Tib (size 4 Kib blocks)
  • 32 Tib (size 8 Kib blocks)

ext3:

• Maximum number of files: min(sizeVolume / 2¹³, numeroDeBlocos)
• Maximum file size: even if ext2
• Maximum volume size: even if ext2

ext4:

• Maximum number of files: 2³² - 1 (4,294,967,295)
• Maximum number of files per folder: unlimited
• Maximum file size: 2⁴⁴ - 1 bytes (16 Tib - 1)
• Maximum volume size: 2⁴⁸ - 1 bytes (256 Tib - 1)

Reiserfs:

• Maximum number of files: 2³²3 (~4 billion)
• Maximum file size: 1 Eib (8 Tib on 32-bit systems)
• Maximum volume size: 16 Tib

ZFS:

• Maximum number of files: unlimited
• Maximum number of files per folder: 2⁴⁸
• Maximum file size: 16 exbibytes (2⁶⁴ bytes)
• Maximum volume size: 256 trillion yobibytes (2¹²⁸ bytes)

As for performance, it will also depend on the file system, but at first I would say that there is no problem: if you have a full file name, upload it, save it etc does not depend on how many files there are in the folder. It’s time to list them it gets complicated. According to that response in serverfault, the problem is more significant in older file systems, where a very large number of files was not expected.

• NTFS: Although it can store 4 billion files per folder, it degrades relatively fast - around a thousand already begin to notice performance issues, several thousand and the explorer will seem to hang for a considerable time.

• EXT3: the physical limit is 32 thousand files, but the performance suffers after a few thousand files too.

• Reiserfs, XFS, JFS, BTRFS: these are good for many files in a folder as they are more modern and designed to handle many files (the rest were designed at the time when hard drives were measured in MB rather than GB). The performance is much better because they use binary search to get the file you want (others use a more linear algorithm).

(This last comment left me half in doubt if my statement that the performance when accessing an individual file is really correct. I am still convinced that it is so, but a confirmation from reliable sources would be desirable.)

Anyway, if you can avoid 1 million files in the same folder, avoid. Your proposal to group the archives by year and month is good, and personally I do not believe it is necessary to group also on time. But if the cost of doing this isn’t too high, why not? If on the other hand it will require significant changes in your project, I suggest you wait to deal with the problem when and if it arises.