What architecture and operating system properties allow a buffer overflow attack?

Question

Attacks of the kind buffer overflow (data overflow or overflow) occurs when, while writing in a memory space, a program crosses the boundaries of the buffer and writes to an adjacent memory area.

I would like to know what characteristics, from the point of view of architecture and the operating system, allow this type of attack?

Answer 1

It’s kind of a broad question, but come on.

The biggest risk is using a low-level programming language, type C, because in it you are forced to use pointers to manipulate strings and buffers. Sooner or later the programmer makes a mistake and opens a gap.

Using higher-level language decreases the risk. Of course the compiler or interpreter of the language itself may have bugs, because it is probably written in C, but as a rule this type of code goes through more scrutiny, and the risk is concentrated in a smaller area.

If using C is mandatory, the ideal is to use a library for strings, for example the venerable Qmail used an author’s own library. Again, the library may have bugs but you focus the risk on a relatively small code instead of spreading throughout the code of each program.

The operating system can help a lot to mitigate the risks of buffer overflow, with features in style:

• randomization of the memory layout of each process: avoids that the same elements always occupy the same address in the virtual memory. Attacks like "script kiddie" assume a fixed address.

• Bit NX (requires CPU support): prevents code on the stack from being executable, which makes some buffer overflow attack modes unfeasible;

The compiler can collaborate by detecting some types of buffer overflow by checking the stack. I think that every modern compiler has already incorporated the ideas of "Stackguard", a popular GCC Fork in the 90s.

The C library (libc) collaborates by checking errors such as double free(), which are not buffer overflow but are also attack vectors.

In both compiler and libc there are even stronger protections, but they cause performance disruption, so the developer can opt for them if the exchange is advantageous. You often inherit poor quality C code, you can’t rewrite it and the way is to defend yourself.

In short, every modern operating system has protections at various levels, and the situation is much better than in the 1990s. But the best prophylaxis is certainly to avoid low-level language unnecessarily.

Answer 2

When compiling a program in C the characteristic of what is each byte of memory is lost. The size of each array is simply removed by performance and it is not possible to determine where each object starts and ends in memory. That way wherever the program writes, it is assumed that it is right.

The operating system does not maintain such granulated memory control that it is able to detect buffer bursts on its own. The virtual memory of each process is divided into pages (usually 4 Kib or 32 Kib). Each page can be readable, writable, or executable. The program code, for example, is readable and executable, but cannot be modified. So the system will know if you try to write on a page that does not exist or can not be modified and will send a signal to the process (which will usually abort in a crash).

That is, the system’s margin of error is one page. And if the adjacent page is allocated and writable, nothing will happen. Since most attacks are based on writing data in a writable memory, the system will do nothing to protect it.

At this level, a technique that can be useful is marking pages that you do not plan to write as read-only. But this is quite a flawed protection since it will cause a crash in case of attack.

What you should do is to deal with this to protect yourself from these attacks in your own program. It is not the role of the operating system.

If you are programming in a high-level language, without using pointers, then you are most likely safe. For many languages it is impossible to write a code capable of this attack.

But if you are in a lower language (like C, where the motto is trust the programmer, he knows what he does), every care is little. In such cases it is never possible to be completely sure that the code is free from attack opportunities. A good solution is to use the Address Sanitizer. Activate in GCC or Clang with the option -fsanitize=address.