What Is an Integer Overflow?

Overflow is a hacker exploiting a loophole in the operating system, specially developed a program, and after running with the appropriate parameters, you can get control of your computer with administrator qualifications. He can do everything you can run on your own computer. Here, it's your computer. Overflow is an error caused by insufficient design time of the programmer.

Overflow based detection method

Researcher Crispen Cowan has created an interesting method called StackGuard. Stackguard modifies the C compiler (gcc) to insert a "probe" value before the return address. A "detector" is like a detector in a coal mine: it warns if something goes wrong somewhere. Before any function returns, it performs a check to ensure that the probe value has not changed. If an attacker rewrites the return address (as part of a stack-smashing attack), the value of the detector may change and the system will be terminated accordingly. This is a useful method, but be aware that this method does not prevent buffer overflows from overwriting other values (attackers can still use these values to attack the system). This method has also been extended to protect other values (such as values on the heap). Stackguard (among other defensive measures) is used by Immunix.

IBM's stack-smashing protector (ssp, originally called ProPolice) is a variation on StackGuard's approach. Like StackGuard, ssp uses a modified compiler to insert a probe into function calls to detect stack overflows. However, it adds some interesting changes to this basic idea. It reorders where local variables are stored, and copies pointers in function parameters so that they are before any array. This enhances the protection of ssp; it means that a buffer overflow will not modify the pointer value (otherwise an attacker who can control the pointer can use the pointer to control where the program saves data). By default, it does not detect all functions, but only functions that do need protection (mostly functions that use character arrays). Theoretically, this would weaken protection slightly, but this default behavior improves performance while still preventing most problems. For practical reasons, they use gcc to implement their methods in an architecture-independent way, making it easier to apply. Starting with the May 2003 release, the acclaimed OpenBSD (which focuses on security) uses ssp (also known as ProPolice) throughout their distribution.

Based on the results of StackGuard, Microsoft added a compiler flag (/ GS) to implement the probe in its C compiler.

Overflow Stack Defense

Another approach first makes it impossible to execute code on the stack. Unfortunately, the memory protection mechanisms of x86 processors (most common processors) cannot easily support this; usually, if a memory page is readable, it is executable. A developer named Solar Designer came up with a clever combination of kernel and processor mechanisms, and created a "non-executing stack patch" for the Linux kernel; with this patch, programs on the stack can no longer look like Runs on x86 as usual. It turns out that in some cases, the executable needs to be on the stack; this includes signal processing and trampoline processing. trampoline is a wonderful structure sometimes generated by a compiler, such as the GNAT Ada compiler, to support structures like nested subroutines. Solar Designer also addresses how to prevent these special situations from being affected while preventing attacks.

The original patch to achieve this in Linux was rejected by Linus Torvalds in 1998 for an interesting reason. Even if the code cannot be placed on the stack, an attacker can use a buffer overflow to cause the program to "return" to an existing subroutine (such as a subroutine in the C library) to attack. In short, just having a non-executable stack is not enough.

After some time, people came up with a new idea to prevent this problem: transfer all executable code to a memory area called "ASCII armor" area. To understand how this works, one must be aware of the fact that attackers cannot usually insert ASCII NUL characters (0) using normal buffer overflow attacks. This means that an attacker will find it difficult to make a program return an address containing 0. Because of this fact, moving all executable code to an address containing 0 makes it much harder to attack the program.

The largest contiguous memory range with this attribute is a set of memory addresses from 0 to 0x01010100, so they are named ASCII protected areas (there are other addresses with this attribute, but they are scattered). Combined with a non-executable stack, this approach is quite valuable: the non-executable stack prevents an attacker from sending executable code, and ASCII-protected memory makes it difficult for an attacker to bypass non-executable code by leveraging existing code Stack. This protects program code from stack, buffer, and function pointer overflows, and all without recompilation.

However, ASCII-protected memory is not suitable for all programs; large programs may not fit in ASCII-protected memory areas (so this protection is not perfect), and sometimes an attacker can insert 0 into a destination address. In addition, some implementations do not support springboard code, so it may be necessary to disable this feature for programs that require this protection. Red Hat's Ingo Molnar implemented this idea in his "exec-shield" patch, which is used by the Fedora core (a free version available from Red Hat). The latest version of OpenWall GNU / Linux (OWL) uses an implementation of this method provided by Solar Designer (see Resources for a link to these versions). ^[3]