A compiler is a program, that is, it is a set of algorithms. How it is encapsulated to run on a computer matters little. It is common to be a console application, but it can be anything else. There compilers that were written to be primarily a library and be used in many ways, console is just one of them.
Those that were not written so, has no specific form of integration and no adaptation has been made, the communication is made via same console, that is, calls the executable in another process passing the specific arguments to do what you want and captures the output it emits in console (it does not need to be showing on screen). It’s simple and crude like this :) He takes the text that comes out there, gives a slight interpretation and shows it conveniently the way he thinks best.
It is possible that some compiler has some configuration that facilitates something, but in general it does not have or it is something very simple, nothing that will change much the end result. Nothing prevents, but I don’t know anything that is really flexible, that can be used by the IDE for additional information.
In some cases the IDE may want to use its own compiler to do some tasks, usually something more limited and that won’t do all the work the real compiler does, but it can be based on it, since today almost everything is open source and can be tweaked as needed. But of course some are made from scratch and the compiler is from the IDE (even though it is from the IDE, it is not the IDE that is doing the work).
These underlining are usually done by this internal compiler and not by the official compiler. So it’s usually a library, even if it’s adapted for this. It would be unfeasible to keep invoking a complete compiler all the time to indicate the errors in each thing you type.
There are languages that provide a language server where the IDE can communicate through an API. It’s no different than a library, only the API access way is different. I prefer everything in the same process.
One of the reasons that the . NET Compiler Platform was created this way is just not having to write another compiler for every kind of need, and it’s something other languages are doing, as is the case of Rust.
There is no single solution, but the standard compiler in the language console is not usually used to help with code editing. Almost all Ides have their own syntax compiler and even semantics, but no optimizers or code generator, of course.