Each operating system has its strategy to use swap. If you do have leftover memory, your system is probably writing parts of the memory into the swap even without needing to. If there is an urgent need for memory, be quick to release this memory as it is already "Swapped".
RAM consumption will always tend to 100% even with light load because the system takes advantage of the leftover for caching files.
15 or 20 years ago the operating systems were still hitting the hand with these things, and it could happen that a system would be slower with swap than without, with spare memory. There were the tricks of using swap the size of RAM, swap with double RAM, etc. Nowadays it is no longer relevant.