Can references affect performance?

Question

In C#, for example, I can reference several namespaces:

using System;
using System.Collections.Generic;
using System.Web.Security;
using System.Web.UI;
using System.Text;

The very one Visual Studio, from version 2015, if I’m not mistaken, shows which of these references are not being used, fading them.

During development should I worry about this and remove the references that are not being used from certain classes? Or the compiler himself already does it somehow?

I understand that the execution performance of my code will not be affected, since I do not use the reference, correct?

But with it I can generate DLLs unnecessary or even let my project’s Dlls get bigger by undermining performance in some way?

There are differences in other languages that may require more attention?