The most correct approach to deal with the situation is to check if the user is allowed to delete the object before doing so. For this, you need an authentication system and access control. Django itself provides ready-made mechanisms for this, but there are also several other third-party apps that implement more complex mechanisms. The keywords you should search for, associated with Django, are: Authentication backend, Permissions, Object Permissions.
Your concern with this access control is a sign that you have the right concerns. But the idea of controlling the Urls is not a good idea. Remember that the user can always enter manually with the Urls, without having received it from your application.
In WEB programming you need to consider that the user always has full control of the requests made to your application and that it is not limited by what is displayed to him.