I can’t get the input value "csrfmiddlewaretoken"

Question

I can’t get the input value "csrfmiddlewaretoken" from this link, could anyone help me ? " The script would basically have to visit the page that is in the code and return the value of the input whose name is "csrfmiddlewaretoken" in the console

import requests as r
from bs4 import BeautifulSoup as bs
import time
import os
clear = lambda: os.system('cls')



req = r.get('https://www.udemy.com/join/login-popup/?ref=&display_type=popup&locale=pt_BR&response_type=json&next=https%3A%2F%2Fwww.udemy.com%2Fmobile%2Fipad%2F&xref=')

from bs4 import BeautifulSoup as bs

soup = bs(req.text, 'html.parser')
inp = soup.find('input', {'name': 'csrfmiddlewaretoken'})
val_inp = inp.get('value')

Error:

Traceback (most recent call last):
  File "gg.py", line 18, in <module>
    val_inp = inp.get('value')
AttributeError: 'NoneType' object has no attribute 'get'

On the page:

<input type='hidden' name='csrfmiddlewaretoken' value='VyHNALqmdrUIRP72n3Yj8O8lsN0vfQwtsmzbCYbuTNDbtLR1sbg2Xk368y7U8M7d' />

Value changes with each request/F5/refresh

Answer 1

If you analyze the request reset in the variable soup (with the command print(repr(soup))) you will see that the site is returning the message:

Access to this page has been denied because we Believe you are using Automation tools to Browse the website.

That is, the site detects that the request was made by an automatic tool and returns a different content than expected.

A possible solution for this case is to inform the header User-Agent with the same value as a common browser.

Follow the modified code using the user agent of the Firefox browser:

import requests as r
from bs4 import BeautifulSoup as bs
import time
import os
clear = lambda: os.system('cls')


# AQUI => informar o User-Agent'
req = r.get(
         'https://www.udemy.com/join/login-popup/?ref=&display_type=popup&locale=pt_BR&response_type=json&next=https%3A%2F%2Fwww.udemy.com%2Fmobile%2Fipad%2F&xref=', 
          headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0'}
)

from bs4 import BeautifulSoup as bs

soup = bs(req.text, 'html.parser')
inp = soup.find('input', {'name': 'csrfmiddlewaretoken'})
val_inp = inp.get('value')
print("Valor = {}".format(val_inp))

After running, the program prints:

Valor = tnRToqAGFLH7gsc6TEbGdv30DABebPYFcNf9WnZvsv6xoLmum3hhi3Y3ZUcnVn1a