How to get any kind of cookie using Selenium?

Question

Summary

    To summarize everything I said here below, I need to open Whatsapp Web read the QR Code and save cookies so that at the next Selenium startup I do not need to read the QR Code again.
    I found a solution using profile, the idea is good, but I need something scalable, and cookies would be the best option.

The beginning of the great journey

I can get some cookies on a given page, but on another page there is no "type" of cookie different?

I am using the following code to save the cookies:

import pickle
import selenium.webdriver 

driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))

And this one to carry the cookies:

import pickle
import selenium.webdriver 

driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    driver.add_cookie(cookie)

Source: How to save and load cookies using python Selenium webdriver.

The way this works perfectly with Google, but what I need is to get the cookies Whatsapp Web. At first I thought it could be because of the time, since the cookies are created only after the QR Code is read, but I put a team 20 seconds, enough time for reading, but it still didn’t work:

import pickle
import selenium.webdriver
import time

driver = selenium.webdriver.Firefox()
driver.get("https://web.whatsapp.com/")
time.sleep(20)
pickle.dump(driver.get_cookies(), open("cookies.pkl", "wb"))

And the cookies are created, already checked in the browser:

I’m not understanding why he can pick up on Google and Whatsapp Web no. There is some kind of cookie protected or different? If I can get it? What I need is to start Firefox with the QR Code already read, the idea is to take the cookie at first initialization and other "exports" this cookie not to read again since Selenium initializes a clean browser.

Update 1 - Hope in Javascript

I did a new test using a script Javascript.

I used the following script:

var getCookies = function(){
  var pairs = document.cookie.split(";");
  var cookies = {};
  for (var i=0; i<pairs.length; i++){
    var pair = pairs[i].split("=");
    cookies[pair[0]] = unescape(pair[1]);
  }
  return cookies;
}

return getCookies();

Note: This 'Return' is for Selenium to get the return of script.

Source: I need to get all the cookies from the browser

So I make Selenium perform this script and catch the result:

import selenium.webdriver
import os
from pprint import pprint
import time

driver = selenium.webdriver.Firefox()
driver.get("https://web.whatsapp.com/")
#driver.get("https://www.google.com/")

time.sleep(20)
print 'Pegando os cookies'

script_path = os.path.dirname(os.path.abspath(__file__))
script = open(os.path.join(script_path, "js_scripts/get_all_cookies.js"), "r").read()
cookies = driver.execute_script(script)
pprint(cookies)

print 'Pegou!'

Using Whatsapp the result is (cmd):

Pegando os cookies
{u'': u'undefined'}
Pegou!

And when I use Google the result is (cmd):

Pegando os cookies
{u'1P_JAR': u'2017-11-21-12'}
Pegou!

Is there some kind of blockage, or some kind of cookie different as I said before? I can see its content in Firefox settings:

Cookie 1:

Cookie 2:

You can pick up these cookies in any way, or save this session so you don’t have to read the QR Code again?

Update 2 - Profile, a scalable solution

I came close to a solution, I do not know if this will help, but anyway I will post:

import selenium.webdriver
from selenium.webdriver.firefox.webdriver import FirefoxProfile
profile = FirefoxProfile('C:\Users\meu_usuario\AppData\Roaming\Mozilla\Firefox\Profiles\oedq4f8r.default')
driver = selenium.webdriver.Firefox(firefox_profile=profile)
driver.get("https://web.whatsapp.com/")

What this code does, it takes the default profile and opens Firefox with it, so if I read the QR Code before, and I’ll have Selenium open it works, but if I read the QR Code with Selenium and open again does not work, meaning somehow Selenium can not save the cookies in the profile/session.

Update 3 - Cookies are protected?

I was checking the cookies, I noticed that the ones I can pick up have the following description in "send" "Any kind of connection", while the ones I can’t get this so "Only encrypted connections".

Update 4 - Is it possible to access these cookies?

Instructs the browser to never send the cookie under an HTTP request without it being in an encrypted communication channel. The cookie only can be sent via HTTPS. This works even if the user manually type an HTTP request. The HTTP request will be sent, but the browser will not send any cookies marked as "safe". This measure aims to minimize the scenario in which a user malicious induces the victim to perform an HTTP request, which may allow the capture of cookie session of the victim. Such capture may be made through tools that analyze network traffic, such as sniffers or through the log application access.

That is if I’ve been using an http communication I won’t be able to get these cookies, I don’t know how access works drive of Selenium, at first I thought it was really like a drive of hardware, served only as a bridge, but when I was using the drive for Chrome it looked like a local HTTP address with a port, is there any way to access using HTTPS? It would be different the communication of Google Chrome and Firefox, because when running the drive Firefox this information does not appear.

Update 5 - How this protection works?

I took a few more tests, I thought, I’m going to create a cookie cure, so I’ll make sure that’s it, but here comes the surprise, he can catch the cookie cure that I created, but can’t take any cookie with the encrypted value (at least I think this encrypted, I will show here the images).

I ran the following code in Python:

#!/usr/bin/env python2
import selenium.webdriver
from pprint import pprint
import os


def prepara_script(nome_script):
    try:
        script_path = os.path.dirname(os.path.abspath(__file__))
    except NameError:
        script_path = os.getcwd()
    return open(os.path.join(script_path, nome_script + ".js"), "r").read()

os.system('cls')
driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
script = prepara_script("get_cookies")
Store = driver.execute_script(script)
pprint(Store)

The Javascript used was this:

var getCookies = function(){
  var pairs = document.cookie.split(";");
  var cookies = {};
  for (var i=0; i<pairs.length; i++){
    var pair = pairs[i].split("=");
    cookies[pair[0]] = unescape(pair[1]);
  }
  return cookies;
}
document.cookie = "tagname = test;secure";
var cookies = getCookies();
console.log(cookies);
console.log(document.cookie);

Result in console:

Browser cookies:

Cookie 1

Cookie 2

Cookie 3

Cookie 4

As you can see, all the cookies with the encrypted value I can not catch, I made this test using Google, since to make tests using Whatsapp Web I have to read at all times the QR Code.

Update 6 - The Return of Update 4

"Is it possible to access these cookies?"

There are also the supercookies extremely invasive. The operator American Verizon is one of the companies that is supporting this type of tracking. These cookies are entered at the ISP level (Provider of Internet service), so they don’t stay on your machine, but identify the websites you visit. Because they are at the ISP level and not at PC level it is impossible to eradicate them quickly by erasing your history. After a check by the FCC (US regulatory body, equivalent to Anatel) last year, the supercookies verizon are optional and easier to configure by users.

I can at least know if these cookies I’m trying to catch is a "Supercookie"?

Source: http://gizmodo.uol.com.br/guia-completo-cookies-navegadores/

Answer 1

This is a cookie HTTP(S) Only, that is, it cannot be accessed via javascript.

To get it, you can access the HTTP headers of the request. I use the lib in python seleninum-requests, which is similar to Python lib requests.

from seleniumrequests import Firefox
webdriver = Firefox()

response = webdriver.request('GET', 'https://www.google.com/')
print (response.cookies)

But testing on the web.whatsapp.com, it doesn’t really pick up any cookie, because web.whatsapp.com doesn’t send any cookie, at least not via HTTP. Which means he can only be setting this in Javascript.

Taking a look at the javascript codes looking for Cookie, discovered one of the methods of a javascript class called setRefTokCookies. The method is minified, but I formatted it and it was like this:

setRefTokCookies:function(e, p){
    d.setCookie({
        name:g.COOKIE_REF,
        value:e,
        path:g.PP_REF,
        domain:g.COOKIE_DOMAIN,
        secure:!0
    });

    var n={
        name:g.COOKIE_TOK,
        value:p,
        path:g.PP_TOK,
        domain:g.COOKIE_DOMAIN,
        secure:!0
    };
    d.setCookie(n)
}

Oops, now it’s interesting. This method arrow 2 cookies, being the name of one coming from the constant COOKIE_REF and the other coming from the constant COOKIE_TOK. Very similar to the names of the cookies stored, right?

Now we have to find out who the value is p and e passed to this method. The problem is that, as the code is minified, it is not trivial to understand it.

Looking calmly at this code, to see if it would discover the class name of this method, I realized that it always made a test before continuing to call the rest of the code. That test is:

if (!Store.Conn.me)
    return r.error("userPrefs: Me has not loaded yet.")

Cool. We got a guy now named Store, global, which seems to have interesting data and is used by our interest class. Opening the Developer Toolbar on the https://web.whatsapp.com and analyzing the content of the object Store.Conn, we have a pleasant surprise. 2 variables contain the exact content of these 2 tokens, are they Store.Conn.__x_ref and Store.Conn.__x_serverToken. Knowing this, making the code that picks up this information becomes trivial:

import selenium.webdriver

driver = selenium.webdriver.Firefox()
driver.get("https://web.whatsapp.com/")

driver.execute_script('console.log(Store.Conn.__x_ref);')
driver.execute_script('console.log(Store.Conn.__x_serverToken)')

After passing the QR Code, it brings the same values as cookies.

ps: There are other interesting authentication data in this Store.Conn, maybe the most interesting thing is to save it whole and then just reassign the content in Store.Conn, since this variable is global.