Ways to use Curl with PHP?

Question

I’m studying cURL to continue in my applying, I reviewed my code and monitored the network from Twitter and got the following HTTP Headers:

curl "https://twitter.com/" 

-H "accept-encoding: gzip, deflate, br" 
-H "accept-language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4" 
-H "upgrade-insecure-requests: 1" 
-H "user-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" 
-H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" 
-H "cache-control: max-age=0" 
-H "authority: twitter.com" 
-H "cookie: (RETIRADO POR SEGURANÇA!) --compressed

My code is very simple, but the doubt has arisen, there is need to make the same calls?

Beneath my simple code:

$cookies = [];

$url_twitter = 'https://twitter.com';

$twitter_cookies = curl_init();
curl_setopt_array($twitter_cookies, [
        CURLOPT_URL             => $url_twitter,
        CURLOPT_RETURNTRANSFER  => 1,
        CURLOPT_SSL_VERIFYPEER  => 0,
        CURLOPT_SSL_VERIFYHOST  => 0,
        CURLOPT_COOKIEJAR               => ROOT . 'system' . SEPARATOR . 'cookies' . SEPARATOR . $TwitterUser . '.txt',
        CURLOPT_HEADERFUNCTION  => function($twitter_cookies, $header) use (&$cookies) {
            if (stripos($header, 'Set-Cookie:') === 0) {
                if (preg_match('/Set-Cookie:\s?(.*?);/i', $header, $matches)) {
                    $cookies[] = $matches[1];
                }
            }

            return strlen($header);
        }
    ]
); 

As you can see I haven’t implemented even half of the HEADERS that Twitter showed when copying cURL quoted at the beginning of the topic.

I don’t know if it adds automatically, I understand the basics of HTTPS, but there is need to make these calls or it is added automatically?

"accept-encoding: gzip, deflate, br"

"accept-language: pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4"

"upgrade-insecure-requests: 1"

"user-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

"accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"

"cache-control: max-age=0"

Answer 1

Usually HTTP requests are performed via browser, which naturally provides several HTTP Headers within the request.

The use of the word need It’s complicated, because it depends more on the application side to be requested than what you’re sending. It is necessary to study the application and understand the types of parameters it accepts.

Say you can send any information inside the Curl header, such as teste=123, and the return of the application may vary, such as an Exception of additional arguments or even the application may ignore this.

Answer 2

Headers are used so that both parties can understand each other in general, they define what the client supports or expects as a result, among other options, some for security.

---

The first accept-encoding tells you what kind of client compression it supports. If you use a URL that has no support for this the system will break down, a lot of "strange characters" will appear. Curl can set these headers automatically, stating what it supports, as long as you set the CURLOPT_ACCEPT_ENCODING (before was the CURLOPT_ENCODING) for "" (empty string), if set to null (the default) it will not use any kind of compression.

The accept-language and the upgrade-insecure-requests are self-explanatory, the first may be useful if the website returns language-based information, the second indicates that the client prefers to be redirected to HTTPS if it is on HTTP.

The user-agent extremely important for Curl, so it can pass as any browser, including mobile browsers. Curl has up to CURLOPT_USERAGENT so that you set the desired value, or can do manually, which I usually do.

The accept and the cache-control sets not to use a cached version, but in Curl this is not enough. You should also use the CURLOPT_FRESH_CONNECT and the CURLOPT_FORBID_REUSE depending on the cases to actually create a new request.

The authority: (actually should be :authority) is a "Pseudo-Header" HTTP/2, equivalent to HOST, is recommended to set, but Curl does this automatically, most of the time. But, it is extremely necessary to set manually if connecting directly to an IP, to avoid DNS-Lookup.

The cookie: are cookies can be used in many ways, from CURLOPT_COOKIEFILE or CURLOPT_COOKIE or manually. There is also the option to use CURLOPT_COOKIESESSION to ignore session cookies and the like.

Several other headers may exist (such as referer, pragma, origin, dnt besides things like content-type and content-length which are set automatically when necessary).

But, it is still possible to create a custom header (usually started in x-), in case Twitter itself uses some as the x-csrf-token.

---

Any header can be set in Curl using -H or the CURLOPT_HTTPHEADER, no need to use any other function for this.