Read data from a url

Question

Read data from a url

Asked 6 years, 3 months ago

Viewed 217 times

0

I would like to get data from a url address

Researching I found that one reply

I implemented for my reality

var http = require('http');

            var options = {
                host: 'http://192.168.1.6',
                path: '/sistema.cgi?lermc=0,33:80'
            }
            var request = http.request(options, function (res) {
                var data = '';
                res.on('data', function (chunk) {
                    data += chunk;
                });
                res.on('end', function () {
                    console.log(data);

                });
            });
            request.on('error', function (e) {
                console.log(e.message);
            });
            request.end();

I adapted it to the ip I have that returns a vector

[124.15,1409.51,174988.56,95284.50,0.54,0.00,....]

Just that, no formatting, no header, no nothing. If you look at the source code of the page only the vector appears.

This is an internal network address.

Your google post shows

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

But mine shows the message:

Parse error

1 answer

Browser other questions tagged javascript node.js url request

You are not signed in. Login or sign up in order to post.

by Sorack • **27,430** points · Answer 1 · 2018-08-22T01:19:53+00:00

The first thing to do to recreate a requisition HTTP within the Node.js is to identify the information that is sent in such a request.

I will suggest a step by step using the URL / as an example. For this you will need the Chrome, of Postman and the module got of NPM:

Open an empty tab of Chrome, right click and select the option Inspecionar. A new window with the DevTools will open (referencing the new tab that was opened earlier);

Access the destination address;

Identify the request in the tab Network of DevTools;

Right-click on the request and select the option `Copy > Copy as Curl (cmd);

In case of my request the result was:

curl "/" -H "Connection: keep-alive" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Accept-Encoding: gzip, deflate, br" -H "Accept-Language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7" --compressed

In the Postman click on the option Import located in the upper left corner and go to the tab Paste Raw Text. Cole the eating of cURL which has been obtained in DevTools of Chrome;

When importing, request with all headers (Headers) and necessary information will be assembled;

Click on SEND and check the result. If the status is 200 the request was successfully executed. You can also check the content of the reply in the tab Body;

You can use the Postman to generate the Node code for you or continue with the options from step 8. To generate the code you can click on the option code on the right side of the window and select the language you want;

After confirming all request data, the code to execute install the module got:

npm install got
The code for implementation in Node.js with duly completed parameters shall be as follows::

(async () => {
  const { URLSearchParams } = require('url');
  const got = require('got');

  try {
    const params = new URLSearchParams({});
    const { body } = await got(`https://answall.com?${params}`, {
      headers: {
        'Connection': 'keep-alive',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7'
      }
    });

    console.log(body)
  } catch (err) {
    console.log(err.message)
  }
})();

With the information of cURL provided by the request for your question:

curl "http://147.1.31.61/sistema.cgi?lermc=0,33:80" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Upgrade-Insecure-Requests: 1" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;‌q=0.8" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7" --compressed

We will have the following implementation code:

(async () => {
  const { URLSearchParams } = require('url');
  const got = require('got');

  try {
    const params = new URLSearchParams({ 'lermc': '0,33:80' });
    const { body } = await got(`http://147.1.31.61/sistema.cgi?${params}`, {
      headers: {
        'Connection': 'keep-alive',
        'Cache-Control': 'max-age=0',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;‌q=0.8',
        'Accept-Encoding': 'gzip, deflate',
        'Accept-Language': 'pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7'
      }
    });

    console.log(body)
  } catch (err) {
    console.log(err.message)
  }
})();

Of course we should also look at some things, such as status return of the service and the type of the response, but in essence with the above information it is possible to carry out the request.