Extracting json value from page

Question

Extracting json value from page

Asked 5 years, 7 months ago

Viewed 103 times

1

Good evening. I’m trying to extract a json value from a certain page. Trial code:

package main  

import (
    "encoding/json"
    "io/ioutil"
    "net/http"
    "fmt"
)

type Response struct {
    proxy             string `json:"proxy"`
    ip                string `json:"ip"`
    port              string `json:"port"`
    connectionType    string `json:"connectionType"`
    asn               string `json:"asn"`
    isp               string `json:"isp"`
    resType           string `json:"type"`
    lastChecked       int    `json:"lastChecked"`
    get               bool   `json:"get"`
    post              bool   `json:"post"`
    cookies           bool   `json:"cookies"`
    referer           bool   `json:"referer"`
    userAgent         bool   `json:"userAgent"`
    city              string `json:"city"`
    state             string `json:"state"`
    country           string `json:"country"`
    randomUserAgent   string `json:"randomUserName"`
    requestsRemaining int    `json:"requestsRemaining"`
}

func main()  {
    res, _ := http.Get("http://falcon.proxyrotator.com:51337/?apiKey=&country=BR")
    body, _ := ioutil.ReadAll(res.Body)

    var myStoredVariable Response
    json.Unmarshal(body, &myStoredVariable)
    fmt.Printf(myStoredVariable.proxy)
}

The code doesn’t return anything.

Reply JSON gives page:

{
    "proxy": "170.80.14.253:57624",
    "ip": "170.80.14.253",
    "port": "57624",
    "connectionType": "Residential",
    "asn": "263603",
    "isp": "Duplanet Internet E Informatica Ltda - Me",
    "type": "elite",
    "lastChecked": 1552253479,
    "get": true,
    "post": true,
    "cookies": true,
    "referer": true,
    "userAgent": true,
    "city": "Guaramirim",
    "state": "SC",
    "country": "BR",
    "randomUserAgent": "Mozilla\/5.0 (Linux Android 4.1.2 DROID RAZR HD Build\/9.8.1Q-94) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/32.0.1700.99 Mobile Safari\/537.36",
    "requestsRemaining": 15588
}

I need to get the value of proxy, How can I do it? Thank you.

Just one detail: you can convert JSON to Struct using https://mholt.github.io/json-to-go/. The fields must be public/exportable, with the initial uppercase, after all you are passing this struct to another library.

– Inkeliz

2019/04/02 at 16:53

4 answers

Browser other questions tagged golang

You are not signed in. Login or sign up in order to post.

by Júlio César • 33 points · Answer 1 · 2019-03-11T01:49:22+00:00

I got it this way:

request, _ := http.Get("http://falcon.proxyrotator.com:51337/?apiKey=&country=BR")
data, _ := ioutil.ReadAll(request.Body)
var estrutura map[string]interface{}
json.Unmarshal(data, &estrutura)
fmt.Printf(estrutura["proxy"])

by Ainar-G • **227** points · Answer 2 · 2019-03-14T08:44:40+00:00

The fields of the structure should be exported:

type Response struct {
    Proxy             string `json:"proxy"`
    IP                string `json:"ip"`
    Port              string `json:"port"`
    ConnectionType    string `json:"connectionType"`
    ASN               string `json:"asn"`
    ISP               string `json:"isp"`
    ResType           string `json:"type"`
    LastChecked       int    `json:"lastChecked"`
    Get               bool   `json:"get"`
    Post              bool   `json:"post"`
    Cookies           bool   `json:"cookies"`
    Referer           bool   `json:"referer"`
    UserAgent         bool   `json:"userAgent"`
    City              string `json:"city"`
    State             string `json:"state"`
    Country           string `json:"country"`
    RandomUserAgent   string `json:"randomUserName"`
    RequestsRemaining int    `json:"requestsRemaining"`
    Error             string `json:"error"`
}

by user141647 • 1 point · Answer 3 · 2019-03-10T22:50:37+00:00

json.Unmarshal receives a buffer (byte array) and a structure which will be populated with the values of that JSON. Where is the declaration of that structure?

You need to declare the structure and a variable of the same type before passing it to Umarshal.

import (
    "encoding/json"
    "io/ioutil"
    "net/http"
)

type Response struct {
    proxy             string `json:"proxy"`
    ip                string `json:"ip"`
    port              string `json:"port"`
    connectionType    string `json:"connectionType"`
    asn               string `json:"asn"`
    isp               string `json:"isp"`
    resType           string `json:"type"`
    lastChecked       int    `json:"lastChecked"`
    get               bool   `json:"get"`
    post              bool   `json:"post"`
    cookies           bool   `json:"cookies"`
    referer           bool   `json:"referer"`
    userAgent         bool   `json:"userAgent"`
    city              string `json:"city"`
    state             string `json:"state"`
    country           string `json:"country"`
    randomUserAgent   string `json:"randomUserName"`
    requestsRemaining int    `json:"requestsRemaining"`
}

func recuperarProxy() string {
    res, _ := http.Get("http://api.com/?country=BR")
    body, _ := ioutil.ReadAll(res.Body)

    var myStoredVariable Response
    json.Unmarshal(body, &myStoredVariable)
    return myStoredVariable.proxy
}

This code obviously won’t work for http://api.com/? country=BR not being a URL that returns the mentioned JSON, but that’s the idea.

by Ícaro Lima • 48 points · Answer 4 · 2019-03-10T23:45:44+00:00

According to the errors that appears:

You need to define myStoredVariable before.
Utilize myStoredVariable["proxy"] instead of myStoredVariable['proxy'].

How would it look:

url := "http://api.com/?country=BR"
req, _ := http.NewRequest("GET", url, nil)
res, _ := http.DefaultClient.Do(req)
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)

myStoredVariable := map[string]interface{}{}
json.Unmarshal([]byte(body), &myStoredVariable)
return myStoredVariable["proxy"]

In my solution I used a map that the key is a text and the values are of any kind, the good thing is that you write less.