A - Download data from the Hidroweb portal

Question

The National Water Agency makes available in its portal Hydroweb the download of historical series referring to the data obtained by several monitoring stations.

I would like to automate the download of these historical series, however ANA changed its website and I’m having difficulties to accomplish the task. In that question is the R code I based to download the files.

At first I had replaced the links of http://hidroweb.ana.gov.br/ for http://www.snirh.gov.br/hidroweb/, however, the download is not being able to be performed properly.

I imagine the problem is on the line r <- POST(url = paste0(baseurl[1], est, baseurl[2]), body = list(cboTipoReg = "10"), encode = "form"), as the command of POST seems to be unable to properly access the link http://www.snirh.gov.br/hidroweb/Estacao.asp?Codigo=2851050&CriaArq=true&TipoArq=1.

When accessed with the query &CriarArq=true&TipoArq=1 the site shows only the message "Request not allowed", it is not possible to scraping the page to find the file . zip.

I had imagined that only the command POST with the previous URL and form cboTipoReg = '10' would suffice, but seems not to be accessing the page in the expected way. Someone would explain me what is wrong?

Answer 1

Hi

We can do similarly by updating the base url to: "http://www.snirh.gov.br/hidroweb/rest/api/documento/convencionais?tipo=1&documentos="

Suggestion in R:

## Download automático dados de estações convencionais SNIRH -----

library(httr)

baseurl = "http://www.snirh.gov.br/hidroweb/rest/api/documento/convencionais?tipo=&documentos="
ListaEstacaoes = c(40025000,40050000,40070000,40100000)

setwd("D:/PastaParaDownload")
destino = getwd()

# tipo=1 arquivo access *.mdb
# tipo=2 arquivo texto  *.txt
# tipo=3 arquivo excel  *.csv
tipo = 1

#substituindo o tipo
baseurl = gsub("tipo=",paste0("tipo=",tipo),baseurl)

for(i in 1:length(ListaEstacaoes)){
  baseurl_est = paste0(baseurl,ListaEstacaoes[i])

  #Conexao
  r = POST(url = baseurl_est, body = list(cboTipoReg = "10"), encode = "form")
  if (r$status_code == 405) {
    cont = content(r, as = "text", encoding="ISO-8859-1")
    download.file(baseurl_est, paste0(ListaEstacaoes[i], ".zip"), mode = "wb")
  }
}

Additionally, for telemetric stations:

## Download automático dados de estações telemetricas SNIRH -----

library(httr)

# URL Base 1 - gera o arquivo
baseurl1 = "http://www.snirh.gov.br/hidroweb/rest/api/documento/gerarTelemetricas?codigosEstacoes=&tipoArquivo=&periodoInicial=&periodoFinal="
# URL Base 2 - baixa o arquivo
baseurl2 = "http://www.snirh.gov.br/hidroweb/rest/api/documento/baixarTelemetricas?codigosEstacoes="

# Informação necessaria das estacoes telemetricas: ID Codigo
# para obter o ID consultas ficha descritiva da estacao em http://gestorpcd.ana.gov.br/ 
idEstacaoes     = c(84863570,91264360,84763570,84663550,94765311)
CodigoEstacaoes = c(15400000,15341000,15360000,15380000,15326010)

#setwd("D:/PastaParaDownload")
destino = getwd()

#Parametros necessarios e comuns as estacoes
    # Estacoes telemetricas disponiveis apenas em dois formatos
    #tipo = 2 # tipo=2 arquivo texto  *.txt
    tipo = 3 # tipo=3 arquivo excel  *.csv

    # Data de inicio e fim do recorte da serie no formato aaaa-mm-dd 
    dinicio = "2019-01-01"
    dfim = "2019-12-31"

    # Hora de inicio e fim do recorte da serie no formato hh:mm
    hinicio = "00:00"
    hfim = "23:00"

#substituindo os parametros comuns as estacoes
baseurl1 = gsub("&tipoArquivo=",paste0("&tipoArquivo=",tipo),baseurl1)
baseurl1 = gsub("&periodoInicial=",paste0("&periodoInicial=",dinicio,"T03:",hinicio,".000Z"),baseurl1)
baseurl1 = gsub("&periodoFinal=",paste0("&periodoFinal=",dfim,"T03:",hfim,".000Z"),baseurl1)

for(i in 1:length(CodigoEstacaoes)){ 
  baseurl1i = gsub("codigosEstacoes=",paste0("codigosEstacoes=",idEstacaoes[i]),baseurl1)
  baseurl2i = gsub("codigosEstacoes=",paste0("codigosEstacoes=",idEstacaoes[i]),baseurl2)

  #Conexao
  BROWSE(baseurl1i)
  BROWSE(baseurl2i) 
}