Posts by Arthur Alvim • 647 points
17 posts
-
3
votes4
answers684
viewsQ: Variable interpolation in shell script that returns the output of a function
I isolated a problem from a larger script that I am producing. I basically have a function that returns the current date and time in a specific format. I need every time the variable data_hora is…
-
3
votes1
answer55
viewsQ: Is there any way to disable Scrapyd’s web interface?
Is there any way to disable the Scrapyd web interface? I would like to monitor the server only by the api.
-
3
votes4
answers1852
viewsA: Programmatically generate and download links
I implemented something in Python that can help you. It will download the file and name it with the rank number. This does not answer the question. It would be good if you provide an example so that…
ranswered Arthur Alvim 647 -
1
votes2
answers376
viewsA: Problems with Javascript, urllib and Beautifulsoup
In some situations, such as this problem you can study a little the code of the page and make the calls that javascript would make. Here’s a class I implemented that mimics this. It’s in Python 2.7.…
-
1
votes3
answers683
viewsA: list index out of range when trying to resolve the issue of google Developer day 2010
I found the question interesting and would like to complement with a solution with a face more OO. # -*- coding: utf-8 -*- from itertools import tee class ListaTelefones(object): def __init__(self,…
-
3
votes2
answers1301
viewsA: How to format datetime in YYYY-MM-Ddthh:mm:ss.sTZD format in Django/Python?
Apparently it’s the format you want. import datetime data = datetime.datetime.utcnow() print data.isoformat() # output '2015-02-27T00:20:53.351328' And if you are looking for a good app to…
-
4
votes4
answers1398
viewsA: Tool to generate Xpath
I compiled the answers found here: https://stackoverflow.com/questions/3030487/is-there-a-way-to-get-the-xpath-in-google-chrome "Chrome Dev Tools" When using "Chrome Dev Tools" (You can right-click…
-
5
votes3
answers1207
viewsA: What’s the difference when calling a function with parentheses and without in.py urls with Django 1.7?
Basically a "view" is a function that takes a request (resquest) and returns a response (re-sponse). from django.http import HttpResponse import datetime def index_view(request): now =…
-
4
votes1
answer430
viewsA: What is the best way to organize views in Django?
I’m trying to understand your question. If it’s about the organization you can split your.py views into small modules if it gets too big. original views.py follows below: # (imports) ... def…
-
2
votes1
answer694
viewsA: How to build a url in Django?
Basically, it is necessary to define a url, a view and a name for url. # urls.py from django.conf.urls import patterns, url urlpatterns = patterns('', url(r'^expressao_regular/$',…
-
1
votes2
answers843
viewsA: Display contents of a dictionary in sorted mode according to values
You can use an Ordereddict. from collections import OrderedDict months = {'January':31,'February':28,'March':31,'April':30,'May':31,'June':30,…
python-3.xanswered Arthur Alvim 647 -
2
votes2
answers93
viewsA: Does R have a similar command to SAS IN?
Try the function is.element . var1 <- 1 if (is.element(var1, c(1, 2, 4))) { var2 <- 1 print(var2) }
-
3
votes1
answer126
viewsQ: How to manage the operation and failure in the execution of Spiders?
I’m developing a module to get information about the Piders that run on the company’s system. Below is the model where we keep the beginning of operations and the job. I would like to validate if…
-
2
votes2
answers113
viewsQ: How to protect my Scrapyd server from unauthorized calls?
Let’s say I have the following configuration in scrapy.cfg in Scrapyd. [deploy] url = http://example.com/api/scrapyd/ username = user password = secret project = projectX In the Scrapyd…
-
2
votes1
answer138
viewsQ: Deploy queues to manage competition between Piders in Scrapyd
Is there any way that Scrapyd can create Piders queues so that when I send many Piders (with different functions) I can privilege/limit the competition between them? Today, all the Spiders I send…
-
4
votes2
answers126
viewsQ: How to calculate an optimal value for Scrapyd’s CONCURENT_REQUESTS variable?
One of the default settings in Scrapyd is the number of concurrent processes (is 16). CONCURRENT_REQUESTS = 16 What would be the best methodology to calculate an optimal value for this variable? The…
-
3
votes1
answer152
viewsQ: Multiple Pipelines to Treat Different Files in Scrapy
How to treat pipelines.py when we have different Piders? Example: I have a Spider that works by getting posts from a particular blog and another by saving images from jpeg banners found on each…