I have a scrapy spider written. I need to run

Question

I have a scrapy spider written. I need to run

this scraper every 1 hour. This scraper simply scrapes some data and stores it to database using pipeline.

How would I achieve this ?
Should use cronjob? If I am using cronjob, is there any python package to schedule cronjobs rather than manually creating cronjob ?

Any thoughts would be appreciated.

0

15.10.2018

2 ответов

10 просмотров

Fayaz Khan

cronjobs by themselves are good enough, unless you already have some other infrastructure like celery or supervisor.

0

15.10.2018

dataislife · Accepted Answer

dataislife

Have you looked into using scrapinghub?

0

15.10.2018

Похожие чаты

I have a scrapy spider written. I need to run

2 ответов

Похожие вопросы