Question about multiprocessing How can I determine the optimal number of

Question

Question about multiprocessing How can I determine the optimal number of

subprocesses.
Consider this scenario
A web scraping application crawl 100K URL that stored in a thread safe queue and do some data processing, likely to save as a csv file
I created 4 threads to handle the scraping logics, for each thread serve one link, 4 threads keep executing until the size of URL queue is 0
But I found only one CPU getting high usage rate, other 3 cores usages rates are between 5%-20%.
I search on google, people said I need to write multiprocessing logic in order to make use of all cpu cores, if it's true,
Do I need to create 3 more subprocess for each process created 4 threads to do web scraping logics?

0

15.09.2019

1 ответов

16 просмотров

swarnim · Accepted Answer

swarnim

That's a rather low level question and many factors affect it including which machine you are on.

0

15.09.2019

Похожие чаты

Question about multiprocessing How can I determine the optimal number of

1 ответов

Похожие вопросы