subprocesses.
Consider this scenario
A web scraping application crawl 100K URL that stored in a thread safe queue and do some data processing, likely to save as a csv file
I created 4 threads to handle the scraping logics, for each thread serve one link, 4 threads keep executing until the size of URL queue is 0
But I found only one CPU getting high usage rate, other 3 cores usages rates are between 5%-20%.
I search on google, people said I need to write multiprocessing logic in order to make use of all cpu cores, if it's true,
Do I need to create 3 more subprocess for each process created 4 threads to do web scraping logics?
That's a rather low level question and many factors affect it including which machine you are on.
Обсуждают сегодня