Hi, I wanted to crawl a website . I want

Question

Hi, I wanted to crawl a website . I want

to find all the urls which match a specific patterns. E.g :
/products/*
how can I find them all and make sure there are no more urls with this pattern? the urls are not on the page.

0

02.11.2019

2 ответов

14 просмотров

Shahab Els

you can use beautifulsoup from bs4 and work with css selector to find all urls

0

02.11.2019

Gautam Krishna R · Accepted Answer

Gautam Krishna R

try iterating each one and check http status codes... If there's no status code check the body of the page for error message....

0

02.11.2019

Похожие чаты

Hi, I wanted to crawl a website . I want

2 ответов

Похожие вопросы