Hello! I need to parse all the pages with articles on this site. There is no problem with parsing individual pages. The question is: how to get to all the articles on the site? As far as I understand, this should be done through relative links and xpath, but in the code I did not find relative links to articles.

Here is an example article (the first article in the Technology section).

    1 answer 1

    It's simple. Each page with articles is http://worldagnetwork.com/category/technology/page/ <page number> / I am afraid that it will have to be entered with pens since unfortunately the number of pages is not indicated on the page. Incend it until it starts to return 404) Parsing every page you can get all the links on the page, and then parse the pages on the links. Everything.

    • Please give, as an example, a link to the article I indicated in the question. - Tolkachev Ivan
    • Did not quite understand the question ... You have a link to the article. You indicated it. What exactly do you want to get? - Vasiliy Rusin
    • I did not quite understand where to get the page number. - Tolkachev Ivan
    • Get the text from all a.page-numbers and find the largest int this number of pages. - Vasiliy Rusin
    • See, in response, you wrote that each page with articles is worldagnetwork.com/category/technology/page/ < page> /. Could you give a link in the same form to the page that I indicated in the question, so that I have an example. - Tolkachev Ivan