Question

How to get URL page title from N URLs with scrapy and save it to a JSON file in Python

Answer 1

import scrapy

class Spider(scrapy.Spider):
    name = "test"

    start_urls = [
        'https://www.collectivesolver.com/',
        'https://www.collectivesolver.com/31959/how-to-rotate-a-string-left-one-char-in-c',
        'https://www.collectivesolver.com/31889/how-to-generate-random-string-in-node-js'
    ]

    def parse(self, response):
        yield {
            'title': response.css('title::text').get()
        }


# Windows 10
# Visual Studio Code - In TERMINAL

# scrapy crawl test -o data.json


''' 
run:
 
[
{"title": "Collective Solver"},
{"title": "How to rotate a string left one char in C - Collective Solver"},
{"title": "How to generate random string in Node.js - Collective Solver"}
]
 
'''

 

70+ SQL courses for beginners and professionals