scrapy----将数据保存为json文件

方法一:通过在cmd中输命令来实现

scrapy crawl novel -o novel.json -s FEED_EXPORT_ENCIDING=UTF-8

novel为爬虫名称(name)

scrapy将数据保存为csv文件: scrapy crawl novel -o novel.csv -s FEED_EXPORT_ENCIDING=UTF-8

方法二:通过pipelines实现

1.自定义自己的pipeline

pipelines.py

import json
class JsonPipeline(object):def __init__(self):self.file = open('novel.json', 'wb')def process_item(self, item, spider):line = json.dumps(dict(item)) + "\n"self.file.write(line.encode('utf-8'))return item

2.在settings中开启自己的pipeline

settings.py

ITEM_PIPELINES = {# 'NovelSpider.pipelines.NovelspiderPipeline': 300,'NovelSpider.pipelines.JsonPipeline': 300,
}