标签归档:jsonlines

加载和解析具有多个JSON对象的JSON文件

问题:加载和解析具有多个JSON对象的JSON文件

我正在尝试在Python中加载和解析JSON文件。但是我在尝试加载文件时遇到了麻烦:

import json
json_data = open('file')
data = json.load(json_data)

Yield:

ValueError: Extra data: line 2 column 1 - line 225116 column 1 (char 232 - 160128774)

我看着18.2。json Python文档中的JSON编码器和解码器,但是通读这个看起来糟透了的文档非常令人沮丧。

前几行(用随机条目匿名):

{"votes": {"funny": 2, "useful": 5, "cool": 1}, "user_id": "harveydennis", "name": "Jasmine Graham", "url": "http://example.org/user_details?userid=harveydennis", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 2, "cool": 4}, "user_id": "njohnson", "name": "Zachary Ballard", "url": "https://www.example.com/user_details?userid=njohnson", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 0, "cool": 4}, "user_id": "david06", "name": "Jonathan George", "url": "https://example.com/user_details?userid=david06", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 6, "useful": 5, "cool": 0}, "user_id": "santiagoerika", "name": "Amanda Taylor", "url": "https://www.example.com/user_details?userid=santiagoerika", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 8, "cool": 2}, "user_id": "rodriguezdennis", "name": "Jennifer Roach", "url": "http://www.example.com/user_details?userid=rodriguezdennis", "average_stars": 3.5, "review_count": 12, "type": "user"}

I am trying to load and parse a JSON file in Python. But I’m stuck trying to load the file:

import json
json_data = open('file')
data = json.load(json_data)

Yields:

ValueError: Extra data: line 2 column 1 - line 225116 column 1 (char 232 - 160128774)

I looked at 18.2. json — JSON encoder and decoder in the Python documentation, but it’s pretty discouraging to read through this horrible-looking documentation.

First few lines (anonymized with randomized entries):

{"votes": {"funny": 2, "useful": 5, "cool": 1}, "user_id": "harveydennis", "name": "Jasmine Graham", "url": "http://example.org/user_details?userid=harveydennis", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 2, "cool": 4}, "user_id": "njohnson", "name": "Zachary Ballard", "url": "https://www.example.com/user_details?userid=njohnson", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 0, "cool": 4}, "user_id": "david06", "name": "Jonathan George", "url": "https://example.com/user_details?userid=david06", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 6, "useful": 5, "cool": 0}, "user_id": "santiagoerika", "name": "Amanda Taylor", "url": "https://www.example.com/user_details?userid=santiagoerika", "average_stars": 3.5, "review_count": 12, "type": "user"}
{"votes": {"funny": 1, "useful": 8, "cool": 2}, "user_id": "rodriguezdennis", "name": "Jennifer Roach", "url": "http://www.example.com/user_details?userid=rodriguezdennis", "average_stars": 3.5, "review_count": 12, "type": "user"}

回答 0

您有一个JSON Lines格式的文本文件。您需要逐行解析文件:

import json

data = []
with open('file') as f:
    for line in f:
        data.append(json.loads(line))

行都包含有效的JSON,但总体而言,它不是有效的JSON值,因为没有顶级列表或对象定义。

请注意,由于该文件每行包含JSON,因此您无需费力地尝试一次性分析所有内容或找出流JSON解析器。现在,您可以选择在继续进行下一行之前分别处理每一行,从而节省了进程中的内存。如果文件很大,您可能不想将每个结果附加到一个列表中,然后再处理所有内容。

如果您有一个文件,其中包含带有分隔符的单个JSON对象,请使用如何使用“ json”模块一次读取一个JSON对象?使用缓冲方法解析单个对象。

You have a JSON Lines format text file. You need to parse your file line by line:

import json

data = []
with open('file') as f:
    for line in f:
        data.append(json.loads(line))

Each line contains valid JSON, but as a whole, it is not a valid JSON value as there is no top-level list or object definition.

Note that because the file contains JSON per line, you are saved the headaches of trying to parse it all in one go or to figure out a streaming JSON parser. You can now opt to process each line separately before moving on to the next, saving memory in the process. You probably don’t want to append each result to one list and then process everything if your file is really big.

If you have a file containing individual JSON objects with delimiters in-between, use How do I use the ‘json’ module to read in one JSON object at a time? to parse out individual objects using a buffered method.


回答 1

对于那些绊倒这个问题的人:python jsonlines库(比这个问题要年轻得多)优雅地处理每行一个json文档的文件。参见https://jsonlines.readthedocs.io/

for those stumbling upon this question: the python jsonlines library (much younger than this question) elegantly handles files with one json document per line. see https://jsonlines.readthedocs.io/


回答 2

病了格式化。每行有一个JSON对象,但是它们不包含在较大的数据结构(即数组)中。您可能需要重新格式化它,使其以每行结尾处的逗号开头[和结尾],或者将其作为单独的字典逐行进行解析。

That is ill-formatted. You have one JSON object per line, but they are not contained in a larger data structure (ie an array). You’ll either need to reformat it so that it begins with [ and ends with ] with a comma at the end of each line, or parse it line by line as separate dictionaries.