NumPy数组不可JSON序列化

问题:NumPy数组不可JSON序列化

创建NumPy数组并将其另存为Django上下文变量后,加载网页时出现以下错误:

array([   0,  239,  479,  717,  952, 1192, 1432, 1667], dtype=int64) is not JSON serializable

这是什么意思?

After creating a NumPy array, and saving it as a Django context variable, I receive the following error when loading the webpage:

array([   0,  239,  479,  717,  952, 1192, 1432, 1667], dtype=int64) is not JSON serializable

What does this mean?


回答 0

我经常“ jsonify” np.arrays。尝试首先在数组上使用“ .tolist()”方法,如下所示:

import numpy as np
import codecs, json 

a = np.arange(10).reshape(2,5) # a 2 by 5 array
b = a.tolist() # nested lists with same data, indices
file_path = "/path.json" ## your path variable
json.dump(b, codecs.open(file_path, 'w', encoding='utf-8'), separators=(',', ':'), sort_keys=True, indent=4) ### this saves the array in .json format

为了“ unjsonify”数组使用:

obj_text = codecs.open(file_path, 'r', encoding='utf-8').read()
b_new = json.loads(obj_text)
a_new = np.array(b_new)

I regularly “jsonify” np.arrays. Try using the “.tolist()” method on the arrays first, like this:

import numpy as np
import codecs, json 

a = np.arange(10).reshape(2,5) # a 2 by 5 array
b = a.tolist() # nested lists with same data, indices
file_path = "/path.json" ## your path variable
json.dump(b, codecs.open(file_path, 'w', encoding='utf-8'), separators=(',', ':'), sort_keys=True, indent=4) ### this saves the array in .json format

In order to “unjsonify” the array use:

obj_text = codecs.open(file_path, 'r', encoding='utf-8').read()
b_new = json.loads(obj_text)
a_new = np.array(b_new)

回答 1

将numpy.ndarray或任何嵌套列表组合作为JSON存储。

class NumpyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)

a = np.array([[1, 2, 3], [4, 5, 6]])
print(a.shape)
json_dump = json.dumps({'a': a, 'aa': [2, (2, 3, 4), a], 'bb': [2]}, cls=NumpyEncoder)
print(json_dump)

将输出:

(2, 3)
{"a": [[1, 2, 3], [4, 5, 6]], "aa": [2, [2, 3, 4], [[1, 2, 3], [4, 5, 6]]], "bb": [2]}

要从JSON还原:

json_load = json.loads(json_dump)
a_restored = np.asarray(json_load["a"])
print(a_restored)
print(a_restored.shape)

将输出:

[[1 2 3]
 [4 5 6]]
(2, 3)

Store as JSON a numpy.ndarray or any nested-list composition.

class NumpyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)

a = np.array([[1, 2, 3], [4, 5, 6]])
print(a.shape)
json_dump = json.dumps({'a': a, 'aa': [2, (2, 3, 4), a], 'bb': [2]}, cls=NumpyEncoder)
print(json_dump)

Will output:

(2, 3)
{"a": [[1, 2, 3], [4, 5, 6]], "aa": [2, [2, 3, 4], [[1, 2, 3], [4, 5, 6]]], "bb": [2]}

To restore from JSON:

json_load = json.loads(json_dump)
a_restored = np.asarray(json_load["a"])
print(a_restored)
print(a_restored.shape)

Will output:

[[1 2 3]
 [4 5 6]]
(2, 3)

回答 2

您可以使用Pandas

import pandas as pd
pd.Series(your_array).to_json(orient='values')

You can use Pandas:

import pandas as pd
pd.Series(your_array).to_json(orient='values')

回答 3

如果您在字典中嵌套了numpy数组,我找到了最佳解决方案:

import json
import numpy as np

class NumpyEncoder(json.JSONEncoder):
    """ Special json encoder for numpy types """
    def default(self, obj):
        if isinstance(obj, np.integer):
            return int(obj)
        elif isinstance(obj, np.floating):
            return float(obj)
        elif isinstance(obj, np.ndarray):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)

dumped = json.dumps(data, cls=NumpyEncoder)

with open(path, 'w') as f:
    json.dump(dumped, f)

感谢这个家伙

I found the best solution if you have nested numpy arrays in a dictionary:

import json
import numpy as np

class NumpyEncoder(json.JSONEncoder):
    """ Special json encoder for numpy types """
    def default(self, obj):
        if isinstance(obj, np.integer):
            return int(obj)
        elif isinstance(obj, np.floating):
            return float(obj)
        elif isinstance(obj, np.ndarray):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)

dumped = json.dumps(data, cls=NumpyEncoder)

with open(path, 'w') as f:
    json.dump(dumped, f)

Thanks to this guy.


回答 4

使用json.dumps defaultkwarg:

default应该是一个为无法序列化的对象调用的函数。

default函数中,检查对象是否来自numpy模块,如果是,则将其ndarray.tolist用于ndarray或将其.item用于任何其他特定于numpy的类型。

import numpy as np

def default(obj):
    if type(obj).__module__ == np.__name__:
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        else:
            return obj.item()
    raise TypeError('Unknown type:', type(obj))

dumped = json.dumps(data, default=default)

Use the json.dumps default kwarg:

default should be a function that gets called for objects that can’t otherwise be serialized. … or raise a TypeError

In the default function check if the object is from the module numpy, if so either use ndarray.tolist for a ndarray or use .item for any other numpy specific type.

import numpy as np

def default(obj):
    if type(obj).__module__ == np.__name__:
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        else:
            return obj.item()
    raise TypeError('Unknown type:', type(obj))

dumped = json.dumps(data, default=default)

回答 5

默认情况下不支持此功能,但是您可以使其轻松工作!如果您想返回完全相同的数据,则需要对几件事进行编码:

  • 数据本身,您可以获得 obj.tolist() @travelingbones。有时这可能已经足够了。
  • 数据类型。我觉得在某些情况下这很重要。
  • 如果您假设输入确实始终是“矩形”网格,则可以从上面得出尺寸(不一定是2D)。
  • 内存顺序(行或列为主)。这通常并不重要,但有时却很重要(例如性能),那么为什么不保存所有内容呢?

此外,您的numpy数组可能是数据结构的一部分,例如,您有一个包含一些矩阵的列表。为此,您可以使用基本上完成上述操作的自定义编码器。

这应该足以实施解决方案。或者,您可以使用json-tricks来做到这一点(并支持其他各种类型)(免责声明:我做到了)。

pip install json-tricks

然后

data = [
    arange(0, 10, 1, dtype=int).reshape((2, 5)),
    datetime(year=2017, month=1, day=19, hour=23, minute=00, second=00),
    1 + 2j,
    Decimal(42),
    Fraction(1, 3),
    MyTestCls(s='ub', dct={'7': 7}),  # see later
    set(range(7)),
]
# Encode with metadata to preserve types when decoding
print(dumps(data))

This is not supported by default, but you can make it work quite easily! There are several things you’ll want to encode if you want the exact same data back:

  • The data itself, which you can get with obj.tolist() as @travelingbones mentioned. Sometimes this may be good enough.
  • The data type. I feel this is important in quite some cases.
  • The dimension (not necessarily 2D), which could be derived from the above if you assume the input is indeed always a ‘rectangular’ grid.
  • The memory order (row- or column-major). This doesn’t often matter, but sometimes it does (e.g. performance), so why not save everything?

Furthermore, your numpy array could part of your data structure, e.g. you have a list with some matrices inside. For that you could use a custom encoder which basically does the above.

This should be enough to implement a solution. Or you could use json-tricks which does just this (and supports various other types) (disclaimer: I made it).

pip install json-tricks

Then

data = [
    arange(0, 10, 1, dtype=int).reshape((2, 5)),
    datetime(year=2017, month=1, day=19, hour=23, minute=00, second=00),
    1 + 2j,
    Decimal(42),
    Fraction(1, 3),
    MyTestCls(s='ub', dct={'7': 7}),  # see later
    set(range(7)),
]
# Encode with metadata to preserve types when decoding
print(dumps(data))

回答 6

嵌套字典中有一些numpy.ndarrays,我也遇到类似的问题。

def jsonify(data):
    json_data = dict()
    for key, value in data.iteritems():
        if isinstance(value, list): # for lists
            value = [ jsonify(item) if isinstance(item, dict) else item for item in value ]
        if isinstance(value, dict): # for nested lists
            value = jsonify(value)
        if isinstance(key, int): # if key is integer: > to string
            key = str(key)
        if type(value).__module__=='numpy': # if value is numpy.*: > to python list
            value = value.tolist()
        json_data[key] = value
    return json_data

I had a similar problem with a nested dictionary with some numpy.ndarrays in it.

def jsonify(data):
    json_data = dict()
    for key, value in data.iteritems():
        if isinstance(value, list): # for lists
            value = [ jsonify(item) if isinstance(item, dict) else item for item in value ]
        if isinstance(value, dict): # for nested lists
            value = jsonify(value)
        if isinstance(key, int): # if key is integer: > to string
            key = str(key)
        if type(value).__module__=='numpy': # if value is numpy.*: > to python list
            value = value.tolist()
        json_data[key] = value
    return json_data

回答 7

您还可以使用default参数例如:

def myconverter(o):
    if isinstance(o, np.float32):
        return float(o)

json.dump(data, default=myconverter)

You could also use default argument for example:

def myconverter(o):
    if isinstance(o, np.float32):
        return float(o)

json.dump(data, default=myconverter)

回答 8

另外,关于Python中的列表和数组,还有一些非常有趣的信息〜> Python中的列表与数组的 Python列表与数组-何时使用?

可以注意到,在将数组保存到JSON文件之前将其转换为列表之后,无论如何,现在无论如何在我的部署中,一旦读取该JSON文件以备后用,我就可以继续以列表形式使用它(如而不是将其转换回数组)。

这样,与屏幕上的列表(逗号分隔)和数组(非逗号分隔)相比,实际上它看起来更好(在我看来)。

使用上面的@travelingbones的.tolist()方法,我已经这样使用了(也发现了一些我发现的错误):

保存词典

def writeDict(values, name):
    writeName = DIR+name+'.json'
    with open(writeName, "w") as outfile:
        json.dump(values, outfile)

阅读词典

def readDict(name):
    readName = DIR+name+'.json'
    try:
        with open(readName, "r") as infile:
            dictValues = json.load(infile)
            return(dictValues)
    except IOError as e:
        print(e)
        return('None')
    except ValueError as e:
        print(e)
        return('None')

希望这可以帮助!

Also, some very interesting information further on lists vs. arrays in Python ~> Python List vs. Array – when to use?

It could be noted that once I convert my arrays into a list before saving it in a JSON file, in my deployment right now anyways, once I read that JSON file for use later, I can continue to use it in a list form (as opposed to converting it back to an array).

AND actually looks nicer (in my opinion) on the screen as a list (comma seperated) vs. an array (not-comma seperated) this way.

Using @travelingbones’s .tolist() method above, I’ve been using as such (catching a few errors I’ve found too):

SAVE DICTIONARY

def writeDict(values, name):
    writeName = DIR+name+'.json'
    with open(writeName, "w") as outfile:
        json.dump(values, outfile)

READ DICTIONARY

def readDict(name):
    readName = DIR+name+'.json'
    try:
        with open(readName, "r") as infile:
            dictValues = json.load(infile)
            return(dictValues)
    except IOError as e:
        print(e)
        return('None')
    except ValueError as e:
        print(e)
        return('None')

Hope this helps!


回答 9

这是一个对我有用的实现,并删除了所有nan(假设它们是简单的对象(列表或字典)):

from numpy import isnan

def remove_nans(my_obj, val=None):
    if isinstance(my_obj, list):
        for i, item in enumerate(my_obj):
            if isinstance(item, list) or isinstance(item, dict):
                my_obj[i] = remove_nans(my_obj[i], val=val)

            else:
                try:
                    if isnan(item):
                        my_obj[i] = val
                except Exception:
                    pass

    elif isinstance(my_obj, dict):
        for key, item in my_obj.iteritems():
            if isinstance(item, list) or isinstance(item, dict):
                my_obj[key] = remove_nans(my_obj[key], val=val)

            else:
                try:
                    if isnan(item):
                        my_obj[key] = val
                except Exception:
                    pass

    return my_obj

Here is an implementation that work for me and removed all nans (assuming these are simple object (list or dict)):

from numpy import isnan

def remove_nans(my_obj, val=None):
    if isinstance(my_obj, list):
        for i, item in enumerate(my_obj):
            if isinstance(item, list) or isinstance(item, dict):
                my_obj[i] = remove_nans(my_obj[i], val=val)

            else:
                try:
                    if isnan(item):
                        my_obj[i] = val
                except Exception:
                    pass

    elif isinstance(my_obj, dict):
        for key, item in my_obj.iteritems():
            if isinstance(item, list) or isinstance(item, dict):
                my_obj[key] = remove_nans(my_obj[key], val=val)

            else:
                try:
                    if isnan(item):
                        my_obj[key] = val
                except Exception:
                    pass

    return my_obj

回答 10

这是一个不同的答案,但这可能有助于帮助试图保存数据然后再次读取的人们。
有一个比泡菜快和容易的hi。
我试图保存并在泡菜转储中阅读它,但是阅读时有很多问题,浪费了一个小时,尽管我正在处理自己的数据以创建聊天机器人,但仍然找不到解决方案。

vec_x并且vec_y是numpy数组:

data=[vec_x,vec_y]
hkl.dump( data, 'new_data_file.hkl' )

然后,您只需阅读并执行以下操作:

data2 = hkl.load( 'new_data_file.hkl' )

This is a different answer, but this might help to help people who are trying to save data and then read it again.
There is hickle which is faster than pickle and easier.
I tried to save and read it in pickle dump but while reading there were lot of problems and wasted an hour and still didn’t find solution though I was working on my own data to create a chat bot.

vec_x and vec_y are numpy arrays:

data=[vec_x,vec_y]
hkl.dump( data, 'new_data_file.hkl' )

Then you just read it and perform the operations:

data2 = hkl.load( 'new_data_file.hkl' )

回答 11

可以使用检查类型来简化循环:

with open("jsondontdoit.json", 'w') as fp:
    for key in bests.keys():
        if type(bests[key]) == np.ndarray:
            bests[key] = bests[key].tolist()
            continue
        for idx in bests[key]:
            if type(bests[key][idx]) == np.ndarray:
                bests[key][idx] = bests[key][idx].tolist()
    json.dump(bests, fp)
    fp.close()

May do simple for loop with checking types:

with open("jsondontdoit.json", 'w') as fp:
    for key in bests.keys():
        if type(bests[key]) == np.ndarray:
            bests[key] = bests[key].tolist()
            continue
        for idx in bests[key]:
            if type(bests[key][idx]) == np.ndarray:
                bests[key][idx] = bests[key][idx].tolist()
    json.dump(bests, fp)
    fp.close()

回答 12

使用NumpyEncoder它将成功处理json转储。不抛出-NumPy数组不是JSON可序列化的

import numpy as np
import json
from numpyencoder import NumpyEncoder
arr = array([   0,  239,  479,  717,  952, 1192, 1432, 1667], dtype=int64) 
json.dumps(arr,cls=NumpyEncoder)

use NumpyEncoder it will process json dump successfully.without throwing – NumPy array is not JSON serializable

import numpy as np
import json
from numpyencoder import NumpyEncoder
arr = array([   0,  239,  479,  717,  952, 1192, 1432, 1667], dtype=int64) 
json.dumps(arr,cls=NumpyEncoder)

回答 13

TypeError:array([[0.46872085,0.67374235,1.0218339,0.13210179,0.5440686,0.9140083,0.58720225,0.2199381]],dtype = float32)不是JSON可序列化的

当我期望以json格式响应时,尝试将数据列表传递给model.predict()时,抛出了上述错误。

> 1        json_file = open('model.json','r')
> 2        loaded_model_json = json_file.read()
> 3        json_file.close()
> 4        loaded_model = model_from_json(loaded_model_json)
> 5        #load weights into new model
> 6        loaded_model.load_weights("model.h5")
> 7        loaded_model.compile(optimizer='adam', loss='mean_squared_error')
> 8        X =  [[874,12450,678,0.922500,0.113569]]
> 9        d = pd.DataFrame(X)
> 10       prediction = loaded_model.predict(d)
> 11       return jsonify(prediction)

但幸运的是找到了解决抛出错误的提示对象的序列化仅适用于以下转换映射应采用以下方式object-dict array-list string-string integer-integer

如果您向上滚动以查看第10行的代码,则这行代码将生成array数据类型的输出,当您尝试将array转换为json格式时,这是不可能的。

最终我找到了解决方案,只需通过遵循以下几行代码将获得的输出转换为类型列表即可

预测=加载模型。预测(d)
列表类型=预测。列表()返回jsonify(列表类型)

hoo!终于得到了预期的输出,

TypeError: array([[0.46872085, 0.67374235, 1.0218339 , 0.13210179, 0.5440686 , 0.9140083 , 0.58720225, 0.2199381 ]], dtype=float32) is not JSON serializable

The above-mentioned error was thrown when i tried to pass of list of data to model.predict() when i was expecting the response in json format.

> 1        json_file = open('model.json','r')
> 2        loaded_model_json = json_file.read()
> 3        json_file.close()
> 4        loaded_model = model_from_json(loaded_model_json)
> 5        #load weights into new model
> 6        loaded_model.load_weights("model.h5")
> 7        loaded_model.compile(optimizer='adam', loss='mean_squared_error')
> 8        X =  [[874,12450,678,0.922500,0.113569]]
> 9        d = pd.DataFrame(X)
> 10       prediction = loaded_model.predict(d)
> 11       return jsonify(prediction)

But luckily found the hint to resolve the error that was throwing The serializing of the objects is applicable only for the following conversion Mapping should be in following way object – dict array – list string – string integer – integer

If you scroll up to see the line number 10 prediction = loaded_model.predict(d) where this line of code was generating the output of type array datatype , when you try to convert array to json format its not possible

Finally i found the solution just by converting obtained output to the type list by following lines of code

prediction = loaded_model.predict(d)
listtype = prediction.tolist() return jsonify(listtype)

Bhoom! finally got the expected output,