DataFrame中的字符串,但dtype是object

问题:DataFrame中的字符串,但dtype是object

为什么Pandas告诉我我有对象,尽管所选列中的每个项目都是一个字符串-即使经过显式转换也是如此。

这是我的DataFrame:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 56992 entries, 0 to 56991
Data columns (total 7 columns):
id            56992  non-null values
attr1         56992  non-null values
attr2         56992  non-null values
attr3         56992  non-null values
attr4         56992  non-null values
attr5         56992  non-null values
attr6         56992  non-null values
dtypes: int64(2), object(5)

他们五个dtype object。我将这些对象明确转换为字符串:

for c in df.columns:
    if df[c].dtype == object:
        print "convert ", df[c].name, " to string"
        df[c] = df[c].astype(str)

然后,尽管显示,df["attr2"]仍然是正确的。dtype objecttype(df["attr2"].ix[0]str

熊猫区分int64float64object。没有时背后的逻辑是什么dtype str?为什么被str覆盖object

Why does Pandas tell me that I have objects, although every item in the selected column is a string — even after explicit conversion.

This is my DataFrame:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 56992 entries, 0 to 56991
Data columns (total 7 columns):
id            56992  non-null values
attr1         56992  non-null values
attr2         56992  non-null values
attr3         56992  non-null values
attr4         56992  non-null values
attr5         56992  non-null values
attr6         56992  non-null values
dtypes: int64(2), object(5)

Five of them are dtype object. I explicitly convert those objects to strings:

for c in df.columns:
    if df[c].dtype == object:
        print "convert ", df[c].name, " to string"
        df[c] = df[c].astype(str)

Then, df["attr2"] still has dtype object, although type(df["attr2"].ix[0] reveals str, which is correct.

Pandas distinguishes between int64 and float64 and object. What is the logic behind it when there is no dtype str? Why is a str covered by object?


回答 0

dtype对象来自NumPy,它描述ndarray中元素的类型。ndarray中的每个元素都必须具有相同的字节大小。对于int64和float64,它们是8个字节。但是对于字符串,字符串的长度不是固定的。因此,熊猫没有直接将字符串的字节保存在ndarray中,而是使用对象ndarray来保存指向对象的指针,因此,这种ndarray的dtype是object。

这是一个例子:

  • int64数组包含4个int64值。
  • 对象数组包含4个指向3个字符串对象的指针。

The dtype object comes from NumPy, it describes the type of element in a ndarray. Every element in a ndarray must has the same size in byte. For int64 and float64, they are 8 bytes. But for strings, the length of the string is not fixed. So instead of save the bytes of strings in the ndarray directly, Pandas use object ndarray, which save pointers to objects, because of this the dtype of this kind ndarray is object.

Here is an example:

  • the int64 array contains 4 int64 value.
  • the object array contains 4 pointers to 3 string objects.


回答 1

接受的答案是好的。只是想提供一个参考文档的答案。该文档说:

熊猫使用对象dtype来存储字符串。

正如主要评论所说:“不用担心;它应该像这样。” (尽管可接受的答案在解释“为什么”方面做得很好,字符串是可变长度的)

但是对于字符串,字符串的长度不是固定的。

The accepted answer is good. Just wanted to provide an answer which referenced the documentation. The documentation says:

Pandas uses the object dtype for storing strings.

As the leading comment says “Don’t worry about it; it’s supposed to be like this.” (Although the accepted answer did a great job explaining the “why”; strings are variable-length)

But for strings, the length of the string is not fixed.


回答 2

@HYRY的答案很好。我只想提供更多背景信息。

阵列存储的数据作为连续的固定大小的存储器块。这些属性的结合使阵列可以快速进行数据访问。例如,考虑您的计算机可能如何存储32位整数数组[3,0,1]

如果您要求计算机获取数组中的第3个元素,它将从头开始,然后跨64位跳转到第3个元素。确切知道要跳过多少位才可以使数组快速运行

现在考虑字符串的顺序['hello', 'i', 'am', 'a', 'banana']。字符串是大小不同的对象,因此,如果您尝试将它们存储在连续的内存块中,它将最终看起来像这样。

现在,您的计算机没有快速的方法来访问随机请求的元素。克服这个问题的关键是使用指针。基本上,将每个字符串存储在某个随机的内存位置,然后用每个字符串的内存地址填充数组。(内存地址只是整数。)所以现在,事情看起来像这样

现在,如果您像以前一样要求计算机获取第三个元素,它可以跨64位跳转(假设内存地址是32位整数),然后再执行一个步骤来获取字符串。

NumPy面临的挑战是不能保证指针实际上指向字符串。这就是为什么它将dtype报告为“对象”的原因。

无耻地插入我自己的博客文章,最初是在此进行讨论的。

@HYRY’s answer is great. I just want to provide a little more context..

Arrays store data as contiguous, fixed-size memory blocks. The combination of these properties together is what makes arrays lightning fast for data access. For example, consider how your computer might store an array of 32-bit integers, [3,0,1].

If you ask your computer to fetch the 3rd element in the array, it’ll start at the beginning and then jump across 64 bits to get to the 3rd element. Knowing exactly how many bits to jump across is what makes arrays fast.

Now consider the sequence of strings ['hello', 'i', 'am', 'a', 'banana']. Strings are objects that vary in size, so if you tried to store them in contiguous memory blocks, it’d end up looking like this.

Now your computer doesn’t have a fast way to access a randomly requested element. The key to overcoming this is to use pointers. Basically, store each string in some random memory location, and fill the array with the memory address of each string. (Memory addresses are just integers.) So now, things look like this

Now, if you ask your computer to fetch the 3rd element, just as before, it can jump across 64 bits (assuming the memory addresses are 32-bit integers) and then make one extra step to go fetch the string.

The challenge for NumPy is that there’s no guarantee the pointers are actually pointing to strings. That’s why it reports the dtype as ‘object’.

Shamelessly gonna plug my own blog article where I originally discussed this.


回答 3

从1.0.0版开始(2020年1月),pandas作为实验功能被引入,它通过提供对字符串类型的一流支持pandas.StringDtype

虽然您仍然会object默认看到,但是可以通过指定dtypeof pd.StringDtype或简单地使用新类型'string'

>>> pd.Series(['abc', None, 'def'])
0     abc
1    None
2     def
dtype: object
>>> pd.Series(['abc', None, 'def'], dtype=pd.StringDtype())
0     abc
1    <NA>
2     def
dtype: string
>>> pd.Series(['abc', None, 'def']).astype('string')
0     abc
1    <NA>
2     def
dtype: string

As of version 1.0.0 (January 2020), pandas has introduced as an experimental feature providing first-class support for string types through pandas.StringDtype.

While you’ll still be seeing object by default, the new type can be used by specifying a dtype of pd.StringDtype or simply 'string':

>>> pd.Series(['abc', None, 'def'])
0     abc
1    None
2     def
dtype: object
>>> pd.Series(['abc', None, 'def'], dtype=pd.StringDtype())
0     abc
1    <NA>
2     def
dtype: string
>>> pd.Series(['abc', None, 'def']).astype('string')
0     abc
1    <NA>
2     def
dtype: string