来自os.listdir()的非字母数字列表顺序

问题:来自os.listdir()的非字母数字列表顺序

我经常使用python处理数据目录。最近,我注意到列表的默认顺序已更改为几乎毫无意义的内容。例如,如果我位于包含以下子目录的当前目录中:run01,run02,…,run19,run20,然后从以下命令生成列表:

dir = os.listdir(os.getcwd())

然后我通常会按以下顺序获得列表:

dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08', ... ]

等等。该顺序曾经是字母数字。但是这个新订单已经存在了一段时间。

是什么决定这些列表的(显示)顺序?

I often use python to process directories of data. Recently, I have noticed that the default order of the lists has changed to something almost nonsensical. For example, if I am in a current directory containing the following subdirectories: run01, run02, … run19, run20, and then I generate a list from the following command:

dir = os.listdir(os.getcwd())

then I usually get a list in this order:

dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08', ... ]

and so on. The order used to be alphanumeric. But this new order has remained with me for a while now.

What is determining the (displayed) order of these lists?


回答 0

我认为顺序与文件在FileSystem上建立索引的方式有关。如果您确实要使其遵循某些顺序,则可以在获取文件后始终对列表进行排序。

I think the order has to do with the way the files are indexed on your FileSystem. If you really want to make it adhere to some order you can always sort the list after getting the files.


回答 1

您可以使用内置sorted函数对字符串进行任意排序。根据您的描述,

sorted(os.listdir(whatever_directory))

或者,您可以使用.sort列表的方法:

lst = os.listdir(whatever_directory)
lst.sort()

我认为应该可以解决问题。

请注意,os.listdir获取文件名的顺序可能完全取决于您的文件系统。

You can use the builtin sorted function to sort the strings however you want. Based on what you describe,

sorted(os.listdir(whatever_directory))

Alternatively, you can use the .sort method of a list:

lst = os.listdir(whatever_directory)
lst.sort()

I think should do the trick.

Note that the order that os.listdir gets the filenames is probably completely dependent on your filesystem.


回答 2

根据文档

os.listdir(路径)

返回一个列表,其中包含由path给出的目录中条目的名称。该列表按任意顺序排列。它不包括特殊条目“。” 和“ ..”,即使它们存在于目录中。

不能依赖顺序,它是文件系统的产物。

要对结果进行排序,请使用sorted(os.listdir(path))

Per the documentation:

os.listdir(path)

Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

Order cannot be relied upon and is an artifact of the filesystem.

To sort the result, use sorted(os.listdir(path)).


回答 3

不管出于什么原因,Python都没有内置的方法来进行自然排序(意味着1、2、10而不是1、10、2),因此您必须自己编写:

import re
def sorted_alphanumeric(data):
    convert = lambda text: int(text) if text.isdigit() else text.lower()
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ] 
    return sorted(data, key=alphanum_key)

现在,您可以使用此功能对列表进行排序:

dirlist = sorted_alphanumeric(os.listdir(...))

问题: 如果您使用上述函数对字符串(例如文件夹名称)进行排序,并希望像Windows资源管理器一样对它们进行排序,则在某些情况下无法正常使用。
如果您的文件夹名称中带有某些“特殊”字符,则此排序功能将在Windows上返回不正确的结果。例如,此函数将排序1, !1, !a, a,而Windows资源管理器将排序!1, 1, !a, a

因此,如果您想像Python中的Windows资源管理器那样进行排序,则必须通过ctypes 使用Windows内置函数StrCmpLogicalW(这当然在Unix上不起作用):

from ctypes import wintypes, windll
from functools import cmp_to_key
def winsort(data):
    _StrCmpLogicalW = windll.Shlwapi.StrCmpLogicalW
    _StrCmpLogicalW.argtypes = [wintypes.LPWSTR, wintypes.LPWSTR]
    _StrCmpLogicalW.restype  = wintypes.INT

    cmp_fnc = lambda psz1, psz2: _StrCmpLogicalW(psz1, psz2)
    return sorted(data, key=cmp_to_key(cmp_fnc))

此功能比稍慢sorted_alphanumeric()

奖励:winsort还可以在Windows上对完整路径进行排序

另外,尤其是在使用Unix的情况下,可以使用natsort库(pip install natsort)以正确的方式对完整路径进行排序(意味着子文件夹位于正确的位置)。

您可以像这样使用它来排序完整路径:

from natsort import natsorted, ns
dirlist = natsorted(dirlist, alg=ns.PATH | ns.IGNORECASE)

不要将其用于仅对文件夹名称(或通常为字符串)进行常规排序,因为它比sorted_alphanumeric()上面的函数要慢很多。如果您期望Windows资源管理器排序,该
natsorted库将给您不正确的结果,因此可以使用winsort()它。

Python for whatever reason does not come with a built-in way to have natural sorting (meaning 1, 2, 10 instead of 1, 10, 2), so you have to write it yourself:

import re
def sorted_alphanumeric(data):
    convert = lambda text: int(text) if text.isdigit() else text.lower()
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ] 
    return sorted(data, key=alphanum_key)

You can now use this function to sort a list:

dirlist = sorted_alphanumeric(os.listdir(...))

PROBLEMS: In case you use the above function to sort strings (for example folder names) and want them sorted like Windows Explorer does, it will not work properly in some edge cases.
This sorting function will return incorrect results on Windows, if you have folder names with certain ‘special’ characters in them. For example this function will sort 1, !1, !a, a, whereas Windows Explorer would sort !1, 1, !a, a.

So if you want to sort exactly like Windows Explorer does in Python you have to use the Windows built-in function StrCmpLogicalW via ctypes (this of course won’t work on Unix):

from ctypes import wintypes, windll
from functools import cmp_to_key
def winsort(data):
    _StrCmpLogicalW = windll.Shlwapi.StrCmpLogicalW
    _StrCmpLogicalW.argtypes = [wintypes.LPWSTR, wintypes.LPWSTR]
    _StrCmpLogicalW.restype  = wintypes.INT

    cmp_fnc = lambda psz1, psz2: _StrCmpLogicalW(psz1, psz2)
    return sorted(data, key=cmp_to_key(cmp_fnc))

This function is slightly slower than sorted_alphanumeric().

Bonus: winsort can also sort full paths on Windows.

Alternatively, especially if you use Unix, you can use the natsort library (pip install natsort) to sort by full paths in a correct way (meaning subfolders at the correct position).

You can use it like this to sort full paths:

from natsort import natsorted, ns
dirlist = natsorted(dirlist, alg=ns.PATH | ns.IGNORECASE)

Don’t use it for normal sorting of just folder names (or strings in general), as it’s quite a bit slower than then sorted_alphanumeric() function above.
natsorted library will give you incorrect results if you expect Windows Explorer sorting, so use winsort() for that.


回答 4

我认为默认情况下,顺序由ASCII值确定。这个问题的解决方案是这样

dir = sorted(os.listdir(os.getcwd()), key=len)

I think by default the order is determined with the ASCII value. The solution to this problem is this

dir = sorted(os.listdir(os.getcwd()), key=len)

回答 5

这可能只是C的readdir()返回顺序。尝试运行此C程序:

#include <dirent.h>
#include <stdio.h>
int main(void)
{   DIR *dirp;
    struct dirent* de;
    dirp = opendir(".");
    while(de = readdir(dirp)) // Yes, one '='.
        printf("%s\n", de->d_name);
    closedir(dirp);
    return 0;
}

构建线应类似于gcc -o foo foo.c

PS只需运行此代码和您的Python代码,它们都给了我排序的输出,所以我无法重现您看到的内容。

It’s probably just the order that C’s readdir() returns. Try running this C program:

#include <dirent.h>
#include <stdio.h>
int main(void)
{   DIR *dirp;
    struct dirent* de;
    dirp = opendir(".");
    while(de = readdir(dirp)) // Yes, one '='.
        printf("%s\n", de->d_name);
    closedir(dirp);
    return 0;
}

The build line should be something like gcc -o foo foo.c.

P.S. Just ran this and your Python code, and they both gave me sorted output, so I can’t reproduce what you’re seeing.


回答 6

aaa = ['row_163.pkl', 'row_394.pkl', 'row_679.pkl', 'row_202.pkl', 'row_1449.pkl', 'row_247.pkl', 'row_1353.pkl', 'row_749.pkl', 'row_1293.pkl', 'row_1304.pkl', 'row_78.pkl', 'row_532.pkl', 'row_9.pkl', 'row_1435.pkl']                                                                                                                                                                                                                                                                                                 
sorted(aaa, key=lambda x: int(os.path.splitext(x.split('_')[1])[0]))

由于在案件的矿山要求我有这样的情况row_163.pkl在这里os.path.splitext('row_163.pkl')将它分成('row_163', '.pkl')所以需要根据“_”也把它分解。

但如果您有需要,您可以做类似的事情

sorted(aa, key = lambda x: (int(re.sub('\D','',x)),x))

哪里

aa = ['run01', 'run08', 'run11', 'run12', 'run13', 'run14', 'run18']

对于目录检索,您也可以 sorted(os.listdir(path))

对于like 'run01.txt''run01.csv'您可以这样做

sorted(files, key=lambda x : int(os.path.splitext(x)[0]))
aaa = ['row_163.pkl', 'row_394.pkl', 'row_679.pkl', 'row_202.pkl', 'row_1449.pkl', 'row_247.pkl', 'row_1353.pkl', 'row_749.pkl', 'row_1293.pkl', 'row_1304.pkl', 'row_78.pkl', 'row_532.pkl', 'row_9.pkl', 'row_1435.pkl']                                                                                                                                                                                                                                                                                                 
sorted(aaa, key=lambda x: int(os.path.splitext(x.split('_')[1])[0]))

As In case of mine requirement I have the case like row_163.pkl here os.path.splitext('row_163.pkl') will break it into ('row_163', '.pkl') so need to split it based on ‘_’ also.

but in case of your requirement you can do something like

sorted(aa, key = lambda x: (int(re.sub('\D','',x)),x))

where

aa = ['run01', 'run08', 'run11', 'run12', 'run13', 'run14', 'run18']

and also for directory retrieving you can do sorted(os.listdir(path))

and for the case of like 'run01.txt' or 'run01.csv' you can do like this

sorted(files, key=lambda x : int(os.path.splitext(x)[0]))

回答 7

我发现“排序”并不总是按预期进行。例如,我有一个如下目录,“ sort”给我一个非常奇怪的结果:

>>> os.listdir(pathon)
['2', '3', '4', '5', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472']
>>> sorted([ f for f in os.listdir(pathon)])
['2', '3', '4', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472', '5']

看起来它首先比较第一个字符,如果最大,那就是最后一个。

I found “sort” does not always do what I expected. eg, I have a directory as below, and the “sort” give me a very strange result:

>>> os.listdir(pathon)
['2', '3', '4', '5', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472']
>>> sorted([ f for f in os.listdir(pathon)])
['2', '3', '4', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472', '5']

It seems it compares the first character first, if that is the biggest, it would be the last one.


回答 8

文档中

该列表以任意顺序排列,并且不包括特殊条目“。”。和“ ..”,即使它们存在于目录中。

这意味着该顺序可能与OS /文件系统相关,没有特别有意义的顺序,因此不能保证特定顺序。提到了很多答案:如果需要,可以对检索到的列表进行排序。

干杯:)

From the documentation:

The list is in arbitrary order, and does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

This means that the order is probably OS/filesystem dependent, has no particularly meaningful order, and is therefore not guaranteed to be anything in particular. As many answers mentioned: if preferred, the retrieved list can be sorted.

Cheers :)


回答 9

艾略特的答案可以很好地解决它,但是由于它是评论,因此没有引起注意,因此为了帮助某人,我在此重申它为解决方案。

使用natsort库:

使用以下命令为Ubuntu和其他Debian版本安装库

Python 2

sudo pip install natsort

Python 3

sudo pip3 install natsort

有关如何使用此库的详细信息,请参见此处

Elliot’s answer solves it perfectly but because it is a comment, it goes unnoticed so with the aim of helping someone, I am reiterating it as a solution.

Use natsort library:

Install the library with the following command for Ubuntu and other Debian versions

Python 2

sudo pip install natsort

Python 3

sudo pip3 install natsort

Details of how to use this library is found here


回答 10

In [6]: os.listdir?

Type:       builtin_function_or_method
String Form:<built-in function listdir>
Docstring:
listdir(path) -> list_of_strings
Return a list containing the names of the entries in the directory.
path: path of directory to list
The list is in **arbitrary order**.  It does not include the special
entries '.' and '..' even if they are present in the directory.
In [6]: os.listdir?

Type:       builtin_function_or_method
String Form:<built-in function listdir>
Docstring:
listdir(path) -> list_of_strings
Return a list containing the names of the entries in the directory.
path: path of directory to list
The list is in **arbitrary order**.  It does not include the special
entries '.' and '..' even if they are present in the directory.

回答 11

os.listdirsorted命令的建议组合产生的结果与Linux下的ls -l命令相同。以下示例验证了此假设:

user@user-PC:/tmp/test$ touch 3a 4a 5a b c d1 d2 d3 k l p0 p1 p3 q 410a 409a 408a 407a
user@user-PC:/tmp/test$ ls -l
total 0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 3a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 407a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 408a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 409a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 410a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 4a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 5a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 b
-rw-rw-r-- 1 user user 0 Feb  15 10:31 c
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d2
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 k
-rw-rw-r-- 1 user user 0 Feb  15 10:31 l
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 q

user@user-PC:/tmp/test$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir( './' )
['d3', 'k', 'p1', 'b', '410a', '5a', 'l', 'p0', '407a', '409a', '408a', 'd2', '4a', 'p3', '3a', 'q', 'c', 'd1']
>>> sorted( os.listdir( './' ) )
['3a', '407a', '408a', '409a', '410a', '4a', '5a', 'b', 'c', 'd1', 'd2', 'd3', 'k', 'l', 'p0', 'p1', 'p3', 'q']
>>> exit()
user@user-PC:/tmp/test$ 

因此,对于想要在其Python代码中重现著名的ls -l命令的结果的人来说,sorted(os.listdir(DIR))效果很好。

The proposed combination of os.listdir and sorted commands generates the same result as ls -l command under Linux. The following example verifies this assumption:

user@user-PC:/tmp/test$ touch 3a 4a 5a b c d1 d2 d3 k l p0 p1 p3 q 410a 409a 408a 407a
user@user-PC:/tmp/test$ ls -l
total 0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 3a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 407a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 408a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 409a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 410a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 4a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 5a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 b
-rw-rw-r-- 1 user user 0 Feb  15 10:31 c
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d2
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 k
-rw-rw-r-- 1 user user 0 Feb  15 10:31 l
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 q

user@user-PC:/tmp/test$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir( './' )
['d3', 'k', 'p1', 'b', '410a', '5a', 'l', 'p0', '407a', '409a', '408a', 'd2', '4a', 'p3', '3a', 'q', 'c', 'd1']
>>> sorted( os.listdir( './' ) )
['3a', '407a', '408a', '409a', '410a', '4a', '5a', 'b', 'c', 'd1', 'd2', 'd3', 'k', 'l', 'p0', 'p1', 'p3', 'q']
>>> exit()
user@user-PC:/tmp/test$ 

So, for someone who wants to reproduce the result of the well-known ls -l command in their python code, sorted( os.listdir( DIR ) ) works pretty well.