问题:将字符串转换为有效的文件名?
我有一个要用作文件名的字符串,因此我想使用Python删除文件名中不允许的所有字符。
我宁愿严格一点,所以假设我只保留字母,数字和一小部分其他字符,例如"_-.() "
。什么是最优雅的解决方案?
文件名在多个操作系统(Windows,Linux和Mac OS)上必须有效-这是我库中的MP3文件,歌曲名作为文件名,并且在3台计算机之间共享和备份。
I have a string that I want to use as a filename, so I want to remove all characters that wouldn’t be allowed in filenames, using Python.
I’d rather be strict than otherwise, so let’s say I want to retain only letters, digits, and a small set of other characters like "_-.() "
. What’s the most elegant solution?
The filename needs to be valid on multiple operating systems (Windows, Linux and Mac OS) – it’s an MP3 file in my library with the song title as the filename, and is shared and backed up between 3 machines.
回答 0
您可以查看Django框架,了解它们如何从任意文本创建“子弹”。slug是URL和文件名友好的。
Django文本工具定义了一个函数,slugify()
这可能是此类事情的黄金标准。本质上,它们的代码如下。
def slugify(value):
"""
Normalizes string, converts to lowercase, removes non-alpha characters,
and converts spaces to hyphens.
"""
import unicodedata
value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')
value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
value = unicode(re.sub('[-\s]+', '-', value))
# ...
return value
还有更多,但我不予赘述,因为它不解决节段化,而是逃脱。
You can look at the Django framework for how they create a “slug” from arbitrary text. A slug is URL- and filename- friendly.
The Django text utils define a function, slugify()
, that’s probably the gold standard for this kind of thing. Essentially, their code is the following.
def slugify(value):
"""
Normalizes string, converts to lowercase, removes non-alpha characters,
and converts spaces to hyphens.
"""
import unicodedata
value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')
value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
value = unicode(re.sub('[-\s]+', '-', value))
# ...
return value
There’s more, but I left it out, since it doesn’t address slugification, but escaping.
回答 1
如果对文件格式或非法非法字符(例如“ ..”)的组合没有限制,则这种白名单方法(即仅允许有效字符中存在的字符)将起作用,例如,您说的是将允许一个名为“ .txt”的文件名,我认为在Windows上无效。由于这是最简单的方法,因此我会尝试从valid_chars中删除空格,并在出现错误的情况下添加一个已知的有效字符串,因此,任何其他方法都必须知道允许在何处应对Windows文件命名限制,因此复杂得多。
>>> import string
>>> valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
>>> valid_chars
'-_.() abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
>>> filename = "This Is a (valid) - filename%$&$ .txt"
>>> ''.join(c for c in filename if c in valid_chars)
'This Is a (valid) - filename .txt'
This whitelist approach (ie, allowing only the chars present in valid_chars) will work if there aren’t limits on the formatting of the files or combination of valid chars that are illegal (like “..”), for example, what you say would allow a filename named ” . txt” which I think is not valid on Windows. As this is the most simple approach I’d try to remove whitespace from the valid_chars and prepend a known valid string in case of error, any other approach will have to know about what is allowed where to cope with Windows file naming limitations and thus be a lot more complex.
>>> import string
>>> valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
>>> valid_chars
'-_.() abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
>>> filename = "This Is a (valid) - filename%$&$ .txt"
>>> ''.join(c for c in filename if c in valid_chars)
'This Is a (valid) - filename .txt'
回答 2
您可以将列表理解与字符串方法一起使用。
>>> s
'foo-bar#baz?qux@127/\\9]'
>>> "".join(x for x in s if x.isalnum())
'foobarbazqux1279'
You can use list comprehension together with the string methods.
>>> s
'foo-bar#baz?qux@127/\\9]'
>>> "".join(x for x in s if x.isalnum())
'foobarbazqux1279'
回答 3
使用字符串作为文件名的原因是什么?如果不是人类可读性的因素,我将使用base64模块,该模块可以生成文件系统安全的字符串。它不是可读的,但您不必处理碰撞并且它是可逆的。
import base64
file_name_string = base64.urlsafe_b64encode(your_string)
更新:根据马修评论更改。
What is the reason to use the strings as file names? If human readability is not a factor I would go with base64 module which can produce file system safe strings. It won’t be readable but you won’t have to deal with collisions and it is reversible.
import base64
file_name_string = base64.urlsafe_b64encode(your_string)
Update: Changed based on Matthew comment.
回答 4
只是为了使事情更加复杂,不能保证仅通过删除无效字符就可以获得有效的文件名。由于不同文件名上允许的字符不同,因此保守的方法可能最终将有效名称变成无效名称。对于以下情况,您可能需要添加特殊处理:
该字符串是所有无效字符(留空字符串)
您最终得到一个具有特殊含义的字符串,例如“。”。要么 ”..”
在Windows上,某些设备名称被保留。例如,您无法创建名为“ nul”,“ nul.txt”(或实际上为nul.anything)的文件。保留名称为:
CON,PRN,AUX,NUL,COM1,COM2,COM3,COM4,COM5,COM6,COM7,COM8,COM9,LPT1,LPT2,LPT3,LPT4,LPT5,LPT6,LPT7,LPT8和LPT9
您可以通过在文件名前添加一些字符串(它们永远不会导致这些情况之一)并去除无效字符来解决这些问题。
Just to further complicate things, you are not guaranteed to get a valid filename just by removing invalid characters. Since allowed characters differ on different filenames, a conservative approach could end up turning a valid name into an invalid one. You may want to add special handling for the cases where:
The string is all invalid characters (leaving you with an empty string)
You end up with a string with a special meaning, eg “.” or “..”
On windows, certain device names are reserved. For instance, you can’t create a file named “nul”, “nul.txt” (or nul.anything in fact) The reserved names are:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9
You can probably work around these issues by prepending some string to the filenames that can never result in one of these cases, and stripping invalid characters.
回答 5
Github上有一个不错的项目python-slugify:
安装:
pip install python-slugify
然后使用:
>>> from slugify import slugify
>>> txt = "This\ is/ a%#$ test ---"
>>> slugify(txt)
'this-is-a-test'
There is a nice project on Github called python-slugify:
Install:
pip install python-slugify
Then use:
>>> from slugify import slugify
>>> txt = "This\ is/ a%#$ test ---"
>>> slugify(txt)
'this-is-a-test'
回答 6
回答 7
这是我最终使用的解决方案:
import unicodedata
validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
def removeDisallowedFilenameChars(filename):
cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
return ''.join(c for c in cleanedFilename if c in validFilenameChars)
unicodedata.normalize调用将重音字符替换为未重音的等效字符,这比简单地将它们剥离要好。之后,将删除所有不允许的字符。
我的解决方案没有在已知字符串前添加前缀,以避免可能出现的不允许的文件名,因为我知道在给定特定文件名格式的情况下它们不会出现。需要一个更通用的解决方案。
This is the solution I ultimately used:
import unicodedata
validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
def removeDisallowedFilenameChars(filename):
cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
return ''.join(c for c in cleanedFilename if c in validFilenameChars)
The unicodedata.normalize call replaces accented characters with the unaccented equivalent, which is better than simply stripping them out. After that all disallowed characters are removed.
My solution doesn’t prepend a known string to avoid possible disallowed filenames, because I know they can’t occur given my particular filename format. A more general solution would need to do so.
回答 8
请记住,除了Unix系统上的文件名外,实际上没有任何限制。
其他一切都是公平的游戏。
$ touch”
>甚至多行
>哈哈
> ^ [[31m红色^ [[0m
>邪恶”
$ ls -la
-rw-r--r-- Nov 11 23:39?even multiline?haha ?? [31m red?[0m?evil
$ ls -lab
-rw-r--r-- 11月17日23:39 \ neven \ multiline \ nhaha \ n \ 033 [31m \ red \ \ 033 [0m \ nevil
$ perl -e'for my $ i(glob(q {./* even *})){print $ i; }'
./
甚至多行
哈哈
红色
邪恶
是的,我只是将ANSI颜色代码存储在文件名中,并使它们生效。
为了娱乐,请将BEL字符放在目录名称中,并观看将CD放入其中时的乐趣;)
Keep in mind, there are actually no restrictions on filenames on Unix systems other than
- It may not contain \0
- It may not contain /
Everything else is fair game.
$ touch "
> even multiline
> haha
> ^[[31m red ^[[0m
> evil"
$ ls -la
-rw-r--r-- 0 Nov 17 23:39 ?even multiline?haha??[31m red ?[0m?evil
$ ls -lab
-rw-r--r-- 0 Nov 17 23:39 \neven\ multiline\nhaha\n\033[31m\ red\ \033[0m\nevil
$ perl -e 'for my $i ( glob(q{./*even*}) ){ print $i; } '
./
even multiline
haha
red
evil
Yes, i just stored ANSI Colour Codes in a file name and had them take effect.
For entertainment, put a BEL character in a directory name and watch the fun that ensues when you CD into it ;)
回答 9
一行:
valid_file_name = re.sub('[^\w_.)( -]', '', any_string)
您还可以添加’_’字符以使其更具可读性(例如,在替换斜杠的情况下)
In one line:
valid_file_name = re.sub('[^\w_.)( -]', '', any_string)
you can also put ‘_’ character to make it more readable (in case of replacing slashs, for example)
回答 10
您可以使用re.sub()方法替换非“类似文件”的任何内容。但实际上,每个字符都可以有效;因此,没有预构建的功能(我相信)可以完成它。
import re
str = "File!name?.txt"
f = open(os.path.join("/tmp", re.sub('[^-a-zA-Z0-9_.() ]+', '', str))
会导致/tmp/filename.txt的文件句柄。
You could use the re.sub() method to replace anything not “filelike”. But in effect, every character could be valid; so there are no prebuilt functions (I believe), to get it done.
import re
str = "File!name?.txt"
f = open(os.path.join("/tmp", re.sub('[^-a-zA-Z0-9_.() ]+', '', str))
Would result in a filehandle to /tmp/filename.txt.
回答 11
>>> import string
>>> safechars = bytearray(('_-.()' + string.digits + string.ascii_letters).encode())
>>> allchars = bytearray(range(0x100))
>>> deletechars = bytearray(set(allchars) - set(safechars))
>>> filename = u'#ab\xa0c.$%.txt'
>>> safe_filename = filename.encode('ascii', 'ignore').translate(None, deletechars).decode()
>>> safe_filename
'abc..txt'
它不处理空字符串,特殊文件名(“ nul”,“ con”等)。
>>> import string
>>> safechars = bytearray(('_-.()' + string.digits + string.ascii_letters).encode())
>>> allchars = bytearray(range(0x100))
>>> deletechars = bytearray(set(allchars) - set(safechars))
>>> filename = u'#ab\xa0c.$%.txt'
>>> safe_filename = filename.encode('ascii', 'ignore').translate(None, deletechars).decode()
>>> safe_filename
'abc..txt'
It doesn’t handle empty strings, special filenames (‘nul’, ‘con’, etc).
回答 12
虽然您必须要小心。如果您只看常规语言,则在介绍中并没有明确指出。如果仅使用ascii字符对某些单词进行消毒,则某些单词可能变得毫无意义,或变得另一种含义。
假设您有“Forêtpoésie”(森林诗歌),您的卫生处理可能会给您带来“ fort-posie”(强+毫无意义的东西)
更糟糕的是,如果您不得不处理汉字。
您的系统“下北沢”可能最终会执行“ —”,这注定会在一段时间后失败,并且不是很有帮助。因此,如果您只处理文件,我建议您将它们称为您控制的通用链,或者保持其原样。对于URI,大致相同。
Though you have to be careful. It is not clearly said in your intro, if you are looking only at latine language. Some words can become meaningless or another meaning if you sanitize them with ascii characters only.
imagine you have “forêt poésie” (forest poetry), your sanitization might give “fort-posie” (strong + something meaningless)
Worse if you have to deal with chinese characters.
“下北沢” your system might end up doing “—” which is doomed to fail after a while and not very helpful. So if you deal with only files I would encourage to either call them a generic chain that you control or to keep the characters as it is. For URIs, about the same.
回答 13
为什么不只用try / except包裹“ osopen”,然后让底层的OS整理出文件是否有效?
这似乎工作量少得多,并且无论使用哪种操作系统,这都是有效的。
Why not just wrap the “osopen” with a try/except and let the underlying OS sort out whether the file is valid?
This seems like much less work and is valid no matter which OS you use.
回答 14
其他注释尚未解决的另一个问题是空字符串,这显然不是有效的文件名。您还可以通过剥离太多字符而最终得到一个空字符串。
对于Windows保留的文件名和带点的问题,对“如何从任意用户输入中对有效文件名进行规范化”这个问题的最安全答案是什么?是“什至不用费劲尝试”:如果您可以找到其他避免它的方法(例如,使用数据库中的整数主键作为文件名),则可以这样做。
如果需要,并且确实需要允许空格和“。” 对于文件扩展名,请尝试以下操作:
import re
badchars= re.compile(r'[^A-Za-z0-9_. ]+|^\.|\.$|^ | $|^$')
badnames= re.compile(r'(aux|com[1-9]|con|lpt[1-9]|prn)(\.|$)')
def makeName(s):
name= badchars.sub('_', s)
if badnames.match(name):
name= '_'+name
return name
即使是这种情况也无法保证,尤其是在意外的OS上,例如RISC OS讨厌空格并使用’。作为目录分隔符。
Another issue that the other comments haven’t addressed yet is the empty string, which is obviously not a valid filename. You can also end up with an empty string from stripping too many characters.
What with the Windows reserved filenames and issues with dots, the safest answer to the question “how do I normalise a valid filename from arbitrary user input?” is “don’t even bother try”: if you can find any other way to avoid it (eg. using integer primary keys from a database as filenames), do that.
If you must, and you really need to allow spaces and ‘.’ for file extensions as part of the name, try something like:
import re
badchars= re.compile(r'[^A-Za-z0-9_. ]+|^\.|\.$|^ | $|^$')
badnames= re.compile(r'(aux|com[1-9]|con|lpt[1-9]|prn)(\.|$)')
def makeName(s):
name= badchars.sub('_', s)
if badnames.match(name):
name= '_'+name
return name
Even this can’t be guaranteed right especially on unexpected OSs — for example RISC OS hates spaces and uses ‘.’ as a directory separator.
回答 15
我喜欢这里的python-slugify方法,但是它也剥离了点,这是不希望的。所以我对其进行了优化,以便以这种方式将干净的文件名上传到s3:
pip install python-slugify
示例代码:
s = 'Very / Unsafe / file\nname hähä \n\r .txt'
clean_basename = slugify(os.path.splitext(s)[0])
clean_extension = slugify(os.path.splitext(s)[1][1:])
if clean_extension:
clean_filename = '{}.{}'.format(clean_basename, clean_extension)
elif clean_basename:
clean_filename = clean_basename
else:
clean_filename = 'none' # only unclean characters
输出:
>>> clean_filename
'very-unsafe-file-name-haha.txt'
这是如此的故障安全,它适用于没有扩展名的文件名,甚至只适用于不安全字符的文件名(结果在none
这里)。
I liked the python-slugify approach here but it was stripping dots also away which was not desired. So I optimized it for uploading a clean filename to s3 this way:
pip install python-slugify
Example code:
s = 'Very / Unsafe / file\nname hähä \n\r .txt'
clean_basename = slugify(os.path.splitext(s)[0])
clean_extension = slugify(os.path.splitext(s)[1][1:])
if clean_extension:
clean_filename = '{}.{}'.format(clean_basename, clean_extension)
elif clean_basename:
clean_filename = clean_basename
else:
clean_filename = 'none' # only unclean characters
Output:
>>> clean_filename
'very-unsafe-file-name-haha.txt'
This is so failsafe, it works with filenames without extension and it even works for only unsafe characters file names (result is none
here).
回答 16
为python 3.6修改的答案
import string
import unicodedata
validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
def removeDisallowedFilenameChars(filename):
cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
return ''.join(chr(c) for c in cleanedFilename if chr(c) in validFilenameChars)
Answer modified for python 3.6
import string
import unicodedata
validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
def removeDisallowedFilenameChars(filename):
cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
return ''.join(chr(c) for c in cleanedFilename if chr(c) in validFilenameChars)
回答 17
我知道有很多答案,但是它们大多依赖于正则表达式或外部模块,因此我想提出自己的答案。一个纯python函数,不需要外部模块,不使用正则表达式。我的方法不是清除无效字符,而仅允许有效字符。
def normalizefilename(fn):
validchars = "-_.() "
out = ""
for c in fn:
if str.isalpha(c) or str.isdigit(c) or (c in validchars):
out += c
else:
out += "_"
return out
如果愿意,可以将自己的有效字符添加到 validchars
变量开头,例如英文字母中不存在的国家字母。这是您可能想要或不想要的:某些未在UTF-8上运行的文件系统可能仍存在非ASCII字符问题。
此函数用于测试单个文件名的有效性,因此它将使用_替换路径分隔符,因为它们是无效字符。如果要添加它,修改if
包含OS路径分隔符很简单。
I realise there are many answers but they mostly rely on regular expressions or external modules, so I’d like to throw in my own answer. A pure python function, no external module needed, no regular expression used. My approach is not to clean invalid chars, but to only allow valid ones.
def normalizefilename(fn):
validchars = "-_.() "
out = ""
for c in fn:
if str.isalpha(c) or str.isdigit(c) or (c in validchars):
out += c
else:
out += "_"
return out
if you like, you can add your own valid chars to the validchars
variable at the beginning, such as your national letters that don’t exist in English alphabet. This is something you may or may not want: some file systems that don’t run on UTF-8 might still have problems with non-ASCII chars.
This function is to test for a single file name validity, so it will replace path separators with _ considering them invalid chars. If you want to add that, it is trivial to modify the if
to include os path separator.
回答 18
这些解决方案大多数都不起作用。
‘/ hello / world’->’helloworld’
‘/ helloworld’/->’helloworld’
通常,这不是您想要的,例如,您要为每个链接保存html,而要覆盖其他网页的html。
我腌一个字典,如:
{'helloworld':
(
{'/hello/world': 'helloworld', '/helloworld/': 'helloworld1'},
2)
}
2表示应该附加到下一个文件名的数字。
每次从字典中查找文件名。如果不存在,我创建一个新的,如果需要的话,添加最大数量。
Most of these solutions don’t work.
‘/hello/world’ -> ‘helloworld’
‘/helloworld’/ -> ‘helloworld’
This isn’t what you want generally, say you are saving the html for each link, you’re going to overwrite the html for a different webpage.
I pickle a dict such as:
{'helloworld':
(
{'/hello/world': 'helloworld', '/helloworld/': 'helloworld1'},
2)
}
2 represents the number that should be appended to the next filename.
I look up the filename each time from the dict. If it’s not there, I create a new one, appending the max number if needed.
回答 19
不完全是OP的要求,但这是我使用的,因为我需要唯一且可逆的转换:
# p3 code
def safePath (url):
return ''.join(map(lambda ch: chr(ch) if ch in safePath.chars else '%%%02x' % ch, url.encode('utf-8')))
safePath.chars = set(map(lambda x: ord(x), '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+-_ .'))
至少从sysadmin的角度来看,结果是“有点”可读的。
Not exactly what OP was asking for but this is what I use because I need unique and reversible conversions:
# p3 code
def safePath (url):
return ''.join(map(lambda ch: chr(ch) if ch in safePath.chars else '%%%02x' % ch, url.encode('utf-8')))
safePath.chars = set(map(lambda x: ord(x), '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+-_ .'))
Result is “somewhat” readable, at least from a sysadmin point of view.
回答 20
回答 21
我确定这不是一个很好的答案,因为它修改了循环的字符串,但似乎可以正常工作:
import string
for chr in your_string:
if chr == ' ':
your_string = your_string.replace(' ', '_')
elif chr not in string.ascii_letters or chr not in string.digits:
your_string = your_string.replace(chr, '')
I’m sure this isn’t a great answer, since it modifies the string it’s looping over, but it seems to work alright:
import string
for chr in your_string:
if chr == ' ':
your_string = your_string.replace(' ', '_')
elif chr not in string.ascii_letters or chr not in string.digits:
your_string = your_string.replace(chr, '')
回答 22
更新
在这6年的历史中,所有链接都无法修复。
另外,我也不会再这样做了,只base64
编码或删除不安全的字符。Python 3示例:
import re
t = re.compile("[a-zA-Z0-9.,_-]")
unsafe = "abc∂éåß®∆˚˙©¬ñ√ƒµ©∆∫ø"
safe = [ch for ch in unsafe if t.match(ch)]
# => 'abc'
使用base64
可以进行编码和解码,因此可以再次检索原始文件名。
但是根据使用情况,最好生成一个随机文件名并将元数据存储在单独的文件或数据库中。
from random import choice
from string import ascii_lowercase, ascii_uppercase, digits
allowed_chr = ascii_lowercase + ascii_uppercase + digits
safe = ''.join([choice(allowed_chr) for _ in range(16)])
# => 'CYQ4JDKE9JfcRzAZ'
原始链接答案:
该bobcat
项目包含一个执行此操作的python模块。
它并不完全健壮,请参阅此帖子和此回复。
因此,如前所述:base64
如果可读性无关紧要,则编码可能是一个更好的主意。
UPDATE
All links broken beyond repair in this 6 year old answer.
Also, I also wouldn’t do it this way anymore, just base64
encode or drop unsafe chars. Python 3 example:
import re
t = re.compile("[a-zA-Z0-9.,_-]")
unsafe = "abc∂éåß®∆˚˙©¬ñ√ƒµ©∆∫ø"
safe = [ch for ch in unsafe if t.match(ch)]
# => 'abc'
With base64
you can encode and decode, so you can retrieve the original filename again.
But depending on the use case you might be better off generating a random filename and storing the metadata in separate file or DB.
from random import choice
from string import ascii_lowercase, ascii_uppercase, digits
allowed_chr = ascii_lowercase + ascii_uppercase + digits
safe = ''.join([choice(allowed_chr) for _ in range(16)])
# => 'CYQ4JDKE9JfcRzAZ'
ORIGINAL LINKROTTEN ANSWER:
The bobcat
project contains a python module that does just this.
It’s not completely robust, see this post and this reply.
So, as noted: base64
encoding is probably a better idea if readability doesn’t matter.