问题:解析配置文件,环境和命令行参数,以获取单个选项集合
Python的标准库具有用于配置文件解析(configparser),环境变量读取(os.environ)和命令行参数解析(argparse)的模块。我想编写一个可以完成所有这些任务的程序,并且:
具有一系列的选项值:
- 默认选项值,被覆盖
- 配置文件选项,被覆盖
- 环境变量,被覆盖
- 命令行选项。
允许在命令行上使用例如指定一个或多个配置文件位置--config-file foo.conf
,并读取该位置(代替或添加到常规配置文件中)。这仍然必须遵循上述级联。
允许在单个位置定义选项,以确定配置文件和命令行的解析行为。
将已解析的选项统一为一个选项值集合,供程序的其余部分访问,而无需关心它们的来源。
我需要的所有内容显然都在Python标准库中,但它们不能一起正常工作。
如何以最小的Python标准库偏差实现此目标?
Python’s standard library has modules for configuration file parsing (configparser), environment variable reading (os.environ), and command-line argument parsing (argparse). I want to write a program that does all those, and also:
Has a cascade of option values:
- default option values, overridden by
- config file options, overridden by
- environment variables, overridden by
- command-line options.
Allows one or more configuration file locations specified on the command line with e.g. --config-file foo.conf
, and reads that (either instead of, or additional to, the usual configuration file). This must still obey the above cascade.
Allows option definitions in a single place to determine the parsing behaviour for configuration files and the command line.
Unifies the parsed options into a single collection of option values for the rest of the program to access without caring where they came from.
Everything I need is apparently in the Python standard library, but they don’t work together smoothly.
How can I achieve this with minimum deviation from the Python standard library?
回答 0
只要您对看起来像命令行的配置文件感到满意,argparse模块就不会让您感到困惑。(我认为这是一个优势,因为用户只需要学习一种语法即可。)例如,将fromfile_prefix_chars设置为@
,可以做到,
my_prog --foo=bar
相当于
my_prog @baz.conf
如果@baz.conf
是,
--foo
bar
您甚至可以foo.conf
通过修改来自动寻找代码argv
if os.path.exists('foo.conf'):
argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)
通过配置ArgumentParser的子类并添加convert_arg_line_to_args方法,可以修改这些配置文件的格式。
The argparse module makes this not nuts, as long as you’re happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @
, makes it so that,
my_prog --foo=bar
is equivalent to
my_prog @baz.conf
if @baz.conf
is,
--foo
bar
You can even have your code look for foo.conf
automatically by modifying argv
if os.path.exists('foo.conf'):
argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)
The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.
回答 1
更新: 我终于把它放在pypi上了。通过以下方式安装最新版本:
pip install configargparser
完整的帮助和说明在这里。
原始帖子
这是我一起砍的一些东西。随意在评论中提出改进建议/错误报告:
import argparse
import ConfigParser
import os
def _identity(x):
return x
_SENTINEL = object()
class AddConfigFile(argparse.Action):
def __call__(self,parser,namespace,values,option_string=None):
# I can never remember if `values` is a list all the time or if it
# can be a scalar string; this takes care of both.
if isinstance(values,basestring):
parser.config_files.append(values)
else:
parser.config_files.extend(values)
class ArgumentConfigEnvParser(argparse.ArgumentParser):
def __init__(self,*args,**kwargs):
"""
Added 2 new keyword arguments to the ArgumentParser constructor:
config --> List of filenames to parse for config goodness
default_section --> name of the default section in the config file
"""
self.config_files = kwargs.pop('config',[]) #Must be a list
self.default_section = kwargs.pop('default_section','MAIN')
self._action_defaults = {}
argparse.ArgumentParser.__init__(self,*args,**kwargs)
def add_argument(self,*args,**kwargs):
"""
Works like `ArgumentParser.add_argument`, except that we've added an action:
config: add a config file to the parser
This also adds the ability to specify which section of the config file to pull the
data from, via the `section` keyword. This relies on the (undocumented) fact that
`ArgumentParser.add_argument` actually returns the `Action` object that it creates.
We need this to reliably get `dest` (although we could probably write a simple
function to do this for us).
"""
if 'action' in kwargs and kwargs['action'] == 'config':
kwargs['action'] = AddConfigFile
kwargs['default'] = argparse.SUPPRESS
# argparse won't know what to do with the section, so
# we'll pop it out and add it back in later.
#
# We also have to prevent argparse from doing any type conversion,
# which is done explicitly in parse_known_args.
#
# This way, we can reliably check whether argparse has replaced the default.
#
section = kwargs.pop('section', self.default_section)
type = kwargs.pop('type', _identity)
default = kwargs.pop('default', _SENTINEL)
if default is not argparse.SUPPRESS:
kwargs.update(default=_SENTINEL)
else:
kwargs.update(default=argparse.SUPPRESS)
action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)
kwargs.update(section=section, type=type, default=default)
self._action_defaults[action.dest] = (args,kwargs)
return action
def parse_known_args(self,args=None, namespace=None):
# `parse_args` calls `parse_known_args`, so we should be okay with this...
ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace)
config_parser = ConfigParser.SafeConfigParser()
config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
config_parser.read(config_files)
for dest,(args,init_dict) in self._action_defaults.items():
type_converter = init_dict['type']
default = init_dict['default']
obj = default
if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line
obj = getattr(ns,dest)
else: # not found on commandline
try: # get from config file
obj = config_parser.get(init_dict['section'],dest)
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file
try: # get from environment
obj = os.environ[dest.upper()]
except KeyError:
pass
if obj is _SENTINEL:
setattr(ns,dest,None)
elif obj is argparse.SUPPRESS:
pass
else:
setattr(ns,dest,type_converter(obj))
return ns, argv
if __name__ == '__main__':
fake_config = """
[MAIN]
foo:bar
bar:1
"""
with open('_config.file','w') as fout:
fout.write(fake_config)
parser = ArgumentConfigEnvParser()
parser.add_argument('--config-file', action='config', help="location of config file")
parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...")
parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)")
parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)")
parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int")
ns = parser.parse_args([])
parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
config_defaults = {'foo':'bar','bar':1}
env_defaults = {"baz":3.14159}
# This should be the defaults we gave the parser
print ns
assert ns.__dict__ == parser_defaults
# This should be the defaults we gave the parser + config defaults
d = parser_defaults.copy()
d.update(config_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
os.environ['BAZ'] = "3.14159"
# This should be the parser defaults + config defaults + env_defaults
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
# This should be the parser defaults + config defaults + env_defaults + commandline
commandline = {'foo':'3','qux':4}
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
d.update(commandline)
ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
print ns
assert ns.__dict__ == d
os.remove('_config.file')
去做
此实现仍未完成。以下是部分待办事项清单:
符合记录的行为
- (简单)编写一个
dest
从args
in 弄清楚的函数add_argument
,而不是依赖于Action
对象
- (平凡的)编写一个
parse_args
使用的函数parse_known_args
。(例如,parse_args
从cpython
实现中复制以确保调用它parse_known_args
。)
不太容易的东西
我还没有尝试过。它不太可能(但仍然有可能!)能够正常工作……
- (很难?)互斥
- (很难?)参数组 (如果实现,这些组应
section
在配置文件中获得一个。)
- (硬吗?)子命令 (子命令也应
section
在配置文件中得到一个。)
UPDATE: I finally got around to putting this on pypi. Install latest version via:
pip install configargparser
Full help and instructions are here.
Original post
Here’s a little something that I hacked together. Feel free suggest improvements/bug-reports in the comments:
import argparse
import ConfigParser
import os
def _identity(x):
return x
_SENTINEL = object()
class AddConfigFile(argparse.Action):
def __call__(self,parser,namespace,values,option_string=None):
# I can never remember if `values` is a list all the time or if it
# can be a scalar string; this takes care of both.
if isinstance(values,basestring):
parser.config_files.append(values)
else:
parser.config_files.extend(values)
class ArgumentConfigEnvParser(argparse.ArgumentParser):
def __init__(self,*args,**kwargs):
"""
Added 2 new keyword arguments to the ArgumentParser constructor:
config --> List of filenames to parse for config goodness
default_section --> name of the default section in the config file
"""
self.config_files = kwargs.pop('config',[]) #Must be a list
self.default_section = kwargs.pop('default_section','MAIN')
self._action_defaults = {}
argparse.ArgumentParser.__init__(self,*args,**kwargs)
def add_argument(self,*args,**kwargs):
"""
Works like `ArgumentParser.add_argument`, except that we've added an action:
config: add a config file to the parser
This also adds the ability to specify which section of the config file to pull the
data from, via the `section` keyword. This relies on the (undocumented) fact that
`ArgumentParser.add_argument` actually returns the `Action` object that it creates.
We need this to reliably get `dest` (although we could probably write a simple
function to do this for us).
"""
if 'action' in kwargs and kwargs['action'] == 'config':
kwargs['action'] = AddConfigFile
kwargs['default'] = argparse.SUPPRESS
# argparse won't know what to do with the section, so
# we'll pop it out and add it back in later.
#
# We also have to prevent argparse from doing any type conversion,
# which is done explicitly in parse_known_args.
#
# This way, we can reliably check whether argparse has replaced the default.
#
section = kwargs.pop('section', self.default_section)
type = kwargs.pop('type', _identity)
default = kwargs.pop('default', _SENTINEL)
if default is not argparse.SUPPRESS:
kwargs.update(default=_SENTINEL)
else:
kwargs.update(default=argparse.SUPPRESS)
action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)
kwargs.update(section=section, type=type, default=default)
self._action_defaults[action.dest] = (args,kwargs)
return action
def parse_known_args(self,args=None, namespace=None):
# `parse_args` calls `parse_known_args`, so we should be okay with this...
ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace)
config_parser = ConfigParser.SafeConfigParser()
config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
config_parser.read(config_files)
for dest,(args,init_dict) in self._action_defaults.items():
type_converter = init_dict['type']
default = init_dict['default']
obj = default
if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line
obj = getattr(ns,dest)
else: # not found on commandline
try: # get from config file
obj = config_parser.get(init_dict['section'],dest)
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file
try: # get from environment
obj = os.environ[dest.upper()]
except KeyError:
pass
if obj is _SENTINEL:
setattr(ns,dest,None)
elif obj is argparse.SUPPRESS:
pass
else:
setattr(ns,dest,type_converter(obj))
return ns, argv
if __name__ == '__main__':
fake_config = """
[MAIN]
foo:bar
bar:1
"""
with open('_config.file','w') as fout:
fout.write(fake_config)
parser = ArgumentConfigEnvParser()
parser.add_argument('--config-file', action='config', help="location of config file")
parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...")
parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)")
parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)")
parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int")
ns = parser.parse_args([])
parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
config_defaults = {'foo':'bar','bar':1}
env_defaults = {"baz":3.14159}
# This should be the defaults we gave the parser
print ns
assert ns.__dict__ == parser_defaults
# This should be the defaults we gave the parser + config defaults
d = parser_defaults.copy()
d.update(config_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
os.environ['BAZ'] = "3.14159"
# This should be the parser defaults + config defaults + env_defaults
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
# This should be the parser defaults + config defaults + env_defaults + commandline
commandline = {'foo':'3','qux':4}
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
d.update(commandline)
ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
print ns
assert ns.__dict__ == d
os.remove('_config.file')
TODO
This implementation is still incomplete. Here’s a partial TODO list:
Conform to documented behavior
- (easy) Write a function that figures out
dest
from args
in add_argument
, instead of relying on the Action
object
- (trivial) Write a
parse_args
function which uses parse_known_args
. (e.g. copy parse_args
from the cpython
implementation to guarantee it calls parse_known_args
.)
Less Easy Stuff…
I haven’t tried any of this yet. It’s unlikely—but still possible!—that it could just work…
回答 2
有一个库可以做到这一点,叫做configglue。
configglue是一个将python的optparse.OptionParser和ConfigParser.ConfigParser粘合在一起的库,因此当您要将相同的选项导出到配置文件和命令行界面时,不必重复自己的操作。
它还支持环境变量。
还有一种叫库ConfigArgParse这是
argparse的直接替代品,允许通过配置文件和/或环境变量设置选项。
您可能对ŁukaszLanga进行的有关PyCon的配置感兴趣- 让他们进行配置!
There’s library that does exactly this called configglue.
configglue is a library that glues together python’s
optparse.OptionParser and ConfigParser.ConfigParser, so that you don’t
have to repeat yourself when you want to export the same options to a
configuration file and a commandline interface.
It also supports environment variables.
There’s also another library called ConfigArgParse which is
A drop-in replacement for argparse that allows options to also be set
via config files and/or environment variables.
You might be interested in PyCon talk about configuration by Łukasz Langa – Let Them Configure!
回答 3
虽然我自己没有尝试过,但是有一个ConfigArgParse库,它指出它可以完成您想要的大多数事情:
argparse的直接替代品,允许通过配置文件和/或环境变量设置选项。
While I haven’t tried it by my own, there is ConfigArgParse library which states that it does most of things that you want:
A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.
回答 4
似乎标准库没有解决这个问题,从而使每个程序员都难以为继configparser
,argparse
并且os.environ
以笨拙的方式将它们放在一起。
It seems the standard library doesn’t address this, leaving each programmer to cobble configparser
and argparse
and os.environ
all together in clunky ways.
回答 5
据我所知,Python标准库不提供此功能。我通过编写要使用optparse
和ConfigParser
解析命令行和配置文件,并在它们之上提供了抽象层,从而。但是,您需要将此作为单独的依赖项,从您先前的评论看来,这是令人讨厌的。
如果要查看我编写的代码,请访问http://liw.fi/cliapp/。它已集成到我的“命令行应用程序框架”库中,因为这是框架需要执行的大部分工作。
The Python standard library does not provide this, as far as I know. I solved this for myself by writing code to use optparse
and ConfigParser
to parse the command line and config files, and provide an abstraction layer on top of them. However, you would need this as a separate dependency, which from your earlier comment seems to be unpalatable.
If you want to look at the code I wrote, it’s at http://liw.fi/cliapp/. It’s integrated into my “command line application framework” library, since that’s a large part of what the framework needs to do.
回答 6
最近,我使用“ optparse”进行了类似的尝试。
我使用“ –Store”和“ –Check”命令将其设置为OptonParser的子类。
下面的代码几乎可以涵盖您。您只需要定义自己的“加载”和“存储”方法即可接受/返回字典,因此您已经设置好了。
class SmartParse(optparse.OptionParser):
def __init__(self,defaults,*args,**kwargs):
self.smartDefaults=defaults
optparse.OptionParser.__init__(self,*args,**kwargs)
fileGroup = optparse.OptionGroup(self,'handle stored defaults')
fileGroup.add_option(
'-S','--Store',
dest='Action',
action='store_const',const='Store',
help='store command line settings'
)
fileGroup.add_option(
'-C','--Check',
dest='Action',
action='store_const',const='Check',
help ='check stored settings'
)
self.add_option_group(fileGroup)
def parse_args(self,*args,**kwargs):
(options,arguments) = optparse.OptionParser.parse_args(self,*args,**kwargs)
action = options.__dict__.pop('Action')
if action == 'Check':
assert all(
value is None
for (key,value) in options.__dict__.iteritems()
)
print 'defaults:',self.smartDefaults
print 'config:',self.load()
sys.exit()
elif action == 'Store':
self.store(options.__dict__)
sys.exit()
else:
config=self.load()
commandline=dict(
[key,val]
for (key,val) in options.__dict__.iteritems()
if val is not None
)
result = {}
result.update(self.defaults)
result.update(config)
result.update(commandline)
return result,arguments
def load(self):
return {}
def store(self,optionDict):
print 'Storing:',optionDict
I was tried something like this recently, using “optparse”.
I set it up as a sub-class of OptonParser, with a ‘–Store’ and a ‘–Check’ command.
The code below should pretty much have you covered. You just need to define your own ‘load’ and ‘store’ methods which accept/return dictionaries and you’re prey much set.
class SmartParse(optparse.OptionParser):
def __init__(self,defaults,*args,**kwargs):
self.smartDefaults=defaults
optparse.OptionParser.__init__(self,*args,**kwargs)
fileGroup = optparse.OptionGroup(self,'handle stored defaults')
fileGroup.add_option(
'-S','--Store',
dest='Action',
action='store_const',const='Store',
help='store command line settings'
)
fileGroup.add_option(
'-C','--Check',
dest='Action',
action='store_const',const='Check',
help ='check stored settings'
)
self.add_option_group(fileGroup)
def parse_args(self,*args,**kwargs):
(options,arguments) = optparse.OptionParser.parse_args(self,*args,**kwargs)
action = options.__dict__.pop('Action')
if action == 'Check':
assert all(
value is None
for (key,value) in options.__dict__.iteritems()
)
print 'defaults:',self.smartDefaults
print 'config:',self.load()
sys.exit()
elif action == 'Store':
self.store(options.__dict__)
sys.exit()
else:
config=self.load()
commandline=dict(
[key,val]
for (key,val) in options.__dict__.iteritems()
if val is not None
)
result = {}
result.update(self.defaults)
result.update(config)
result.update(commandline)
return result,arguments
def load(self):
return {}
def store(self,optionDict):
print 'Storing:',optionDict
回答 7
为了满足所有这些要求,我建议编写自己的库,该库同时使用[opt | arg] parse和configparser作为基础功能。
鉴于前两个条件和最后一个要求,我想说你想要:
第一步:执行仅用于–config-file选项的命令行解析器传递。
第二步:解析配置文件。
第三步:使用配置文件pass的输出作为默认值,设置第二个命令行解析器pass。
第三个要求可能意味着您必须设计自己的选项定义系统以公开您关心的optparse和configparser的所有功能,并编写一些管道以在它们之间进行转换。
To hit all those requirements, I would recommend writing your own library that uses both [opt|arg]parse and configparser for the underlying functionality.
Given the first two and the last requirement, I’d say you want:
Step one: Do a command line parser pass that only looks for the –config-file option.
Step two: Parse the config file.
Step three: set up a second command line parser pass using the output of the config file pass as the defaults.
The third requirement likely means you have to design your own option definition system to expose all the functionality of optparse and configparser that you care about, and write some plumbing to do conversions in between.
回答 8
这是我一起学习过的一个模块,可以读取命令行参数,环境设置,ini文件以及密钥环值。它的要点也是可用的。
"""
Configuration Parser
Configurable parser that will parse config files, environment variables,
keyring, and command-line arguments.
Example test.ini file:
[defaults]
gini=10
[app]
xini = 50
Example test.arg file:
--xfarg=30
Example test.py file:
import os
import sys
import config
def main(argv):
'''Test.'''
options = [
config.Option("xpos",
help="positional argument",
nargs='?',
default="all",
env="APP_XPOS"),
config.Option("--xarg",
help="optional argument",
default=1,
type=int,
env="APP_XARG"),
config.Option("--xenv",
help="environment argument",
default=1,
type=int,
env="APP_XENV"),
config.Option("--xfarg",
help="@file argument",
default=1,
type=int,
env="APP_XFARG"),
config.Option("--xini",
help="ini argument",
default=1,
type=int,
ini_section="app",
env="APP_XINI"),
config.Option("--gini",
help="global ini argument",
default=1,
type=int,
env="APP_GINI"),
config.Option("--karg",
help="secret keyring arg",
default=-1,
type=int),
]
ini_file_paths = [
'/etc/default/app.ini',
os.path.join(os.path.dirname(os.path.abspath(__file__)),
'test.ini')
]
# default usage
conf = config.Config(prog='app', options=options,
ini_paths=ini_file_paths)
conf.parse()
print conf
# advanced usage
cli_args = conf.parse_cli(argv=argv)
env = conf.parse_env()
secrets = conf.parse_keyring(namespace="app")
ini = conf.parse_ini(ini_file_paths)
sources = {}
if ini:
for key, value in ini.iteritems():
conf[key] = value
sources[key] = "ini-file"
if secrets:
for key, value in secrets.iteritems():
conf[key] = value
sources[key] = "keyring"
if env:
for key, value in env.iteritems():
conf[key] = value
sources[key] = "environment"
if cli_args:
for key, value in cli_args.iteritems():
conf[key] = value
sources[key] = "command-line"
print '\n'.join(['%s:\t%s' % (k, v) for k, v in sources.items()])
if __name__ == "__main__":
if config.keyring:
config.keyring.set_password("app", "karg", "13")
main(sys.argv)
Example results:
$APP_XENV=10 python test.py api --xarg=2 @test.arg
<Config xpos=api, gini=1, xenv=10, xini=50, karg=13, xarg=2, xfarg=30>
xpos: command-line
xenv: environment
xini: ini-file
karg: keyring
xarg: command-line
xfarg: command-line
"""
import argparse
import ConfigParser
import copy
import os
import sys
try:
import keyring
except ImportError:
keyring = None
class Option(object):
"""Holds a configuration option and the names and locations for it.
Instantiate options using the same arguments as you would for an
add_arguments call in argparse. However, you have two additional kwargs
available:
env: the name of the environment variable to use for this option
ini_section: the ini file section to look this value up from
"""
def __init__(self, *args, **kwargs):
self.args = args or []
self.kwargs = kwargs or {}
def add_argument(self, parser, **override_kwargs):
"""Add an option to a an argparse parser."""
kwargs = {}
if self.kwargs:
kwargs = copy.copy(self.kwargs)
try:
del kwargs['env']
except KeyError:
pass
try:
del kwargs['ini_section']
except KeyError:
pass
kwargs.update(override_kwargs)
parser.add_argument(*self.args, **kwargs)
@property
def type(self):
"""The type of the option.
Should be a callable to parse options.
"""
return self.kwargs.get("type", str)
@property
def name(self):
"""The name of the option as determined from the args."""
for arg in self.args:
if arg.startswith("--"):
return arg[2:].replace("-", "_")
elif arg.startswith("-"):
continue
else:
return arg.replace("-", "_")
@property
def default(self):
"""The default for the option."""
return self.kwargs.get("default")
class Config(object):
"""Parses configuration sources."""
def __init__(self, options=None, ini_paths=None, **parser_kwargs):
"""Initialize with list of options.
:param ini_paths: optional paths to ini files to look up values from
:param parser_kwargs: kwargs used to init argparse parsers.
"""
self._parser_kwargs = parser_kwargs or {}
self._ini_paths = ini_paths or []
self._options = copy.copy(options) or []
self._values = {option.name: option.default
for option in self._options}
self._parser = argparse.ArgumentParser(**parser_kwargs)
self.pass_thru_args = []
@property
def prog(self):
"""Program name."""
return self._parser.prog
def __getitem__(self, key):
return self._values[key]
def __setitem__(self, key, value):
self._values[key] = value
def __delitem__(self, key):
del self._values[key]
def __contains__(self, key):
return key in self._values
def __iter__(self):
return iter(self._values)
def __len__(self):
return len(self._values)
def get(self, key, *args):
"""
Return the value for key if it exists otherwise the default.
"""
return self._values.get(key, *args)
def __getattr__(self, attr):
if attr in self._values:
return self._values[attr]
else:
raise AttributeError("'config' object has no attribute '%s'"
% attr)
def build_parser(self, options, **override_kwargs):
"""."""
kwargs = copy.copy(self._parser_kwargs)
kwargs.update(override_kwargs)
if 'fromfile_prefix_chars' not in kwargs:
kwargs['fromfile_prefix_chars'] = '@'
parser = argparse.ArgumentParser(**kwargs)
if options:
for option in options:
option.add_argument(parser)
return parser
def parse_cli(self, argv=None):
"""Parse command-line arguments into values."""
if not argv:
argv = sys.argv
options = []
for option in self._options:
temp = Option(*option.args, **option.kwargs)
temp.kwargs['default'] = argparse.SUPPRESS
options.append(temp)
parser = self.build_parser(options=options)
parsed, extras = parser.parse_known_args(argv[1:])
if extras:
valid, pass_thru = self.parse_passthru_args(argv[1:])
parsed, extras = parser.parse_known_args(valid)
if extras:
raise AttributeError("Unrecognized arguments: %s" %
' ,'.join(extras))
self.pass_thru_args = pass_thru + extras
return vars(parsed)
def parse_env(self):
results = {}
for option in self._options:
env_var = option.kwargs.get('env')
if env_var and env_var in os.environ:
value = os.environ[env_var]
results[option.name] = option.type(value)
return results
def get_defaults(self):
"""Use argparse to determine and return dict of defaults."""
parser = self.build_parser(options=self._options)
parsed, _ = parser.parse_known_args([])
return vars(parsed)
def parse_ini(self, paths=None):
"""Parse config files and return configuration options.
Expects array of files that are in ini format.
:param paths: list of paths to files to parse (uses ConfigParse logic).
If not supplied, uses the ini_paths value supplied on
initialization.
"""
results = {}
config = ConfigParser.SafeConfigParser()
config.read(paths or self._ini_paths)
for option in self._options:
ini_section = option.kwargs.get('ini_section')
if ini_section:
try:
value = config.get(ini_section, option.name)
results[option.name] = option.type(value)
except ConfigParser.NoSectionError:
pass
return results
def parse_keyring(self, namespace=None):
"""."""
results = {}
if not keyring:
return results
if not namespace:
namespace = self.prog
for option in self._options:
secret = keyring.get_password(namespace, option.name)
if secret:
results[option.name] = option.type(secret)
return results
def parse(self, argv=None):
"""."""
defaults = self.get_defaults()
args = self.parse_cli(argv=argv)
env = self.parse_env()
secrets = self.parse_keyring()
ini = self.parse_ini()
results = defaults
results.update(ini)
results.update(secrets)
results.update(env)
results.update(args)
self._values = results
return self
@staticmethod
def parse_passthru_args(argv):
"""Handles arguments to be passed thru to a subprocess using '--'.
:returns: tuple of two lists; args and pass-thru-args
"""
if '--' in argv:
dashdash = argv.index("--")
if dashdash == 0:
return argv[1:], []
elif dashdash > 0:
return argv[0:dashdash], argv[dashdash + 1:]
return argv, []
def __repr__(self):
return "<Config %s>" % ', '.join([
'%s=%s' % (k, v) for k, v in self._values.iteritems()])
def comma_separated_strings(value):
"""Handles comma-separated arguments passed in command-line."""
return map(str, value.split(","))
def comma_separated_pairs(value):
"""Handles comma-separated key/values passed in command-line."""
pairs = value.split(",")
results = {}
for pair in pairs:
key, pair_value = pair.split('=')
results[key] = pair_value
return results
Here’s a module I hacked together that reads command-line arguments, environment settings, ini files, and keyring values as well. It’s also available in a gist.
"""
Configuration Parser
Configurable parser that will parse config files, environment variables,
keyring, and command-line arguments.
Example test.ini file:
[defaults]
gini=10
[app]
xini = 50
Example test.arg file:
--xfarg=30
Example test.py file:
import os
import sys
import config
def main(argv):
'''Test.'''
options = [
config.Option("xpos",
help="positional argument",
nargs='?',
default="all",
env="APP_XPOS"),
config.Option("--xarg",
help="optional argument",
default=1,
type=int,
env="APP_XARG"),
config.Option("--xenv",
help="environment argument",
default=1,
type=int,
env="APP_XENV"),
config.Option("--xfarg",
help="@file argument",
default=1,
type=int,
env="APP_XFARG"),
config.Option("--xini",
help="ini argument",
default=1,
type=int,
ini_section="app",
env="APP_XINI"),
config.Option("--gini",
help="global ini argument",
default=1,
type=int,
env="APP_GINI"),
config.Option("--karg",
help="secret keyring arg",
default=-1,
type=int),
]
ini_file_paths = [
'/etc/default/app.ini',
os.path.join(os.path.dirname(os.path.abspath(__file__)),
'test.ini')
]
# default usage
conf = config.Config(prog='app', options=options,
ini_paths=ini_file_paths)
conf.parse()
print conf
# advanced usage
cli_args = conf.parse_cli(argv=argv)
env = conf.parse_env()
secrets = conf.parse_keyring(namespace="app")
ini = conf.parse_ini(ini_file_paths)
sources = {}
if ini:
for key, value in ini.iteritems():
conf[key] = value
sources[key] = "ini-file"
if secrets:
for key, value in secrets.iteritems():
conf[key] = value
sources[key] = "keyring"
if env:
for key, value in env.iteritems():
conf[key] = value
sources[key] = "environment"
if cli_args:
for key, value in cli_args.iteritems():
conf[key] = value
sources[key] = "command-line"
print '\n'.join(['%s:\t%s' % (k, v) for k, v in sources.items()])
if __name__ == "__main__":
if config.keyring:
config.keyring.set_password("app", "karg", "13")
main(sys.argv)
Example results:
$APP_XENV=10 python test.py api --xarg=2 @test.arg
<Config xpos=api, gini=1, xenv=10, xini=50, karg=13, xarg=2, xfarg=30>
xpos: command-line
xenv: environment
xini: ini-file
karg: keyring
xarg: command-line
xfarg: command-line
"""
import argparse
import ConfigParser
import copy
import os
import sys
try:
import keyring
except ImportError:
keyring = None
class Option(object):
"""Holds a configuration option and the names and locations for it.
Instantiate options using the same arguments as you would for an
add_arguments call in argparse. However, you have two additional kwargs
available:
env: the name of the environment variable to use for this option
ini_section: the ini file section to look this value up from
"""
def __init__(self, *args, **kwargs):
self.args = args or []
self.kwargs = kwargs or {}
def add_argument(self, parser, **override_kwargs):
"""Add an option to a an argparse parser."""
kwargs = {}
if self.kwargs:
kwargs = copy.copy(self.kwargs)
try:
del kwargs['env']
except KeyError:
pass
try:
del kwargs['ini_section']
except KeyError:
pass
kwargs.update(override_kwargs)
parser.add_argument(*self.args, **kwargs)
@property
def type(self):
"""The type of the option.
Should be a callable to parse options.
"""
return self.kwargs.get("type", str)
@property
def name(self):
"""The name of the option as determined from the args."""
for arg in self.args:
if arg.startswith("--"):
return arg[2:].replace("-", "_")
elif arg.startswith("-"):
continue
else:
return arg.replace("-", "_")
@property
def default(self):
"""The default for the option."""
return self.kwargs.get("default")
class Config(object):
"""Parses configuration sources."""
def __init__(self, options=None, ini_paths=None, **parser_kwargs):
"""Initialize with list of options.
:param ini_paths: optional paths to ini files to look up values from
:param parser_kwargs: kwargs used to init argparse parsers.
"""
self._parser_kwargs = parser_kwargs or {}
self._ini_paths = ini_paths or []
self._options = copy.copy(options) or []
self._values = {option.name: option.default
for option in self._options}
self._parser = argparse.ArgumentParser(**parser_kwargs)
self.pass_thru_args = []
@property
def prog(self):
"""Program name."""
return self._parser.prog
def __getitem__(self, key):
return self._values[key]
def __setitem__(self, key, value):
self._values[key] = value
def __delitem__(self, key):
del self._values[key]
def __contains__(self, key):
return key in self._values
def __iter__(self):
return iter(self._values)
def __len__(self):
return len(self._values)
def get(self, key, *args):
"""
Return the value for key if it exists otherwise the default.
"""
return self._values.get(key, *args)
def __getattr__(self, attr):
if attr in self._values:
return self._values[attr]
else:
raise AttributeError("'config' object has no attribute '%s'"
% attr)
def build_parser(self, options, **override_kwargs):
"""."""
kwargs = copy.copy(self._parser_kwargs)
kwargs.update(override_kwargs)
if 'fromfile_prefix_chars' not in kwargs:
kwargs['fromfile_prefix_chars'] = '@'
parser = argparse.ArgumentParser(**kwargs)
if options:
for option in options:
option.add_argument(parser)
return parser
def parse_cli(self, argv=None):
"""Parse command-line arguments into values."""
if not argv:
argv = sys.argv
options = []
for option in self._options:
temp = Option(*option.args, **option.kwargs)
temp.kwargs['default'] = argparse.SUPPRESS
options.append(temp)
parser = self.build_parser(options=options)
parsed, extras = parser.parse_known_args(argv[1:])
if extras:
valid, pass_thru = self.parse_passthru_args(argv[1:])
parsed, extras = parser.parse_known_args(valid)
if extras:
raise AttributeError("Unrecognized arguments: %s" %
' ,'.join(extras))
self.pass_thru_args = pass_thru + extras
return vars(parsed)
def parse_env(self):
results = {}
for option in self._options:
env_var = option.kwargs.get('env')
if env_var and env_var in os.environ:
value = os.environ[env_var]
results[option.name] = option.type(value)
return results
def get_defaults(self):
"""Use argparse to determine and return dict of defaults."""
parser = self.build_parser(options=self._options)
parsed, _ = parser.parse_known_args([])
return vars(parsed)
def parse_ini(self, paths=None):
"""Parse config files and return configuration options.
Expects array of files that are in ini format.
:param paths: list of paths to files to parse (uses ConfigParse logic).
If not supplied, uses the ini_paths value supplied on
initialization.
"""
results = {}
config = ConfigParser.SafeConfigParser()
config.read(paths or self._ini_paths)
for option in self._options:
ini_section = option.kwargs.get('ini_section')
if ini_section:
try:
value = config.get(ini_section, option.name)
results[option.name] = option.type(value)
except ConfigParser.NoSectionError:
pass
return results
def parse_keyring(self, namespace=None):
"""."""
results = {}
if not keyring:
return results
if not namespace:
namespace = self.prog
for option in self._options:
secret = keyring.get_password(namespace, option.name)
if secret:
results[option.name] = option.type(secret)
return results
def parse(self, argv=None):
"""."""
defaults = self.get_defaults()
args = self.parse_cli(argv=argv)
env = self.parse_env()
secrets = self.parse_keyring()
ini = self.parse_ini()
results = defaults
results.update(ini)
results.update(secrets)
results.update(env)
results.update(args)
self._values = results
return self
@staticmethod
def parse_passthru_args(argv):
"""Handles arguments to be passed thru to a subprocess using '--'.
:returns: tuple of two lists; args and pass-thru-args
"""
if '--' in argv:
dashdash = argv.index("--")
if dashdash == 0:
return argv[1:], []
elif dashdash > 0:
return argv[0:dashdash], argv[dashdash + 1:]
return argv, []
def __repr__(self):
return "<Config %s>" % ', '.join([
'%s=%s' % (k, v) for k, v in self._values.iteritems()])
def comma_separated_strings(value):
"""Handles comma-separated arguments passed in command-line."""
return map(str, value.split(","))
def comma_separated_pairs(value):
"""Handles comma-separated key/values passed in command-line."""
pairs = value.split(",")
results = {}
for pair in pairs:
key, pair_value = pair.split('=')
results[key] = pair_value
return results
回答 9
您可以为此使用ChainMap。看一看我在“哪个是允许在Python的命令行中重写配置选项的最佳方法?”的示例。这样的问题。
回答 10
我构建的库甜点正是为了满足您的大多数需求。
有关更多信息,请检查项目存储库中的README.rst 。请注意,它仅支持Python3.6 up。
例子
附加命令行选项
import click
from proj_X.core import conf
@click.command()
@conf.click_options
def cli():
click.echo(f'cache_expire = {conf.api.cache_expire}')
if __name__ == '__main__':
cli()
它会自动创建全面的帮助消息,并声明所有属性和默认值。
$ python -m proj_X.cli --help
Usage: cli.py [OPTIONS]
Options:
--api-cache_expire INTEGER [default: 86400]
--api-cache_prefix TEXT [default: proj_X_cache]
--api-url_base_path TEXT [default: api/v2/]
--db-db_name TEXT [default: proj_x]
--db-username TEXT [default: proj_x_admin]
--db-password TEXT [default: your_password]
--db-host TEXT [default: 127.0.0.1]
--help Show this message and exit.
加载环境变量
只需一行即可加载环境变量
conf.load_envvars('proj_X')
The library confect I built is precisely to meet most of your needs.
- It can load configuration file multiple times through given file paths or module name.
- It loads configurations from environment variables with a given prefix.
It can attach command line options to some click commands
(sorry, it’s not argparse, but click is better and much more advanced. confect
might support argparse in the future release).
- Most importantly,
confect
loads Python configuration files not JSON/YMAL/TOML/INI. Just like IPython profile file or DJANGO settings file, Python configuration file is flexible and easier to maintain.
For more information, please check the README.rst in the project repository. Be aware of that it supports only Python3.6 up.
Examples
Attaching command line options
import click
from proj_X.core import conf
@click.command()
@conf.click_options
def cli():
click.echo(f'cache_expire = {conf.api.cache_expire}')
if __name__ == '__main__':
cli()
It automatically creates a comprehensive help message with all properties and default values declared.
$ python -m proj_X.cli --help
Usage: cli.py [OPTIONS]
Options:
--api-cache_expire INTEGER [default: 86400]
--api-cache_prefix TEXT [default: proj_X_cache]
--api-url_base_path TEXT [default: api/v2/]
--db-db_name TEXT [default: proj_x]
--db-username TEXT [default: proj_x_admin]
--db-password TEXT [default: your_password]
--db-host TEXT [default: 127.0.0.1]
--help Show this message and exit.
Loading environment variables
It only needs one line to load environment variables
conf.load_envvars('proj_X')