如何“完美”地覆盖字典?

问题:如何“完美”地覆盖字典?

如何使dict的子类尽可能“完美” ?最终目标是要有一个简单的字典,其中的键是小写的。

似乎应该覆盖一些微小的原语才能完成这项工作,但是根据我的所有研究和尝试,似乎并非如此:

这是我的第一个尝试,get()不起作用,毫无疑问,还有许多其他小问题:

class arbitrary_dict(dict):
    """A dictionary that applies an arbitrary key-altering function
       before accessing the keys."""

    def __keytransform__(self, key):
        return key

    # Overridden methods. List from 
    # /programming/2390827/how-to-properly-subclass-dict

    def __init__(self, *args, **kwargs):
        self.update(*args, **kwargs)

    # Note: I'm using dict directly, since super(dict, self) doesn't work.
    # I'm not sure why, perhaps dict is not a new-style class.

    def __getitem__(self, key):
        return dict.__getitem__(self, self.__keytransform__(key))

    def __setitem__(self, key, value):
        return dict.__setitem__(self, self.__keytransform__(key), value)

    def __delitem__(self, key):
        return dict.__delitem__(self, self.__keytransform__(key))

    def __contains__(self, key):
        return dict.__contains__(self, self.__keytransform__(key))


class lcdict(arbitrary_dict):
    def __keytransform__(self, key):
        return str(key).lower()

How can I make as “perfect” a subclass of dict as possible? The end goal is to have a simple dict in which the keys are lowercase.

It would seem that there should be some tiny set of primitives I can override to make this work, but according to all my research and attempts it seem like this isn’t the case:

  • If I override __getitem__/__setitem__, then get/set don’t work. How can I make them work? Surely I don’t need to implement them individually?

  • Am I preventing pickling from working, and do I need to implement __setstate__ etc?

  • Do I need repr, update and __init__?

  • Should I just use mutablemapping (it seems one shouldn’t use UserDict or DictMixin)? If so, how? The docs aren’t exactly enlightening.

Here is my first go at it, get() doesn’t work and no doubt there are many other minor problems:

class arbitrary_dict(dict):
    """A dictionary that applies an arbitrary key-altering function
       before accessing the keys."""

    def __keytransform__(self, key):
        return key

    # Overridden methods. List from 
    # https://stackoverflow.com/questions/2390827/how-to-properly-subclass-dict

    def __init__(self, *args, **kwargs):
        self.update(*args, **kwargs)

    # Note: I'm using dict directly, since super(dict, self) doesn't work.
    # I'm not sure why, perhaps dict is not a new-style class.

    def __getitem__(self, key):
        return dict.__getitem__(self, self.__keytransform__(key))

    def __setitem__(self, key, value):
        return dict.__setitem__(self, self.__keytransform__(key), value)

    def __delitem__(self, key):
        return dict.__delitem__(self, self.__keytransform__(key))

    def __contains__(self, key):
        return dict.__contains__(self, self.__keytransform__(key))


class lcdict(arbitrary_dict):
    def __keytransform__(self, key):
        return str(key).lower()

回答 0

您可以使用模块中的ABC(抽象基类)编写行为dict非常简单的对象。它甚至会告诉您是否错过了一种方法,因此以下是关闭ABC的最低版本。collections.abc

from collections.abc import MutableMapping


class TransformedDict(MutableMapping):
    """A dictionary that applies an arbitrary key-altering
       function before accessing the keys"""

    def __init__(self, *args, **kwargs):
        self.store = dict()
        self.update(dict(*args, **kwargs))  # use the free update to set keys

    def __getitem__(self, key):
        return self.store[self.__keytransform__(key)]

    def __setitem__(self, key, value):
        self.store[self.__keytransform__(key)] = value

    def __delitem__(self, key):
        del self.store[self.__keytransform__(key)]

    def __iter__(self):
        return iter(self.store)

    def __len__(self):
        return len(self.store)

    def __keytransform__(self, key):
        return key

您可以从ABC获得一些免费方法:

class MyTransformedDict(TransformedDict):

    def __keytransform__(self, key):
        return key.lower()


s = MyTransformedDict([('Test', 'test')])

assert s.get('TEST') is s['test']   # free get
assert 'TeSt' in s                  # free __contains__
                                    # free setdefault, __eq__, and so on

import pickle
# works too since we just use a normal dict
assert pickle.loads(pickle.dumps(s)) == s

我不会dict直接继承(或其他内置)。这通常没有任何意义,因为您真正想要做的是实现a的接口dict。而这正是ABC的目的。

You can write an object that behaves like a dict quite easily with ABCs (Abstract Base Classes) from the collections.abc module. It even tells you if you missed a method, so below is the minimal version that shuts the ABC up.

from collections.abc import MutableMapping


class TransformedDict(MutableMapping):
    """A dictionary that applies an arbitrary key-altering
       function before accessing the keys"""

    def __init__(self, *args, **kwargs):
        self.store = dict()
        self.update(dict(*args, **kwargs))  # use the free update to set keys

    def __getitem__(self, key):
        return self.store[self._keytransform(key)]

    def __setitem__(self, key, value):
        self.store[self._keytransform(key)] = value

    def __delitem__(self, key):
        del self.store[self._keytransform(key)]

    def __iter__(self):
        return iter(self.store)
    
    def __len__(self):
        return len(self.store)

    def _keytransform(self, key):
        return key

You get a few free methods from the ABC:

class MyTransformedDict(TransformedDict):

    def _keytransform(self, key):
        return key.lower()


s = MyTransformedDict([('Test', 'test')])

assert s.get('TEST') is s['test']   # free get
assert 'TeSt' in s                  # free __contains__
                                    # free setdefault, __eq__, and so on

import pickle
# works too since we just use a normal dict
assert pickle.loads(pickle.dumps(s)) == s

I wouldn’t subclass dict (or other builtins) directly. It often makes no sense, because what you actually want to do is implement the interface of a dict. And that is exactly what ABCs are for.


回答 1

如何使dict的子类尽可能“完美”?

最终目标是要有一个简单的字典,其中的键是小写的。

  • 如果我覆盖__getitem__/ __setitem__,则获取/设置不起作用。我如何使它们工作?当然,我不需要单独实施它们吗?

  • 我是否在阻止酸洗,我需要实施 __setstate__等吗?

  • 我需要repr,update和__init__吗?

  • 我应该只使用mutablemapping(似乎不应该使用UserDictDictMixin)吗?如果是这样,怎么办?这些文档并不完全具有启发性。

可接受的答案将是我的第一种方法,但是由于它存在一些问题,并且由于没有人解决替代方法,实际上dict是将a子类化,因此我将在此处进行操作。

接受的答案有什么问题?

对我来说,这似乎是一个非常简单的请求:

如何使dict的子类尽可能“完美”?最终目标是要有一个简单的字典,其中的键是小写的。

接受的答案实际上不是子类dict,并且对此的测试失败:

>>> isinstance(MyTransformedDict([('Test', 'test')]), dict)
False

理想情况下,任何类型检查代码都将测试我们期望的接口或抽象基类,但是如果将我们的数据对象传递给正在测试的函数,dict而我们无法“修复”这些函数,则此代码将失败。

其他可能引起的争议:

  • 可接受的答案也缺少类方法:fromkeys
  • 可接受的答案也有冗余__dict__-因此会占用更多的内存空间:

    >>> s.foo = 'bar'
    >>> s.__dict__
    {'foo': 'bar', 'store': {'test': 'test'}}

实际上是子类化 dict

我们可以通过继承重用dict方法。我们需要做的就是创建一个接口层,以确保键(如果是字符串)以小写形式传递到字典中。

如果我覆盖__getitem__/ __setitem__,则获取/设置不起作用。我如何使它们工作?当然,我不需要单独实施它们吗?

好吧,分别实现它们是此方法的缺点,也是使用方法的不利之处MutableMapping(请参阅接受的答案),但实际上并不需要太多工作。

首先,让我们排除Python 2和Python 3之间的差异,创建一个singleton(_RaiseKeyError)以确保我们知道是否确实获得的参数dict.pop,并创建一个函数以确保我们的字符串键是小写的:

from itertools import chain
try:              # Python 2
    str_base = basestring
    items = 'iteritems'
except NameError: # Python 3
    str_base = str, bytes, bytearray
    items = 'items'

_RaiseKeyError = object() # singleton for no-default behavior

def ensure_lower(maybe_str):
    """dict keys can be any hashable object - only call lower if str"""
    return maybe_str.lower() if isinstance(maybe_str, str_base) else maybe_str

现在我们实现-我使用super了完整参数,因此该代码适用于Python 2和3:

class LowerDict(dict):  # dicts take a mapping or iterable as their optional first argument
    __slots__ = () # no __dict__ - that would be redundant
    @staticmethod # because this doesn't make sense as a global function.
    def _process_args(mapping=(), **kwargs):
        if hasattr(mapping, items):
            mapping = getattr(mapping, items)()
        return ((ensure_lower(k), v) for k, v in chain(mapping, getattr(kwargs, items)()))
    def __init__(self, mapping=(), **kwargs):
        super(LowerDict, self).__init__(self._process_args(mapping, **kwargs))
    def __getitem__(self, k):
        return super(LowerDict, self).__getitem__(ensure_lower(k))
    def __setitem__(self, k, v):
        return super(LowerDict, self).__setitem__(ensure_lower(k), v)
    def __delitem__(self, k):
        return super(LowerDict, self).__delitem__(ensure_lower(k))
    def get(self, k, default=None):
        return super(LowerDict, self).get(ensure_lower(k), default)
    def setdefault(self, k, default=None):
        return super(LowerDict, self).setdefault(ensure_lower(k), default)
    def pop(self, k, v=_RaiseKeyError):
        if v is _RaiseKeyError:
            return super(LowerDict, self).pop(ensure_lower(k))
        return super(LowerDict, self).pop(ensure_lower(k), v)
    def update(self, mapping=(), **kwargs):
        super(LowerDict, self).update(self._process_args(mapping, **kwargs))
    def __contains__(self, k):
        return super(LowerDict, self).__contains__(ensure_lower(k))
    def copy(self): # don't delegate w/ super - dict.copy() -> dict :(
        return type(self)(self)
    @classmethod
    def fromkeys(cls, keys, v=None):
        return super(LowerDict, cls).fromkeys((ensure_lower(k) for k in keys), v)
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__, super(LowerDict, self).__repr__())

我们使用的样板化的做法对任何方法或特殊方法引用的关键,但在其他方面,通过继承,我们获得方法:lenclearitemskeyspopitem,和values是免费的。尽管这需要一些仔细的思考才能正确解决,但看到它可行却是微不足道的。

(请注意,haskey在Python 2 中已弃用,在Python 3中已删除。)

这是一些用法:

>>> ld = LowerDict(dict(foo='bar'))
>>> ld['FOO']
'bar'
>>> ld['foo']
'bar'
>>> ld.pop('FoO')
'bar'
>>> ld.setdefault('Foo')
>>> ld
{'foo': None}
>>> ld.get('Bar')
>>> ld.setdefault('Bar')
>>> ld
{'bar': None, 'foo': None}
>>> ld.popitem()
('bar', None)

我是否在阻止酸洗,我需要实施 __setstate__等吗?

酸洗

dict子类的泡菜就可以了:

>>> import pickle
>>> pickle.dumps(ld)
b'\x80\x03c__main__\nLowerDict\nq\x00)\x81q\x01X\x03\x00\x00\x00fooq\x02Ns.'
>>> pickle.loads(pickle.dumps(ld))
{'foo': None}
>>> type(pickle.loads(pickle.dumps(ld)))
<class '__main__.LowerDict'>

__repr__

我需要repr,update和__init__吗?

我们定义了update__init__,但是__repr__默认情况下您会很漂亮:

>>> ld # without __repr__ defined for the class, we get this
{'foo': None}

但是,最好编写一个,__repr__以提高代码的可调试性。理想的测试是eval(repr(obj)) == obj。如果您的代码很简单,我强烈建议您:

>>> ld = LowerDict({})
>>> eval(repr(ld)) == ld
True
>>> ld = LowerDict(dict(a=1, b=2, c=3))
>>> eval(repr(ld)) == ld
True

您会看到,这正是我们重新创建等效对象所需要的-这可能会出现在我们的日志或回溯中:

>>> ld
LowerDict({'a': 1, 'c': 3, 'b': 2})

结论

我应该只使用mutablemapping(似乎不应该使用UserDictDictMixin)吗?如果是这样,怎么办?这些文档并不完全具有启发性。

是的,这些是更多几行代码,但是它们旨在变得更全面。我的第一个倾向是使用公认的答案,如果有问题,我将看一下我的答案-因为它有点复杂,而且没有ABC可以帮助我正确设置界面。

过早的优化将使搜索性能变得更加复杂。 MutableMapping更简单-在其他所有条件相同的情况下,它可以立即获得优势。不过,要列出所有差异,让我们进行比较和对比。

我应该补充一点,是有人试图将类似的字典放入collections模块中,但是被拒绝了。您可能应该这样做:

my_dict[transform(key)]

它应该更容易调试。

比较和对比

MutableMapping(缺少fromkeys)实现的6个接口函数和带有dict子类的11 个接口函数。我并不需要实现__iter__或者__len__,而是我要实现getsetdefaultpopupdatecopy__contains__,和fromkeys-但这些都是相当琐碎,因为我可以使用继承大多数这些实现的。

MutableMapping实现在Python中dict实现了一些用C 实现的东西-因此,我希望dict在某些情况下子类的性能更高。

我们__eq__在两种方法上都获得了自由-只有当另一个dict都为小写时,这两种方法才假定相等-但是,我再次认为,dict子类的比较会更快。

摘要:

  • 子类化MutableMapping更简单,发生错误的机会更少,但更慢,占用更多内存(请参阅冗余字典),并且失败isinstance(x, dict)
  • 子类化dict更快,使用更少的内存并通过isinstance(x, dict),但是实现起来却更加复杂。

哪个更完美?那取决于您对完美的定义。

How can I make as “perfect” a subclass of dict as possible?

The end goal is to have a simple dict in which the keys are lowercase.

  • If I override __getitem__/__setitem__, then get/set don’t work. How do I make them work? Surely I don’t need to implement them individually?

  • Am I preventing pickling from working, and do I need to implement __setstate__ etc?

  • Do I need repr, update and __init__?

  • Should I just use mutablemapping (it seems one shouldn’t use UserDict or DictMixin)? If so, how? The docs aren’t exactly enlightening.

The accepted answer would be my first approach, but since it has some issues, and since no one has addressed the alternative, actually subclassing a dict, I’m going to do that here.

What’s wrong with the accepted answer?

This seems like a rather simple request to me:

How can I make as “perfect” a subclass of dict as possible? The end goal is to have a simple dict in which the keys are lowercase.

The accepted answer doesn’t actually subclass dict, and a test for this fails:

>>> isinstance(MyTransformedDict([('Test', 'test')]), dict)
False

Ideally, any type-checking code would be testing for the interface we expect, or an abstract base class, but if our data objects are being passed into functions that are testing for dict – and we can’t “fix” those functions, this code will fail.

Other quibbles one might make:

  • The accepted answer is also missing the classmethod: fromkeys.
  • The accepted answer also has a redundant __dict__ – therefore taking up more space in memory:

    >>> s.foo = 'bar'
    >>> s.__dict__
    {'foo': 'bar', 'store': {'test': 'test'}}
    

Actually subclassing dict

We can reuse the dict methods through inheritance. All we need to do is create an interface layer that ensures keys are passed into the dict in lowercase form if they are strings.

If I override __getitem__/__setitem__, then get/set don’t work. How do I make them work? Surely I don’t need to implement them individually?

Well, implementing them each individually is the downside to this approach and the upside to using MutableMapping (see the accepted answer), but it’s really not that much more work.

First, let’s factor out the difference between Python 2 and 3, create a singleton (_RaiseKeyError) to make sure we know if we actually get an argument to dict.pop, and create a function to ensure our string keys are lowercase:

from itertools import chain
try:              # Python 2
    str_base = basestring
    items = 'iteritems'
except NameError: # Python 3
    str_base = str, bytes, bytearray
    items = 'items'

_RaiseKeyError = object() # singleton for no-default behavior

def ensure_lower(maybe_str):
    """dict keys can be any hashable object - only call lower if str"""
    return maybe_str.lower() if isinstance(maybe_str, str_base) else maybe_str

Now we implement – I’m using super with the full arguments so that this code works for Python 2 and 3:

class LowerDict(dict):  # dicts take a mapping or iterable as their optional first argument
    __slots__ = () # no __dict__ - that would be redundant
    @staticmethod # because this doesn't make sense as a global function.
    def _process_args(mapping=(), **kwargs):
        if hasattr(mapping, items):
            mapping = getattr(mapping, items)()
        return ((ensure_lower(k), v) for k, v in chain(mapping, getattr(kwargs, items)()))
    def __init__(self, mapping=(), **kwargs):
        super(LowerDict, self).__init__(self._process_args(mapping, **kwargs))
    def __getitem__(self, k):
        return super(LowerDict, self).__getitem__(ensure_lower(k))
    def __setitem__(self, k, v):
        return super(LowerDict, self).__setitem__(ensure_lower(k), v)
    def __delitem__(self, k):
        return super(LowerDict, self).__delitem__(ensure_lower(k))
    def get(self, k, default=None):
        return super(LowerDict, self).get(ensure_lower(k), default)
    def setdefault(self, k, default=None):
        return super(LowerDict, self).setdefault(ensure_lower(k), default)
    def pop(self, k, v=_RaiseKeyError):
        if v is _RaiseKeyError:
            return super(LowerDict, self).pop(ensure_lower(k))
        return super(LowerDict, self).pop(ensure_lower(k), v)
    def update(self, mapping=(), **kwargs):
        super(LowerDict, self).update(self._process_args(mapping, **kwargs))
    def __contains__(self, k):
        return super(LowerDict, self).__contains__(ensure_lower(k))
    def copy(self): # don't delegate w/ super - dict.copy() -> dict :(
        return type(self)(self)
    @classmethod
    def fromkeys(cls, keys, v=None):
        return super(LowerDict, cls).fromkeys((ensure_lower(k) for k in keys), v)
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__, super(LowerDict, self).__repr__())

We use an almost boiler-plate approach for any method or special method that references a key, but otherwise, by inheritance, we get methods: len, clear, items, keys, popitem, and values for free. While this required some careful thought to get right, it is trivial to see that this works.

(Note that haskey was deprecated in Python 2, removed in Python 3.)

Here’s some usage:

>>> ld = LowerDict(dict(foo='bar'))
>>> ld['FOO']
'bar'
>>> ld['foo']
'bar'
>>> ld.pop('FoO')
'bar'
>>> ld.setdefault('Foo')
>>> ld
{'foo': None}
>>> ld.get('Bar')
>>> ld.setdefault('Bar')
>>> ld
{'bar': None, 'foo': None}
>>> ld.popitem()
('bar', None)

Am I preventing pickling from working, and do I need to implement __setstate__ etc?

pickling

And the dict subclass pickles just fine:

>>> import pickle
>>> pickle.dumps(ld)
b'\x80\x03c__main__\nLowerDict\nq\x00)\x81q\x01X\x03\x00\x00\x00fooq\x02Ns.'
>>> pickle.loads(pickle.dumps(ld))
{'foo': None}
>>> type(pickle.loads(pickle.dumps(ld)))
<class '__main__.LowerDict'>

__repr__

Do I need repr, update and __init__?

We defined update and __init__, but you have a beautiful __repr__ by default:

>>> ld # without __repr__ defined for the class, we get this
{'foo': None}

However, it’s good to write a __repr__ to improve the debugability of your code. The ideal test is eval(repr(obj)) == obj. If it’s easy to do for your code, I strongly recommend it:

>>> ld = LowerDict({})
>>> eval(repr(ld)) == ld
True
>>> ld = LowerDict(dict(a=1, b=2, c=3))
>>> eval(repr(ld)) == ld
True

You see, it’s exactly what we need to recreate an equivalent object – this is something that might show up in our logs or in backtraces:

>>> ld
LowerDict({'a': 1, 'c': 3, 'b': 2})

Conclusion

Should I just use mutablemapping (it seems one shouldn’t use UserDict or DictMixin)? If so, how? The docs aren’t exactly enlightening.

Yeah, these are a few more lines of code, but they’re intended to be comprehensive. My first inclination would be to use the accepted answer, and if there were issues with it, I’d then look at my answer – as it’s a little more complicated, and there’s no ABC to help me get my interface right.

Premature optimization is going for greater complexity in search of performance. MutableMapping is simpler – so it gets an immediate edge, all else being equal. Nevertheless, to lay out all the differences, let’s compare and contrast.

I should add that there was a push to put a similar dictionary into the collections module, but it was rejected. You should probably just do this instead:

my_dict[transform(key)]

It should be far more easily debugable.

Compare and contrast

There are 6 interface functions implemented with the MutableMapping (which is missing fromkeys) and 11 with the dict subclass. I don’t need to implement __iter__ or __len__, but instead I have to implement get, setdefault, pop, update, copy, __contains__, and fromkeys – but these are fairly trivial, since I can use inheritance for most of those implementations.

The MutableMapping implements some things in Python that dict implements in C – so I would expect a dict subclass to be more performant in some cases.

We get a free __eq__ in both approaches – both of which assume equality only if another dict is all lowercase – but again, I think the dict subclass will compare more quickly.

Summary:

  • subclassing MutableMapping is simpler with fewer opportunities for bugs, but slower, takes more memory (see redundant dict), and fails isinstance(x, dict)
  • subclassing dict is faster, uses less memory, and passes isinstance(x, dict), but it has greater complexity to implement.

Which is more perfect? That depends on your definition of perfect.


回答 2

我的要求比较严格:

  • 我必须保留大小写信息(字符串是显示给用户的文件的路径,但这是Windows应用程序,因此内部所有操作都必须区分大小写)
  • 我需要密钥尽可能小(它确实在内存性能上有所作为,从370中砍掉了110 mb)。这意味着不能缓存键的小写版本。
  • 我需要尽快创建数据结构(这次再次改变了性能,提高了速度)。我不得不去一个内置的

我最初的想法是用笨拙的Path类代替不区分大小写的unicode子类-但是:

  • 事实证明很难做到这一点-参见:python中不区分大小写的字符串类
  • 事实证明,显式的dict键处理使代码变得冗长而混乱,并且容易出错(结构前后传递,并且不清楚它们是否具有CIStr实例作为键/元素,容易忘记some_dict[CIstr(path)],而且很难看)

因此,我最终不得不写下不区分大小写的字典。感谢@AaronHall 编写的代码,它简化了10倍。

class CIstr(unicode):
    """See https://stackoverflow.com/a/43122305/281545, especially for inlines"""
    __slots__ = () # does make a difference in memory performance

    #--Hash/Compare
    def __hash__(self):
        return hash(self.lower())
    def __eq__(self, other):
        if isinstance(other, CIstr):
            return self.lower() == other.lower()
        return NotImplemented
    def __ne__(self, other):
        if isinstance(other, CIstr):
            return self.lower() != other.lower()
        return NotImplemented
    def __lt__(self, other):
        if isinstance(other, CIstr):
            return self.lower() < other.lower()
        return NotImplemented
    def __ge__(self, other):
        if isinstance(other, CIstr):
            return self.lower() >= other.lower()
        return NotImplemented
    def __gt__(self, other):
        if isinstance(other, CIstr):
            return self.lower() > other.lower()
        return NotImplemented
    def __le__(self, other):
        if isinstance(other, CIstr):
            return self.lower() <= other.lower()
        return NotImplemented
    #--repr
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__,
                                 super(CIstr, self).__repr__())

def _ci_str(maybe_str):
    """dict keys can be any hashable object - only call CIstr if str"""
    return CIstr(maybe_str) if isinstance(maybe_str, basestring) else maybe_str

class LowerDict(dict):
    """Dictionary that transforms its keys to CIstr instances.
    Adapted from: https://stackoverflow.com/a/39375731/281545
    """
    __slots__ = () # no __dict__ - that would be redundant

    @staticmethod # because this doesn't make sense as a global function.
    def _process_args(mapping=(), **kwargs):
        if hasattr(mapping, 'iteritems'):
            mapping = getattr(mapping, 'iteritems')()
        return ((_ci_str(k), v) for k, v in
                chain(mapping, getattr(kwargs, 'iteritems')()))
    def __init__(self, mapping=(), **kwargs):
        # dicts take a mapping or iterable as their optional first argument
        super(LowerDict, self).__init__(self._process_args(mapping, **kwargs))
    def __getitem__(self, k):
        return super(LowerDict, self).__getitem__(_ci_str(k))
    def __setitem__(self, k, v):
        return super(LowerDict, self).__setitem__(_ci_str(k), v)
    def __delitem__(self, k):
        return super(LowerDict, self).__delitem__(_ci_str(k))
    def copy(self): # don't delegate w/ super - dict.copy() -> dict :(
        return type(self)(self)
    def get(self, k, default=None):
        return super(LowerDict, self).get(_ci_str(k), default)
    def setdefault(self, k, default=None):
        return super(LowerDict, self).setdefault(_ci_str(k), default)
    __no_default = object()
    def pop(self, k, v=__no_default):
        if v is LowerDict.__no_default:
            # super will raise KeyError if no default and key does not exist
            return super(LowerDict, self).pop(_ci_str(k))
        return super(LowerDict, self).pop(_ci_str(k), v)
    def update(self, mapping=(), **kwargs):
        super(LowerDict, self).update(self._process_args(mapping, **kwargs))
    def __contains__(self, k):
        return super(LowerDict, self).__contains__(_ci_str(k))
    @classmethod
    def fromkeys(cls, keys, v=None):
        return super(LowerDict, cls).fromkeys((_ci_str(k) for k in keys), v)
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__,
                                 super(LowerDict, self).__repr__())

隐式还是显式仍然是一个问题,但是一旦尘埃落定,就重命名属性/变量以ci开头(以及大量的doc注释说明ci代表不区分大小写),我认为这是一个完美的解决方案-因为代码的读者必须充分意识到我们正在处理不区分大小写的基础数据结构。希望这将修复一些难以重现的错误,我怀疑这些错误归结为区分大小写。

欢迎评论/更正:)

My requirements were a bit stricter:

  • I had to retain case info (the strings are paths to files displayed to the user, but it’s a windows app so internally all operations must be case insensitive)
  • I needed keys to be as small as possible (it did make a difference in memory performance, chopped off 110 mb out of 370). This meant that caching lowercase version of keys is not an option.
  • I needed creation of the data structures to be as fast as possible (again made a difference in performance, speed this time). I had to go with a builtin

My initial thought was to substitute our clunky Path class for a case insensitive unicode subclass – but:

  • proved hard to get that right – see: A case insensitive string class in python
  • turns out that explicit dict keys handling makes code verbose and messy – and error prone (structures are passed hither and thither, and it is not clear if they have CIStr instances as keys/elements, easy to forget plus some_dict[CIstr(path)] is ugly)

So I had finally to write down that case insensitive dict. Thanks to code by @AaronHall that was made 10 times easier.

class CIstr(unicode):
    """See https://stackoverflow.com/a/43122305/281545, especially for inlines"""
    __slots__ = () # does make a difference in memory performance

    #--Hash/Compare
    def __hash__(self):
        return hash(self.lower())
    def __eq__(self, other):
        if isinstance(other, CIstr):
            return self.lower() == other.lower()
        return NotImplemented
    def __ne__(self, other):
        if isinstance(other, CIstr):
            return self.lower() != other.lower()
        return NotImplemented
    def __lt__(self, other):
        if isinstance(other, CIstr):
            return self.lower() < other.lower()
        return NotImplemented
    def __ge__(self, other):
        if isinstance(other, CIstr):
            return self.lower() >= other.lower()
        return NotImplemented
    def __gt__(self, other):
        if isinstance(other, CIstr):
            return self.lower() > other.lower()
        return NotImplemented
    def __le__(self, other):
        if isinstance(other, CIstr):
            return self.lower() <= other.lower()
        return NotImplemented
    #--repr
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__,
                                 super(CIstr, self).__repr__())

def _ci_str(maybe_str):
    """dict keys can be any hashable object - only call CIstr if str"""
    return CIstr(maybe_str) if isinstance(maybe_str, basestring) else maybe_str

class LowerDict(dict):
    """Dictionary that transforms its keys to CIstr instances.
    Adapted from: https://stackoverflow.com/a/39375731/281545
    """
    __slots__ = () # no __dict__ - that would be redundant

    @staticmethod # because this doesn't make sense as a global function.
    def _process_args(mapping=(), **kwargs):
        if hasattr(mapping, 'iteritems'):
            mapping = getattr(mapping, 'iteritems')()
        return ((_ci_str(k), v) for k, v in
                chain(mapping, getattr(kwargs, 'iteritems')()))
    def __init__(self, mapping=(), **kwargs):
        # dicts take a mapping or iterable as their optional first argument
        super(LowerDict, self).__init__(self._process_args(mapping, **kwargs))
    def __getitem__(self, k):
        return super(LowerDict, self).__getitem__(_ci_str(k))
    def __setitem__(self, k, v):
        return super(LowerDict, self).__setitem__(_ci_str(k), v)
    def __delitem__(self, k):
        return super(LowerDict, self).__delitem__(_ci_str(k))
    def copy(self): # don't delegate w/ super - dict.copy() -> dict :(
        return type(self)(self)
    def get(self, k, default=None):
        return super(LowerDict, self).get(_ci_str(k), default)
    def setdefault(self, k, default=None):
        return super(LowerDict, self).setdefault(_ci_str(k), default)
    __no_default = object()
    def pop(self, k, v=__no_default):
        if v is LowerDict.__no_default:
            # super will raise KeyError if no default and key does not exist
            return super(LowerDict, self).pop(_ci_str(k))
        return super(LowerDict, self).pop(_ci_str(k), v)
    def update(self, mapping=(), **kwargs):
        super(LowerDict, self).update(self._process_args(mapping, **kwargs))
    def __contains__(self, k):
        return super(LowerDict, self).__contains__(_ci_str(k))
    @classmethod
    def fromkeys(cls, keys, v=None):
        return super(LowerDict, cls).fromkeys((_ci_str(k) for k in keys), v)
    def __repr__(self):
        return '{0}({1})'.format(type(self).__name__,
                                 super(LowerDict, self).__repr__())

Implicit vs explicit is still a problem, but once dust settles, renaming of attributes/variables to start with ci (and a big fat doc comment explaining that ci stands for case insensitive) I think is a perfect solution – as readers of the code must be fully aware that we are dealing with case insensitive underlying data structures. This will hopefully fix some hard to reproduce bugs, which I suspect boil down to case sensitivity.

Comments/corrections welcome :)


回答 3

您要做的就是

class BatchCollection(dict):
    def __init__(self, *args, **kwargs):
        dict.__init__(*args, **kwargs)

要么

class BatchCollection(dict):
    def __init__(self, inpt={}):
        super(BatchCollection, self).__init__(inpt)

我个人使用的样本用法

### EXAMPLE
class BatchCollection(dict):
    def __init__(self, inpt={}):
        dict.__init__(*args, **kwargs)

    def __setitem__(self, key, item):
        if (isinstance(key, tuple) and len(key) == 2
                and isinstance(item, collections.Iterable)):
            # self.__dict__[key] = item
            super(BatchCollection, self).__setitem__(key, item)
        else:
            raise Exception(
                "Valid key should be a tuple (database_name, table_name) "
                "and value should be iterable")

注意:仅在python3中测试

All you will have to do is

class BatchCollection(dict):
    def __init__(self, *args, **kwargs):
        dict.__init__(*args, **kwargs)

OR

class BatchCollection(dict):
    def __init__(self, inpt={}):
        super(BatchCollection, self).__init__(inpt)

A sample usage for my personal use

### EXAMPLE
class BatchCollection(dict):
    def __init__(self, inpt={}):
        dict.__init__(*args, **kwargs)

    def __setitem__(self, key, item):
        if (isinstance(key, tuple) and len(key) == 2
                and isinstance(item, collections.Iterable)):
            # self.__dict__[key] = item
            super(BatchCollection, self).__setitem__(key, item)
        else:
            raise Exception(
                "Valid key should be a tuple (database_name, table_name) "
                "and value should be iterable")

Note: tested only in python3


回答 4

尝试了两者的后顶部 2的建议,我已经定居在为Python 2.7黑幕,看中间路线。也许3更聪明,但对我来说:

class MyDict(MutableMapping):
   # ... the few __methods__ that mutablemapping requires
   # and then this monstrosity
   @property
   def __class__(self):
       return dict

我真的很讨厌,但似乎符合我的需求,这些需求是:

  • 可以覆盖 **my_dict
    • 如果您从继承dict则绕过您的代码。试试看。
    • 这使得#2 一直都是我无法接受的,因为这在python代码中很常见
  • 伪装成 isinstance(my_dict, dict)
    • 仅排除MutableMapping,所以#1是不够的
    • 我衷心推荐#1,如果您不需要的话,它既简单又可预测
  • 完全可控的行为
    • 所以我不能继承 dict

如果您需要与其他人区分开来,我个人使用这样的名称(尽管我会建议使用更好的名称):

def __am_i_me(self):
  return True

@classmethod
def __is_it_me(cls, other):
  try:
    return other.__am_i_me()
  except Exception:
    return False

只要您只需要在内部识别自己,这种方式就很难__am_i_me因python的名称更改(这_MyDict__am_i_me从此类外部的任何调用重命名)而意外调用。_method在实践和文化上都比s 私密一些。

到目前为止,除了看上去非常阴暗的__class__覆盖之外,我还没有任何抱怨。我很高兴听到别人遇到的任何问题,但我不完全了解后果。但是到目前为止,我还没有遇到任何问题,这使我可以在很多位置迁移很多中等质量的代码,而无需进行任何更改。


作为证据:https : //repl.it/repls/TraumaticToughCockatoo

基本上:复制当前的#2选项print 'method_name'向每个方法添加行,然后尝试执行此操作并观察输出:

d = LowerDict()  # prints "init", or whatever your print statement said
print '------'
splatted = dict(**d)  # note that there are no prints here

您将在其他情况下看到类似的行为。假设您的伪造品dict是其他数据类型的包装,因此没有合理的方法将数据存储在后备字典中;**your_dict不管其他方法做什么,它将为空。

这适用于MutableMapping,但是一旦您继承dict它就变得不可控制。


编辑:作为更新,它已经运行了将近两年没有出现任何问题,使用了数十万行(可能是几百万行)复杂的,遗留了很多经验的python。所以我对此很满意:)

编辑2:很显然,我很早以前就把它复印了。 @classmethod __class__不适用于isinstance支票- @property __class__可以:https : //repl.it/repls/UnitedScientificSequence

After trying out both of the top two suggestions, I’ve settled on a shady-looking middle route for Python 2.7. Maybe 3 is saner, but for me:

class MyDict(MutableMapping):
   # ... the few __methods__ that mutablemapping requires
   # and then this monstrosity
   @property
   def __class__(self):
       return dict

which I really hate, but seems to fit my needs, which are:

  • can override **my_dict
    • if you inherit from dict, this bypasses your code. try it out.
    • this makes #2 unacceptable for me at all times, as this is quite common in python code
  • masquerades as isinstance(my_dict, dict)
    • rules out MutableMapping alone, so #1 is not enough
    • I heartily recommend #1 if you don’t need this, it’s simple and predictable
  • fully controllable behavior
    • so I cannot inherit from dict

If you need to tell yourself apart from others, personally I use something like this (though I’d recommend better names):

def __am_i_me(self):
  return True

@classmethod
def __is_it_me(cls, other):
  try:
    return other.__am_i_me()
  except Exception:
    return False

As long as you only need to recognize yourself internally, this way it’s harder to accidentally call __am_i_me due to python’s name-munging (this is renamed to _MyDict__am_i_me from anything calling outside this class). Slightly more private than _methods, both in practice and culturally.

So far I have no complaints, aside from the seriously-shady-looking __class__ override. I’d be thrilled to hear of any problems that others encounter with this though, I don’t fully understand the consequences. But so far I’ve had no problems whatsoever, and this allowed me to migrate a lot of middling-quality code in lots of locations without needing any changes.


As evidence: https://repl.it/repls/TraumaticToughCockatoo

Basically: copy the current #2 option, add print 'method_name' lines to every method, and then try this and watch the output:

d = LowerDict()  # prints "init", or whatever your print statement said
print '------'
splatted = dict(**d)  # note that there are no prints here

You’ll see similar behavior for other scenarios. Say your fake-dict is a wrapper around some other datatype, so there’s no reasonable way to store the data in the backing-dict; **your_dict will be empty, regardless of what every other method does.

This works correctly for MutableMapping, but as soon as you inherit from dict it becomes uncontrollable.


Edit: as an update, this has been running without a single issue for almost two years now, on several hundred thousand (eh, might be a couple million) lines of complicated, legacy-ridden python. So I’m pretty happy with it :)

Edit 2: apparently I mis-copied this or something long ago. @classmethod __class__ does not work for isinstance checks – @property __class__ does: https://repl.it/repls/UnitedScientificSequence