标签归档:static-analysis

如何分析Python代码以找出问题区域?

问题:如何分析Python代码以找出问题区域?

我有一个跨多个项目的大型资源库。我想生成一份有关源代码运行状况的报告,以确定需要解决的问题区域。

具体来说,我想找出循环复杂度高的例程,确定重复性,并可能进行一些类似于皮棉的静态分析,以发现可疑的(因而可能是错误的)构造。

我将如何构建这样的报告?

I have a large source repository split across multiple projects. I would like to produce a report about the health of the source code, identifying problem areas that need to be addressed.

Specifically, I’d like to call out routines with a high cyclomatic complexity, identify repetition, and perhaps run some lint-like static analysis to spot suspicious (and thus likely erroneous) constructs.

How might I go about constructing such a report?


回答 0

为了测量圈复杂度,traceback.org提供了一个不错的工具。该页面还很好地概述了如何解释结果。

+1为pylint。它非常适合验证是否遵守编码标准(无论是PEP8还是您自己组织的变体),最终可以帮助降低循环复杂性。

For measuring cyclomatic complexity, there’s a nice tool available at traceback.org. The page also gives a good overview of how to interpret the results.

+1 for pylint. It is great at verifying adherence to coding standards (be it PEP8 or your own organization’s variant), which can in the end help to reduce cyclomatic complexity.


回答 1

对于圈复杂度,可以使用radonhttps : //github.com/rubik/radon

(使用pip安装它:pip install radon

此外,它还具有以下功能:

  • 原始指标(包括SLOC,注释行,空白行等)
  • Halstead指标(所有指标)
  • 可维护性指数(在Visual Studio中使用的指数)

For cyclomatic complexity you can use radon: https://github.com/rubik/radon

(Use pip to install it: pip install radon)

Additionally it also has these features:

  • raw metrics (these include SLOC, comment lines, blank lines, &c.)
  • Halstead metrics (all of them)
  • Maintainability Index (the one used in Visual Studio)

回答 2

对于静态分析,有pylintpychecker。我个人使用pylint,因为它似乎比pychecker更全面。

对于圈复杂度,您可以尝试使用此perl程序,或者本文介绍了python程序来执行相同的操作

For static analysis there is pylint and pychecker. Personally I use pylint as it seems to be more comprehensive than pychecker.

For cyclomatic complexity you can try this perl program, or this article which introduces a python program to do the same


回答 3

当您需要了解新项目时,Pycana就像魅力一样工作!

PyCAna(Python代码分析器)是一个简单的python代码分析器的名字,它在执行代码后创建类图。

看看它是如何工作的:http : //pycana.sourceforge.net/

输出:

Pycana works like charm when you need to understand a new project!

PyCAna (Python Code Analyzer) is a fancy name for a simple code analyzer for python that creates a class diagram after executing your code.

See how it works: http://pycana.sourceforge.net/

output:


回答 4

感谢Pydev,您可以真正轻松地将pylint集成Eclipse IDE中,并在每次保存修改后的文件时获得代码报告。

Thanks to Pydev, you can integrate pylint in the Eclipse IDE really easily and get a code report each time you save a modified file.


回答 5

使用flake8,在一个工具中提供pep8,pyflakes和循环复杂性分析

Use flake8, which provides pep8, pyflakes, and cyclomatic complexity analysis in one tool


回答 6

有一个名为CloneDigger的工具 ,可以帮助您找到类似的代码片段。

There is a tool called CloneDigger that helps you find similar code snippets.


回答 7

为了检查圈复杂度,当然有mccabe包装。

安装:

$ pip install --upgrade mccabe

用法:

$ python -m mccabe --min=6 path/to/myfile.py

请注意上面的阈值6。根据这个答案,分数> 5可能应该简化。

输出示例--min=3

68:1: 'Fetcher.fetch' 3
48:1: 'Fetcher._read_dom_tag' 3
103:1: 'main' 3

也可以通过pylint-mccabepytest-mccabe等使用它。

For checking cyclomatic complexity, there is of course the mccabe package.

Installation:

$ pip install --upgrade mccabe

Usage:

$ python -m mccabe --min=6 path/to/myfile.py

Note the threshold of 6 above. Per this answer, scores >5 probably should be simplified.

Sample output with --min=3:

68:1: 'Fetcher.fetch' 3
48:1: 'Fetcher._read_dom_tag' 3
103:1: 'main' 3

It can optionally also be used via pylint-mccabe or pytest-mccabe, etc.


在Django中使用Pylint

问题:在Django中使用Pylint

我非常想将pylint集成到我的python项目的构建过程中,但是我遇到了一个问题:我发现一种非常有用的错误类型:– E1101: *%s %r has no %r member*在使用通用django字段时不断报告错误, 例如:

E1101:125:get_user_tags: Class 'Tag' has no 'objects' member

这是由以下代码引起的:

def get_user_tags(username):
   """
   Gets all the tags that username has used.

   Returns a query set.
   """
   return Tag.objects.filter(  ## This line triggers the error.
       tagownership__users__username__exact=username).distinct()

# Here is the Tag class, models.Model is provided by Django:
class Tag(models.Model):
   """
   Model for user-defined strings that help categorize Events on
   on a per-user basis.
   """
   name = models.CharField(max_length=500, null=False, unique=True)

   def __unicode__(self):
       return self.name

如何调整Pylint以适当考虑诸如对象之类的字段?(我也研究了Django的源代码,但无法找到的实现objects,因此我怀疑它不是“只是”一个类字段。另一方面,我对python还是很陌生,所以我可能已经忽略了一些东西。)

编辑:我发现告诉pylint不要警告这些警告的唯一方法是阻止所有类型为(E1101)的错误,这是不可接受的解决方案,因为(在我看来)这是一个非常有用的错误。如果还有另一种方法,而又不增加pylint的来源,请给我指出细节:)

这里对于我已经受够了问题的总结pycheckerpyflakes-他们已经被证明是远远不稳定的一般用途。(在pychecker的情况下,崩溃源于pychecker代码-并非源于正在加载/调用的源。)

I would very much like to integrate pylint into the build process for my python projects, but I have run into one show-stopper: One of the error types that I find extremely useful–:E1101: *%s %r has no %r member*–constantly reports errors when using common django fields, for example:

E1101:125:get_user_tags: Class 'Tag' has no 'objects' member

which is caused by this code:

def get_user_tags(username):
   """
   Gets all the tags that username has used.

   Returns a query set.
   """
   return Tag.objects.filter(  ## This line triggers the error.
       tagownership__users__username__exact=username).distinct()

# Here is the Tag class, models.Model is provided by Django:
class Tag(models.Model):
   """
   Model for user-defined strings that help categorize Events on
   on a per-user basis.
   """
   name = models.CharField(max_length=500, null=False, unique=True)

   def __unicode__(self):
       return self.name

How can I tune Pylint to properly take fields such as objects into account? (I’ve also looked into the Django source, and I have been unable to find the implementation of objects, so I suspect it is not “just” a class field. On the other hand, I’m fairly new to python, so I may very well have overlooked something.)

Edit: The only way I’ve found to tell pylint to not warn about these warnings is by blocking all errors of the type (E1101) which is not an acceptable solution, since that is (in my opinion) an extremely useful error. If there is another way, without augmenting the pylint source, please point me to specifics :)

See here for a summary of the problems I’ve had with pychecker and pyflakes — they’ve proven to be far to unstable for general use. (In pychecker’s case, the crashes originated in the pychecker code — not source it was loading/invoking.)


回答 0

不要通过添加ignores或禁用或削弱Pylint功能generated-members
使用积极开发的理解 Django的Pylint插件。
这个适用于Django的Pylint插件效果很好:

pip install pylint-django

并且在运行pylint时,将以下标志添加到命令中:

--load-plugins pylint_django

详细的博客文章在这里

Do not disable or weaken Pylint functionality by adding ignores or generated-members.
Use an actively developed Pylint plugin that understands Django.
This Pylint plugin for Django works quite well:

pip install pylint-django

and when running pylint add the following flag to the command:

--load-plugins pylint_django

Detailed blog post here.


回答 1

我使用以下内容: pylint --generated-members=objects

I use the following: pylint --generated-members=objects


回答 2

我的〜/ .pylintrc包含

[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id

最后两个是专门针对Django的。

请注意,PyLint 0.21.1中存在一个错误,需要对其进行修补才能使其正常工作。

编辑:弄乱了一点之后,我决定对PyLint进行一点修改,以允许我将以上内容扩展为:

[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id,[a-zA-Z]+_set

我只是添加:

    import re
    for pattern in self.config.generated_members:
        if re.match(pattern, node.attrname):
            return

在错误报告中提到的修复之后(即,在第129行)。

快乐的时光!

My ~/.pylintrc contains

[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id

the last two are specifically for Django.

Note that there is a bug in PyLint 0.21.1 which needs patching to make this work.

Edit: After messing around with this a little more, I decided to hack PyLint just a tiny bit to allow me to expand the above into:

[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id,[a-zA-Z]+_set

I simply added:

    import re
    for pattern in self.config.generated_members:
        if re.match(pattern, node.attrname):
            return

after the fix mentioned in the bug report (i.e., at line 129).

Happy days!


回答 3

如果您使用Visual Studio Code,请执行以下操作:

pip install pylint-django

并添加到VSC配置:

"python.linting.pylintArgs": [
    "--load-plugins=pylint_django"
],

If you use Visual Studio Code do this:

pip install pylint-django

And add to VSC config:

"python.linting.pylintArgs": [
    "--load-plugins=pylint_django"
],

回答 4

django-lint是一个不错的工具,它使用django特定的设置包装pylint:http ://chris-lamb.co.uk/projects/django-lint/

github项目:https : //github.com/lamby/django-lint

django-lint is a nice tool which wraps pylint with django specific settings : http://chris-lamb.co.uk/projects/django-lint/

github project: https://github.com/lamby/django-lint


回答 5

由于pylint的工作方式(它在不让Python实际执行的情况下检查源代码本身)对pylint来说很难弄清楚元类和复杂的基类实际上如何影响类及其实例。在这方面,“ pychecker”工具要好一些,因为它确实允许Python执行代码。它导入模块并检查生成的对象。但是,该方法还有其他问题,因为它确实允许Python执行代码:-)

您可以扩展pylint来教它有关Django所使用的魔术的知识,或者使它更好地理解元类或复杂的基类,或者在检测到一个或多个它不太了解的功能之后忽略这些情况。我认为这不会特别容易。您还可以通过源代码中的特殊注释,命令行选项或.pylintrc文件,告诉pylint不要警告这些事情。

Because of how pylint works (it examines the source itself, without letting Python actually execute it) it’s very hard for pylint to figure out how metaclasses and complex baseclasses actually affect a class and its instances. The ‘pychecker’ tool is a bit better in this regard, because it does actually let Python execute the code; it imports the modules and examines the resulting objects. However, that approach has other problems, because it does actually let Python execute the code :-)

You could extend pylint to teach it about the magic Django uses, or to make it understand metaclasses or complex baseclasses better, or to just ignore such cases after detecting one or more features it doesn’t quite understand. I don’t think it would be particularly easy. You can also just tell pylint to not warn about these things, through special comments in the source, command-line options or a .pylintrc file.


回答 6

我从使用pylint / pychecker辞职,转而将pyflakes与Django代码一起使用-它只是尝试导入模块并报告发现的任何问题,例如未使用的导入或未初始化的本地名称。

I resigned from using pylint/pychecker in favor of using pyflakes with Django code – it just tries to import module and reports any problem it finds, like unused imports or uninitialized local names.


回答 7

这不是解决方案,但是您可以在objects = models.Manager()不更改任何行为的情况下将其添加到Django模型中。

我本人只使用pyflakes,主要是因为我的部分pylint和惰性有些愚蠢的默认设置(不想查看如何更改默认设置)。

This is not a solution, but you can add objects = models.Manager() to your Django models without changing any behavior.

I myself only use pyflakes, primarily due to some dumb defaults in pylint and laziness on my part (not wanting to look up how to change the defaults).


回答 8

尝试与运行pylint

pylint --ignored-classes=Tags

如果可行,添加所有其他Django类-可能使用脚本,例如python:P

该文档--ignore-classes是:

--ignored-classes=<members names>
不应检查其成员属性的类名列表(对于动态设置了属性的类很有用)。[当前:%default]

我认为这并不是一个特别优雅的解决方案,但应该可以。

Try running pylint with

pylint --ignored-classes=Tags

If that works, add all the other Django classes – possibly using a script, in say, python :P

The documentation for --ignore-classes is:

--ignored-classes=<members names>
List of classes names for which member attributes should not be checked (useful for classes with attributes dynamicaly set). [current: %default]

I should add this is not a particular elegant solution in my view, but it should work.


回答 9

另一个问题中提出的解决方案是,只需将get_attr添加到Tag类中即可。丑陋,但是行得通。

The solution proposed in this other question it to simply add get_attr to your Tag class. Ugly, but works.


回答 10

到目前为止,我没有找到真正的解决方案,但是可以解决:

  • 在我们公司,我们要求pylint得分>8。这允许pylint无法理解编码实践,同时确保代码不太“异常”。到目前为止,我们还没有看到E1101阻止我们达到8分或更高分数的情况。
  • 我们的“ make check”目标过滤掉“ for has no’objects member”消息,以消除由于pylint不了解Django引起的大部分干扰。

So far I have found no real solution to that but work around:

  • In our company we require a pylint score > 8. This allows coding practices pylint doesn’t understand while ensuring that the code isn’t too “unusual”. So far we havn’t seen any instance where E1101 kept us from reaching a score of 8 or higher.
  • Our ‘make check’ targets filter out “for has no ‘objects’ member” messages to remove most of the distraction caused by pylint not understanding Django.

回答 11

对于neovim & vim8使用w0rp's ale插件。如果您已经安装了正确的一切,包括w0rp's alepylintpylint-django。在您的代码中vimrc添加以下内容,并享受使用django开发Web应用程序的乐趣。谢谢。

let g:ale_python_pylint_options = '--load-plugins pylint_django'

For neovim & vim8 use w0rp's ale plugin. If you have installed everything correctly including w0rp's ale, pylint & pylint-django. In your vimrc add the following line & have fun developing web apps using django. Thanks.

let g:ale_python_pylint_options = '--load-plugins pylint_django'