标签归档:buildbot

Python的“漂亮”持续集成

问题:Python的“漂亮”持续集成

这是一个..徒劳的问题,但是BuildBot的输出并不是特别好看。

例如,相比

..及其他,BuildBot看起来..古老

我目前正在与Hudson一起玩,但是它是非常以Java为中心的(尽管使用本指南,我发现它比BuildBot容易设置,并提供了更多信息)

基本上:是否有任何针对python的持续集成系统,它们会生成许多闪亮的图形等?


更新:自从这次以来,Jenkins项目已将Hudson替换为软件包的社区版本。原始作者也已移至该项目。Jenkins现在是Ubuntu / Debian,RedHat / Fedora / CentOS等上的标准软件包。以下更新本质上仍然正确。詹金斯做到这一点的起点是不同的。

更新:尝试了几种选择之后,我认为我会坚持使用哈德森。完整性很好而且很简单,但是非常有限。我认为 Buildbot更适合拥有多个构建从属,而不是像我在使用它那样在一台机器上运行的所有东西。

将Hudson设置为Python项目非常简单:

  • http://hudson-ci.org/下载Hudson
  • 运行它 java -jar hudson.war
  • 打开Web界面的默认地址为 http://localhost:8080
  • 转到管理哈德森,插件,单击“更新”或类似内容
  • 安装Git插件(我必须git在Hudson全局首选项中设置路径)
  • 创建一个新项目,输入存储库,SCM轮询间隔等
  • 如果尚未安装,请nosetests通过安装easy_install
  • 在构建步骤中,添加 nosetests --with-xunit --verbose
  • 选中“发布JUnit测试结果报告”并将“测试报告XML”设置为 **/nosetests.xml

这就是全部。您可以设置电子邮件通知,这些插件值得一看。我目前正在使用一些Python项目:

  • SLOCCount插件可以计算代码行(并绘制图形!)-您需要单独安装sloccount
  • 违反解析PyLint输出(您可以设置警告阈值,绘制每个构建中违反次数的图表)
  • Cobertura可以解析coverage.py的输出。Nosetest可以在运行测试时使用收集覆盖范围nosetests --with-coverage(将输出写入**/coverage.xml

This is a slightly.. vain question, but BuildBot’s output isn’t particularly nice to look at..

For example, compared to..

..and others, BuildBot looks rather.. archaic

I’m currently playing with Hudson, but it is very Java-centric (although with this guide, I found it easier to setup than BuildBot, and produced more info)

Basically: is there any Continuous Integration systems aimed at python, that produce lots of shiny graphs and the likes?


Update: Since this time the Jenkins project has replaced Hudson as the community version of the package. The original authors have moved to this project as well. Jenkins is now a standard package on Ubuntu/Debian, RedHat/Fedora/CentOS, and others. The following update is still essentially correct. The starting point to do this with Jenkins is different.

Update: After trying a few alternatives, I think I’ll stick with Hudson. Integrity was nice and simple, but quite limited. I think Buildbot is better suited to having numerous build-slaves, rather than everything running on a single machine like I was using it.

Setting Hudson up for a Python project was pretty simple:

  • Download Hudson from http://hudson-ci.org/
  • Run it with java -jar hudson.war
  • Open the web interface on the default address of http://localhost:8080
  • Go to Manage Hudson, Plugins, click “Update” or similar
  • Install the Git plugin (I had to set the git path in the Hudson global preferences)
  • Create a new project, enter the repository, SCM polling intervals and so on
  • Install nosetests via easy_install if it’s not already
  • In the a build step, add nosetests --with-xunit --verbose
  • Check “Publish JUnit test result report” and set “Test report XMLs” to **/nosetests.xml

That’s all that’s required. You can setup email notifications, and the plugins are worth a look. A few I’m currently using for Python projects:

  • SLOCCount plugin to count lines of code (and graph it!) – you need to install sloccount separately
  • Violations to parse the PyLint output (you can setup warning thresholds, graph the number of violations over each build)
  • Cobertura can parse the coverage.py output. Nosetest can gather coverage while running your tests, using nosetests --with-coverage (this writes the output to **/coverage.xml)

回答 0

您可能想看看NoseXunit输出插件。您可以使用以下命令运行单元测试和覆盖率检查:

nosetests --with-xunit --enable-cover

如果您想走Jenkins路线,或者要使用其他支持JUnit测试报告的CI服务器,这将很有帮助。

同样,您可以使用Jenkins违规插件捕获pylint的输出

You might want to check out Nose and the Xunit output plugin. You can have it run your unit tests, and coverage checks with this command:

nosetests --with-xunit --enable-cover

That’ll be helpful if you want to go the Jenkins route, or if you want to use another CI server that has support for JUnit test reporting.

Similarly you can capture the output of pylint using the violations plugin for Jenkins


回答 1

不知道它是否会做:Bitten是由写Trac的人制作的,并与Trac集成在一起。Apache Gump是Apache使用的CI工具。它是用Python编写的。

Don’t know if it would do : Bitten is made by the guys who write Trac and is integrated with Trac. Apache Gump is the CI tool used by Apache. It is written in Python.


回答 2

使用TeamCity作为CI服务器并使用鼻子作为测试运行器,我们已经取得了巨大的成功。 Teamcity的鼻子测试插件可让您计数通过/失败,失败的测试的可读显示(可以通过电子邮件发送)。您甚至可以在堆栈运行时查看测试失败的详细信息。

当然,如果支持在多台计算机上运行之类的事情,则它的设置和维护比buildbot容易得多。

We’ve had great success with TeamCity as our CI server and using nose as our test runner. Teamcity plugin for nosetests gives you count pass/fail, readable display for failed test( that can be E-Mailed). You can even see details of the test failures while you stack is running.

If of course supports things like running on multiple machines, and it’s much simpler to setup and maintain than buildbot.


回答 3

Buildbot的瀑布页面可以修饰得很漂亮。这是一个很好的例子http://build.chromium.org/buildbot/waterfall/waterfall

Buildbot’s waterfall page can be considerably prettified. Here’s a nice example http://build.chromium.org/buildbot/waterfall/waterfall


回答 4

Atlassian的Bamboo也绝对值得一试。整个Atlassian套件(JIRA,Confluence,FishEye等)非常漂亮。

Atlassian’s Bamboo is also definitely worth checking out. The entire Atlassian suite (JIRA, Confluence, FishEye, etc) is pretty sweet.


回答 5

我猜这个线程已经很老了,但是这是我和哈德森的看法:

我决定与pip一起建立一个repo(上班很痛苦,但是看上去很漂亮),hudson会自动将其上传到测试成功的仓库中。这是与hudson config执行脚本一起使用的我粗略且准备就绪的脚本,例如:/var/lib/hudson/venv/main/bin/hudson_script.py -w $ WORKSPACE -p my.package -v $ BUILD_NUMBER,只需放入** / coverage.xml,pylint.txt和鼻子测试.xml在配置位中:

#!/var/lib/hudson/venv/main/bin/python
import os
import re
import subprocess
import logging
import optparse

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')

#venvDir = "/var/lib/hudson/venv/main/bin/"

UPLOAD_REPO = "http://ldndev01:3442"

def call_command(command, cwd, ignore_error_code=False):
    try:
        logging.info("Running: %s" % command)
        status = subprocess.call(command, cwd=cwd, shell=True)
        if not ignore_error_code and status != 0:
            raise Exception("Last command failed")

        return status

    except:
        logging.exception("Could not run command %s" % command)
        raise

def main():
    usage = "usage: %prog [options]"
    parser = optparse.OptionParser(usage)
    parser.add_option("-w", "--workspace", dest="workspace",
                      help="workspace folder for the job")
    parser.add_option("-p", "--package", dest="package",
                      help="the package name i.e., back_office.reconciler")
    parser.add_option("-v", "--build_number", dest="build_number",
                      help="the build number, which will get put at the end of the package version")
    options, args = parser.parse_args()

    if not options.workspace or not options.package:
        raise Exception("Need both args, do --help for info")

    venvDir = options.package + "_venv/"

    #find out if venv is there
    if not os.path.exists(venvDir):
        #make it
        call_command("virtualenv %s --no-site-packages" % venvDir,
                     options.workspace)

    #install the venv/make sure its there plus install the local package
    call_command("%sbin/pip install -e ./ --extra-index %s" % (venvDir, UPLOAD_REPO),
                 options.workspace)

    #make sure pylint, nose and coverage are installed
    call_command("%sbin/pip install nose pylint coverage epydoc" % venvDir,
                 options.workspace)

    #make sure we have an __init__.py
    #this shouldn't be needed if the packages are set up correctly
    #modules = options.package.split(".")
    #if len(modules) > 1: 
    #    call_command("touch '%s/__init__.py'" % modules[0], 
    #                 options.workspace)
    #do the nosetests
    test_status = call_command("%sbin/nosetests %s --with-xunit --with-coverage --cover-package %s --cover-erase" % (venvDir,
                                                                                     options.package.replace(".", "/"),
                                                                                     options.package),
                 options.workspace, True)
    #produce coverage report -i for ignore weird missing file errors
    call_command("%sbin/coverage xml -i" % venvDir,
                 options.workspace)
    #move it so that the code coverage plugin can find it
    call_command("mv coverage.xml %s" % (options.package.replace(".", "/")),
                 options.workspace)
    #run pylint
    call_command("%sbin/pylint --rcfile ~/pylint.rc -f parseable %s > pylint.txt" % (venvDir, 
                                                                                     options.package),
                 options.workspace, True)

    #remove old dists so we only have the newest at the end
    call_command("rm -rfv %s" % (options.workspace + "/dist"),
                 options.workspace)

    #if the build passes upload the result to the egg_basket
    if test_status == 0:
        logging.info("Success - uploading egg")
        upload_bit = "upload -r %s/upload" % UPLOAD_REPO
    else:
        logging.info("Failure - not uploading egg")
        upload_bit = ""

    #create egg
    call_command("%sbin/python setup.py egg_info --tag-build=.0.%s --tag-svn-revision --tag-date sdist %s" % (venvDir,
                                                                                                              options.build_number,
                                                                                                              upload_bit),
                 options.workspace)

    call_command("%sbin/epydoc --html --graph all %s" % (venvDir, options.package),
                 options.workspace)

    logging.info("Complete")

if __name__ == "__main__":
    main()

在部署内容时,您可以执行以下操作:

pip -E /location/of/my/venv/ install my_package==X.Y.Z --extra-index http://my_repo

然后人们可以使用以下方法开发东西:

pip -E /location/of/my/venv/ install -e ./ --extra-index http://my_repo

这些东西假设您每个包都有一个带有setup.py和依赖项的仓库结构,然后就可以检出中继并在上面运行它。

我希望这可以帮助某人。

——更新———

我添加了epydoc,它非常适合hudson。只需使用html文件夹将javadoc添加到您的配置中

请注意,pip目前不正确支持-E标志,因此您必须单独创建venv

I guess this thread is quite old but here is my take on it with hudson:

I decided to go with pip and set up a repo (the painful to get working but nice looking eggbasket), which hudson auto uploads to with a successful tests. Here is my rough and ready script for use with a hudson config execute script like: /var/lib/hudson/venv/main/bin/hudson_script.py -w $WORKSPACE -p my.package -v $BUILD_NUMBER, just put in **/coverage.xml, pylint.txt and nosetests.xml in the config bits:

#!/var/lib/hudson/venv/main/bin/python
import os
import re
import subprocess
import logging
import optparse

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')

#venvDir = "/var/lib/hudson/venv/main/bin/"

UPLOAD_REPO = "http://ldndev01:3442"

def call_command(command, cwd, ignore_error_code=False):
    try:
        logging.info("Running: %s" % command)
        status = subprocess.call(command, cwd=cwd, shell=True)
        if not ignore_error_code and status != 0:
            raise Exception("Last command failed")

        return status

    except:
        logging.exception("Could not run command %s" % command)
        raise

def main():
    usage = "usage: %prog [options]"
    parser = optparse.OptionParser(usage)
    parser.add_option("-w", "--workspace", dest="workspace",
                      help="workspace folder for the job")
    parser.add_option("-p", "--package", dest="package",
                      help="the package name i.e., back_office.reconciler")
    parser.add_option("-v", "--build_number", dest="build_number",
                      help="the build number, which will get put at the end of the package version")
    options, args = parser.parse_args()

    if not options.workspace or not options.package:
        raise Exception("Need both args, do --help for info")

    venvDir = options.package + "_venv/"

    #find out if venv is there
    if not os.path.exists(venvDir):
        #make it
        call_command("virtualenv %s --no-site-packages" % venvDir,
                     options.workspace)

    #install the venv/make sure its there plus install the local package
    call_command("%sbin/pip install -e ./ --extra-index %s" % (venvDir, UPLOAD_REPO),
                 options.workspace)

    #make sure pylint, nose and coverage are installed
    call_command("%sbin/pip install nose pylint coverage epydoc" % venvDir,
                 options.workspace)

    #make sure we have an __init__.py
    #this shouldn't be needed if the packages are set up correctly
    #modules = options.package.split(".")
    #if len(modules) > 1: 
    #    call_command("touch '%s/__init__.py'" % modules[0], 
    #                 options.workspace)
    #do the nosetests
    test_status = call_command("%sbin/nosetests %s --with-xunit --with-coverage --cover-package %s --cover-erase" % (venvDir,
                                                                                     options.package.replace(".", "/"),
                                                                                     options.package),
                 options.workspace, True)
    #produce coverage report -i for ignore weird missing file errors
    call_command("%sbin/coverage xml -i" % venvDir,
                 options.workspace)
    #move it so that the code coverage plugin can find it
    call_command("mv coverage.xml %s" % (options.package.replace(".", "/")),
                 options.workspace)
    #run pylint
    call_command("%sbin/pylint --rcfile ~/pylint.rc -f parseable %s > pylint.txt" % (venvDir, 
                                                                                     options.package),
                 options.workspace, True)

    #remove old dists so we only have the newest at the end
    call_command("rm -rfv %s" % (options.workspace + "/dist"),
                 options.workspace)

    #if the build passes upload the result to the egg_basket
    if test_status == 0:
        logging.info("Success - uploading egg")
        upload_bit = "upload -r %s/upload" % UPLOAD_REPO
    else:
        logging.info("Failure - not uploading egg")
        upload_bit = ""

    #create egg
    call_command("%sbin/python setup.py egg_info --tag-build=.0.%s --tag-svn-revision --tag-date sdist %s" % (venvDir,
                                                                                                              options.build_number,
                                                                                                              upload_bit),
                 options.workspace)

    call_command("%sbin/epydoc --html --graph all %s" % (venvDir, options.package),
                 options.workspace)

    logging.info("Complete")

if __name__ == "__main__":
    main()

When it comes to deploying stuff you can do something like:

pip -E /location/of/my/venv/ install my_package==X.Y.Z --extra-index http://my_repo

And then people can develop stuff using:

pip -E /location/of/my/venv/ install -e ./ --extra-index http://my_repo

This stuff assumes you have a repo structure per package with a setup.py and dependencies all set up then you can just check out the trunk and run this stuff on it.

I hope this helps someone out.

——update———

I’ve added epydoc which fits in really nicely with hudson. Just add javadoc to your config with the html folder

Note that pip doesn’t support the -E flag properly these days, so you have to create your venv separately


回答 6

另一个:Shining Panda是python的托管工具

another one : Shining Panda is a hosted tool for python


回答 7

如果您正在考虑托管CI解决方案并进行开源,那么您也应该研究Travis CI-它与GitHub的集成非常好。当它最初是Ruby工具时,他们不久前就添加了Python支持

If you’re considering hosted CI solution, and doing open source, you should look into Travis CI as well – it has very nice integration with GitHub. While it started as a Ruby tool, they have added Python support a while ago.


回答 8

信号是另一种选择。您可以在此处了解更多信息并观看视频。

Signal is another option. You can know more about it and watch a video also here.


回答 9

我会考虑使用CircleCi-它具有强大的Python支持,并且输出非常漂亮。

I would consider CircleCi – it has great Python support, and very pretty output.


回答 10

Continuousumbinstar现在可以触发来自github的构建,并且可以针对linux,osx和Windows(32/64)进行编译。整洁的是,它确实允许您紧密耦合分发和持续集成。这是跨越t并整合I的点。该站点,工作流和工具确实经过了完善,并且AFAIK conda是分发复杂的python模块的最可靠,最pythonic的方式,您需要在其中包装分发C / C ++ / Fotran库。

continuum’s binstar now is able to trigger builds from github and can compile for linux, osx and windows ( 32 / 64 ). the neat thing is that it really allows you to closely couple distribution and continuous integration. That’s crossing the t’s and dotting the I’s of Integration. The site, workflow and tools are really polished and AFAIK conda is the most robust and pythonic way to distributing complex python modules, where you need to wrap and distribute C/C++/Fotran libraries.


回答 11

我们已经使用过很多咬了。它很漂亮,并且可以与Trac很好地集成在一起,但是如果您有任何非标准的工作流程,则很难自定义。而且,与流行工具相比,插件的数量也不多。目前,我们正在评估哈德森作为替代者。

We have used bitten quite a bit. It is pretty and integrates well with Trac, but it is a pain in the butt to customize if you have any nonstandard workflow. Also there just aren’t as many plugins as there are for the more popular tools. Currently we are evaluating Hudson as a replacement.


回答 12

检查rultor.com。如本文所述,它在每个构建中都使用Docker。因此,您可以在Docker映像中配置任何所需的内容,包括Python。

Check rultor.com. As this article explains, it uses Docker for every build. Thanks to that, you can configure whatever you like inside your Docker image, including Python.


回答 13

小小免责声明,我实际上已经为客户端构建了这样的解决方案,该客户端想要一种方法来自动测试和部署git push上的任何代码,以及通过git notes管理问题单。这也导致我从事AIMS项目

人们很容易只安装,通过有一个内置的用户和管理他们构建裸节点系统make(1)expect(1)crontab(1)/ systemd.unit(5),和incrontab(1)。甚至可以更进一步,并通过gridfs / nfs文件存储将ansible和celery用于分布式构建。

虽然,我不希望Graybeard UNIX的人或Principles级的工程师/架构师能走这么远。这只是一个好主意和潜在的学习经验,因为构建服务器无非是一种以自动化方式任意执行脚本任务的方法。

Little disclaimer, I’ve actually had to build a solution like this for a client that wanted a way to automatically test and deploy any code on a git push plus manage the issue tickets via git notes. This also lead to my work on the AIMS project.

One could easily just setup a bare node system that has a build user and manage their build through make(1), expect(1), crontab(1)/systemd.unit(5), and incrontab(1). One could even go a step further and use ansible and celery for distributed builds with a gridfs/nfs file store.

Although, I would not expect anyone other than a Graybeard UNIX guy or Principle level engineer/architect to actually go this far. Just makes for a nice idea and potential learning experience since a build server is nothing more than a way to arbitrarily execute scripted tasks in an automated fashion.