标签归档：python-rq

知识问答

使用Celery vs. RQ的利弊[关闭]

问题：使用Celery vs. RQ的利弊[关闭]

目前，我正在处理需要实施一些后台作业（主要用于电子邮件发送和大量数据库更新）的python项目。我将Redis用于任务代理。因此，在这一点上，我有两个候选人：Celery和RQ。我对这些工作队列有一定的经验，但我想请大家分享使用此工具的经验。所以。

  1. 使用Celery vs.RQ有什么优缺点。
  2. 适合使用Celery vs. RQ的项目/任务的任何示例。

Celery看起来很复杂，但是它是功能齐全的解决方案。实际上，我认为我不需要所有这些功能。从另一方面讲，RQ非常简单（例如，配置，集成），但是它似乎缺少一些有用的功能（例如，任务吊销，代码自动重载）

点击查看英文原文

Currently I’m working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience with these job queues, but I want to ask you guys to share you experience of using this tools. So.

  1. What pros and cons to use Celery vs. RQ.
  2. Any examples of projects/task suitable to use Celery vs. RQ.

Celery looks pretty complicated but it’s full featured solution. Actually I don’t think that I need all these features. From other side RQ is very simple (e.g configuration, integration), but it seems that it lacks some useful features (e.g task revoking, code auto-reloading)

回答 0

这是我在尝试回答这个完全相同的问题时发现的。它可能不全面，甚至在某些方面可能不准确。

简而言之，RQ被设计为更简单。Celery被设计为更坚固。他们俩都很棒。

  • 文档。RQ的文档全面而又不复杂，并且反映了项目的整体简单性-您永远不会感到迷茫或困惑。Celery的文档也很全面，但是在您第一次进行设置时，由于有太多可供选择的内部化选项，因此希望能重新访问它

  • 监控。Celery的Flower和RQ仪表板都很容易设置，并且至少为您提供了90％的所有信息

  • 经纪人支持。Celery无疑是赢家，RQ仅支持Redis。这意味着有关“什么是经纪人”的文档更少，但是也意味着，如果Redis不再为您服务，则您将来将无法切换经纪人。例如，Instagram在Celery中同时考虑了Redis和RabbitMQ。这很重要，因为不同的代理有不同的保证，例如，Redis 无法（截至撰写时）保证100％传递您的消息。

  • 优先级队列。RQ的优先级队列模型简单有效，工作人员按顺序从队列中读取。Celery要求纺纱多个工人从不同的队列消费。两种方法都有效

  • 操作系统支持。Celery无疑是赢家，因为RQ仅在支持forkUnix系统的系统上运行

  • 语言支持。RQ仅支持Python，而Celery允许您将任务从一种语言发送到另一种语言

  • API。Celery非常灵活（多个结果后端，漂亮的配置格式，工作流画布支持），但是自然地，这种功能可能会令人困惑。相比之下，RQ API很简单。

  • 子任务支持。Celery支持子任务（例如，从现有任务中创建新任务）。我不知道RQ是否

  • 社区与稳定。Celery可能更成熟，但它们都是活跃的项目。截至撰写本文时，Celery在Github上拥有约3500颗星，而RQ拥有约2000颗星，并且两个项目都显示出积极的发展

我认为，Celery并不像其声誉可能会让您相信的那么复杂，但是您将不得不使用RTFM。

那么，为什么有人愿意将（可能功能更全的）Celery换成RQ？在我看来，这全都归结为简单性。通过限制自己使用Redis + Unix，RQ提供了更简单的文档，更简单的代码库和更简单的API。这意味着您（以及潜在的项目贡献者）可以专注于您关心的代码，而不必在工作内存中保留有关任务队列系统的详细信息。我们每个人一次都有多少个限制，并且不再需要在其中保留任务队列详细信息，因此RQ让您回到您关心的代码。这种简单性是以语言间任务队列，广泛的操作系统支持，100％可靠的消息保证以及轻松切换消息代理的功能为代价的。

点击查看英文原文

Here is what I have found while trying to answer this exact same question. It’s probably not comprehensive, and may even be inaccurate on some points.

In short, RQ is designed to be simpler all around. Celery is designed to be more robust. They are both excellent.

  • Documentation. RQ’s documentation is comprehensive without being complex, and mirrors the project’s overall simplicity – you never feel lost or confused. Celery’s documentation is also comprehensive, but expect to be re-visiting it quite a lot when you’re first setting things up as there are too many options to internalize

  • Monitoring. Celery’s Flower and the RQ dashboard are both very simple to setup and give you at least 90% of all information you would ever want

  • Broker support. Celery is the clear winner, RQ only supports Redis. This means less documentation on “what is a broker”, but also means you cannot switch brokers in the future if Redis no longer works for you. For example, Instagram considered both Redis and RabbitMQ with Celery. This is important because different brokers have different guarantees e.g. Redis cannot (as of writing) guarantee 100% that your messages are delivered.

  • Priority queues. RQs priority queue model is simple and effective – workers read from queues in order. Celery requires spinning up multiple workers to consume from different queues. Both approaches work

  • OS Support. Celery is the clear winner here, as RQ only runs on systems that support fork e.g. Unix systems

  • Language support. RQ only supports Python, whereas Celery lets you send tasks from one language to a different language

  • API. Celery is extremely flexible (multiple result backends, nice config format, workflow canvas support) but naturally this power can be confusing. By contrast, the RQ api is simple.

  • Subtask support. Celery supports subtasks (e.g. creating new tasks from within existing tasks). I don’t know if RQ does

  • Community and Stability. Celery is probably more established, but they are both active projects. As of writing, Celery has ~3500 stars on Github while RQ has ~2000 and both projects show active development

In my opinion, Celery is not as complex as its reputation might lead you to believe, but you will have to RTFM.

So, why would anyone be willing to trade the (arguably more full-featured) Celery for RQ? In my mind, it all comes down to the simplicity. By restricting itself to Redis+Unix, RQ provides simpler documentation, simpler codebase, and a simpler API. This means you (and potential contributors to your project) can focus on the code you care about, instead of having to keep details about the task queue system in your working memory. We all have a limit on how many details can be in our head at once, and by removing the need to keep task queue details in there RQ lets get back to the code you care about. That simplicity comes at the expense of features like inter-language task queues, wide OS support, 100% reliable message guarantees, and ability to switch message brokers easily.

回答 1

Celery并不那么复杂。从本质上讲，您可以从进行逐步配置tutorials，创建一个celery实例，用装饰您的功能，@celery.task然后使用来运行任务my_task.delay(*args, **kwargs)。

从您自己的评估来看，似乎您必须在缺少（关键）功能或有些多余的东西之间进行选择。在我的书中，这并不是很难选择的选择。

点击查看英文原文

Celery is not that complicated. At its core, you do the step by step configuration from the tutorials, create a celery instance, decorate your function with @celery.task then run the task with my_task.delay(*args, **kwargs).

Judging from your own assessment, it seems you have to choose between lacking (key) features or having some excess hanging around. That is not too hard of a choice in my book.