标签归档:c

如何从C#运行Python脚本?

问题:如何从C#运行Python脚本?

之前曾在不同程度上提出过这样的问题,但我觉得还没有以简明的方式回答,因此我再次提出。

我想在Python中运行脚本。可以说是这样的:

if __name__ == '__main__':
    with open(sys.argv[1], 'r') as f:
        s = f.read()
    print s

它获取文件位置,读取它,然后打印其内容。没那么复杂。

好吧,那我该如何在C#中运行它呢?

这就是我现在所拥有的:

    private void run_cmd(string cmd, string args)
    {
        ProcessStartInfo start = new ProcessStartInfo();
        start.FileName = cmd;
        start.Arguments = args;
        start.UseShellExecute = false;
        start.RedirectStandardOutput = true;
        using (Process process = Process.Start(start))
        {
            using (StreamReader reader = process.StandardOutput)
            {
                string result = reader.ReadToEnd();
                Console.Write(result);
            }
        }
    }

当我传递code.py位置as cmdfilename位置as args无效时。有人告诉我,我应该通过python.execmd,然后code.py filename作为args

我已经寻找了一段时间,只能找到建议使用IronPython之类的人。但是必须有一种从C#调用Python脚本的方法。

一些澄清:

我需要从C#运行它,我需要捕获输出,并且不能使用IronPython或其他任何东西。无论您有什么技巧,都可以。

PS:我正在运行的实际Python代码要复杂得多,并且它返回我在C#中需要的输出,并且C#代码将不断调用Python代码。

假设这是我的代码:

    private void get_vals()
    {
        for (int i = 0; i < 100; i++)
        {
            run_cmd("code.py", i);
        }
    }

This sort of question has been asked before in varying degrees, but I feel it has not been answered in a concise way and so I ask it again.

I want to run a script in Python. Let’s say it’s this:

if __name__ == '__main__':
    with open(sys.argv[1], 'r') as f:
        s = f.read()
    print s

Which gets a file location, reads it, then prints its contents. Not so complicated.

Okay, so how do I run this in C#?

This is what I have now:

    private void run_cmd(string cmd, string args)
    {
        ProcessStartInfo start = new ProcessStartInfo();
        start.FileName = cmd;
        start.Arguments = args;
        start.UseShellExecute = false;
        start.RedirectStandardOutput = true;
        using (Process process = Process.Start(start))
        {
            using (StreamReader reader = process.StandardOutput)
            {
                string result = reader.ReadToEnd();
                Console.Write(result);
            }
        }
    }

When I pass the code.py location as cmd and the filename location as args it doesn’t work. I was told I should pass python.exe as the cmd, and then code.py filename as the args.

I have been looking for a while now and can only find people suggesting to use IronPython or such. But there must be a way to call a Python script from C#.

Some clarification:

I need to run it from C#, I need to capture the output, and I can’t use IronPython or anything else. Whatever hack you have will be fine.

P.S.: The actual Python code I’m running is much more complex than this, and it returns output which I need in C#, and the C# code will be constantly calling the Python code.

Pretend this is my code:

    private void get_vals()
    {
        for (int i = 0; i < 100; i++)
        {
            run_cmd("code.py", i);
        }
    }

回答 0

它不起作用的原因是因为您有UseShellExecute = false

如果不使用外壳,则必须以方式提供python可执行文件的完整路径FileName,并构建Arguments字符串以提供脚本和要读取的文件。

另请注意,RedirectStandardOutput除非您不能这样做UseShellExecute = false

我不太确定应如何为python格式化参数字符串,但是您将需要以下内容:

private void run_cmd(string cmd, string args)
{
     ProcessStartInfo start = new ProcessStartInfo();
     start.FileName = "my/full/path/to/python.exe";
     start.Arguments = string.Format("{0} {1}", cmd, args);
     start.UseShellExecute = false;
     start.RedirectStandardOutput = true;
     using(Process process = Process.Start(start))
     {
         using(StreamReader reader = process.StandardOutput)
         {
             string result = reader.ReadToEnd();
             Console.Write(result);
         }
     }
}

The reason it isn’t working is because you have UseShellExecute = false.

If you don’t use the shell, you will have to supply the complete path to the python executable as FileName, and build the Arguments string to supply both your script and the file you want to read.

Also note, that you can’t RedirectStandardOutput unless UseShellExecute = false.

I’m not quite sure how the argument string should be formatted for python, but you will need something like this:

private void run_cmd(string cmd, string args)
{
     ProcessStartInfo start = new ProcessStartInfo();
     start.FileName = "my/full/path/to/python.exe";
     start.Arguments = string.Format("{0} {1}", cmd, args);
     start.UseShellExecute = false;
     start.RedirectStandardOutput = true;
     using(Process process = Process.Start(start))
     {
         using(StreamReader reader = process.StandardOutput)
         {
             string result = reader.ReadToEnd();
             Console.Write(result);
         }
     }
}

回答 1

如果您愿意使用IronPython,则可以直接在C#中执行脚本:

using IronPython.Hosting;
using Microsoft.Scripting.Hosting;

private static void doPython()
{
    ScriptEngine engine = Python.CreateEngine();
    engine.ExecuteFile(@"test.py");
}

在此处获取IronPython。

If you’re willing to use IronPython, you can execute scripts directly in C#:

using IronPython.Hosting;
using Microsoft.Scripting.Hosting;

private static void doPython()
{
    ScriptEngine engine = Python.CreateEngine();
    engine.ExecuteFile(@"test.py");
}

Get IronPython here.


回答 2

从C执行Python脚本

创建一个C#项目并编写以下代码。

using System;
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            run_cmd();
        }

        private void run_cmd()
        {

            string fileName = @"C:\sample_script.py";

            Process p = new Process();
            p.StartInfo = new ProcessStartInfo(@"C:\Python27\python.exe", fileName)
            {
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true
            };
            p.Start();

            string output = p.StandardOutput.ReadToEnd();
            p.WaitForExit();

            Console.WriteLine(output);

            Console.ReadLine();

        }
    }
}

Python sample_script

打印“ Python C#测试”

您将在C#控制台中看到“ Python C#测试”

Execute Python script from C

Create a C# project and write the following code.

using System;
using System.Diagnostics;
using System.IO;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            run_cmd();
        }

        private void run_cmd()
        {

            string fileName = @"C:\sample_script.py";

            Process p = new Process();
            p.StartInfo = new ProcessStartInfo(@"C:\Python27\python.exe", fileName)
            {
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true
            };
            p.Start();

            string output = p.StandardOutput.ReadToEnd();
            p.WaitForExit();

            Console.WriteLine(output);

            Console.ReadLine();

        }
    }
}

Python sample_script

print "Python C# Test"

You will see the ‘Python C# Test’ in the console of C#.


回答 3

我遇到了同样的问题,道德大师的回答对我没有帮助。以下是基于先前答案的方法:

private void run_cmd(string cmd, string args)
{
 ProcessStartInfo start = new ProcessStartInfo();
 start.FileName = cmd;//cmd is full path to python.exe
 start.Arguments = args;//args is path to .py file and any cmd line args
 start.UseShellExecute = false;
 start.RedirectStandardOutput = true;
 using(Process process = Process.Start(start))
 {
     using(StreamReader reader = process.StandardOutput)
     {
         string result = reader.ReadToEnd();
         Console.Write(result);
     }
 }
}

例如,如果要使用cmd行参数100执行test.py ,则cmd为,@C:/Python26/python.exe而args为C://Python26//test.py 100。请注意,.py文件的路径中没有@符号。

I ran into the same problem and Master Morality’s answer didn’t do it for me. The following, which is based on the previous answer, worked:

private void run_cmd(string cmd, string args)
{
 ProcessStartInfo start = new ProcessStartInfo();
 start.FileName = cmd;//cmd is full path to python.exe
 start.Arguments = args;//args is path to .py file and any cmd line args
 start.UseShellExecute = false;
 start.RedirectStandardOutput = true;
 using(Process process = Process.Start(start))
 {
     using(StreamReader reader = process.StandardOutput)
     {
         string result = reader.ReadToEnd();
         Console.Write(result);
     }
 }
}

As an example, cmd would be @C:/Python26/python.exe and args would be C://Python26//test.py 100 if you wanted to execute test.py with cmd line argument 100. Note that the path the .py file does not have the @ symbol.


回答 4


回答 5

设置WorkingDirectory或在Argument中指定python脚本的完整路径

ProcessStartInfo start = new ProcessStartInfo();
start.FileName = "C:\\Python27\\python.exe";
//start.WorkingDirectory = @"D:\script";
start.Arguments = string.Format("D:\\script\\test.py -a {0} -b {1} ", "some param", "some other param");
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
    using (StreamReader reader = process.StandardOutput)
    {
        string result = reader.ReadToEnd();
        Console.Write(result);
    }
}

Set WorkingDirectory or specify the full path of the python script in the Argument

ProcessStartInfo start = new ProcessStartInfo();
start.FileName = "C:\\Python27\\python.exe";
//start.WorkingDirectory = @"D:\script";
start.Arguments = string.Format("D:\\script\\test.py -a {0} -b {1} ", "some param", "some other param");
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
using (Process process = Process.Start(start))
{
    using (StreamReader reader = process.StandardOutput)
    {
        string result = reader.ReadToEnd();
        Console.Write(result);
    }
}

回答 6

实际上,使用IronPython在Csharp(VS)和Python之间进行集成非常容易。并没有那么复杂…正如克里斯·邓纳维(Chris Dunaway)在回答部分中所述,我开始为自己的项目建立这种整合。N非常简单。只需按照这些步骤N,您将获得结果。

步骤1:打开VS并创建新的空ConsoleApp项目。

步骤2:转到工具-> NuGet软件包管理器->软件包管理器控制台。

步骤3:在此之后,在浏览器中打开此链接并复制NuGet命令。链接:https//www.nuget.org/packages/IronPython/2.7.9

步骤4:打开以上链接后,复制PM> Install-Package IronPython -Version 2.7.9命令并将其粘贴到VS的NuGet控制台中。它将安装支持包。

步骤5:这是我用来运行存储在我的Python.exe目录中的.py文件的代码。

using IronPython.Hosting;//for DLHE
using Microsoft.Scripting.Hosting;//provides scripting abilities comparable to batch files
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Sockets;
class Hi
{
private static void Main(string []args)
{
Process process = new Process(); //to make a process call
ScriptEngine engine = Python.CreateEngine(); //For Engine to initiate the script
engine.ExecuteFile(@"C:\Users\daulmalik\AppData\Local\Programs\Python\Python37\p1.py");//Path of my .py file that I would like to see running in console after running my .cs file from VS.//process.StandardInput.Flush();
process.StandardInput.Close();//to close
process.WaitForExit();//to hold the process i.e. cmd screen as output
}
} 

步骤6:保存并执行代码

Actually its pretty easy to make integration between Csharp (VS) and Python with IronPython. It’s not that much complex… As Chris Dunaway already said in answer section I started to build this inegration for my own project. N its pretty simple. Just follow these steps N you will get your results.

step 1 : Open VS and create new empty ConsoleApp project.

step 2 : Go to tools –> NuGet Package Manager –> Package Manager Console.

step 3 : After this open this link in your browser and copy the NuGet Command. Link: https://www.nuget.org/packages/IronPython/2.7.9

step 4 : After opening the above link copy the PM>Install-Package IronPython -Version 2.7.9 command and paste it in NuGet Console in VS. It will install the supportive packages.

step 5 : This is my code that I have used to run a .py file stored in my Python.exe directory.

using IronPython.Hosting;//for DLHE
using Microsoft.Scripting.Hosting;//provides scripting abilities comparable to batch files
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Sockets;
class Hi
{
private static void Main(string []args)
{
Process process = new Process(); //to make a process call
ScriptEngine engine = Python.CreateEngine(); //For Engine to initiate the script
engine.ExecuteFile(@"C:\Users\daulmalik\AppData\Local\Programs\Python\Python37\p1.py");//Path of my .py file that I would like to see running in console after running my .cs file from VS.//process.StandardInput.Flush();
process.StandardInput.Close();//to close
process.WaitForExit();//to hold the process i.e. cmd screen as output
}
} 

step 6 : save and execute the code


回答 7

我遇到问题stdin/stout-当有效负载大小超过几千字节时,它挂起了。我不仅需要使用一些简短的参数来调用Python函数,还需要使用可能很大的自定义有效负载。

前一段时间,我编写了一个虚拟角色库,该库允许通过Redis在不同计算机上分发任务。为了调用Python代码,我添加了侦听来自Python的消息,对其进行处理并将结果返回给.NET的功能。 这是它如何工作的简要说明

它也可以在一台机器上运行,但是需要一个Redis实例。Redis增加了一些可靠性保证-有效负载被存储,直到工作确认完成为止。如果工作人员死亡,则有效负载将返回到作业队列,然后由另一个工作人员重新处理。

I am having problems with stdin/stout – when payload size exceeds several kilobytes it hangs. I need to call Python functions not only with some short arguments, but with a custom payload that could be big.

A while ago, I wrote a virtual actor library that allows to distribute task on different machines via Redis. To call Python code, I added functionality to listen for messages from Python, process them and return results back to .NET. Here is a brief description of how it works.

It works on a single machine as well, but requires a Redis instance. Redis adds some reliability guarantees – payload is stored until a worked acknowledges completion. If a worked dies, the payload is returned to a job queue and then is reprocessed by another worker.


Python(和Python C API):__new__与__init__

问题:Python(和Python C API):__new__与__init__

我要问的问题似乎是Python对__new__和__init__的重复使用?,但无论如何,我仍然不清楚__new__和之间的实际区别是什么__init__

在您急于告诉我__new__创建对象和__init__初始化对象之前,请让我明确:我明白了。 实际上,这种区分对我来说是很自然的,因为我在C ++中有经验,在那里我们放置了new,它类似地将对象分配与初始化分开。

Python的C API教程解释它是这样的:

新成员负责创建(而不是初始化)该类型的对象。它在Python中作为__new__()方法公开。… 实施新方法的原因之一是要确保实例变量的初始值

所以,是的-我明白__new__,但是尽管如此,我仍然不明白为什么它在Python中很有用。给出的示例说,__new__如果要“确保实例变量的初始值” ,这可能会很有用。好吧,这不正是要做__init__什么吗?

在C API教程中,显示​​了一个示例,其中创建了新的Type(称为“ Noddy”),并__new__定义了Type的功能。Noddy类型包含一个名为的字符串成员first,并且该字符串成员被初始化为一个空字符串,如下所示:

static PyObject * Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    .....

    self->first = PyString_FromString("");
    if (self->first == NULL)
    {
       Py_DECREF(self);
       return NULL;
    }

    .....
}

请注意,如果没有在此__new__定义的方法,我们将不得不使用PyType_GenericNew,它只会将所有实例变量成员初始化为NULL。因此,该__new__方法的唯一好处是实例变量将从一个空字符串开始,而不是NULL。 但是,为什么这会有用呢,因为如果我们要确保将实例变量初始化为某个默认值,那么我们可以在__init__方法中做到这一点?

The question I’m about to ask seems to be a duplicate of Python’s use of __new__ and __init__?, but regardless, it’s still unclear to me exactly what the practical difference between __new__ and __init__ is.

Before you rush to tell me that __new__ is for creating objects and __init__ is for initializing objects, let me be clear: I get that. In fact, that distinction is quite natural to me, since I have experience in C++ where we have placement new, which similarly separates object allocation from initialization.

The Python C API tutorial explains it like this:

The new member is responsible for creating (as opposed to initializing) objects of the type. It is exposed in Python as the __new__() method. … One reason to implement a new method is to assure the initial values of instance variables.

So, yeah – I get what __new__ does, but despite this, I still don’t understand why it’s useful in Python. The example given says that __new__ might be useful if you want to “assure the initial values of instance variables”. Well, isn’t that exactly what __init__ will do?

In the C API tutorial, an example is shown where a new Type (called a “Noddy”) is created, and the Type’s __new__ function is defined. The Noddy type contains a string member called first, and this string member is initialized to an empty string like so:

static PyObject * Noddy_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    .....

    self->first = PyString_FromString("");
    if (self->first == NULL)
    {
       Py_DECREF(self);
       return NULL;
    }

    .....
}

Note that without the __new__ method defined here, we’d have to use PyType_GenericNew, which simply initializes all of the instance variable members to NULL. So the only benefit of the __new__ method is that the instance variable will start out as an empty string, as opposed to NULL. But why is this ever useful, since if we cared about making sure our instance variables are initialized to some default value, we could have just done that in the __init__ method?


回答 0

差异主要发生在可变与不可变类型之间。

__new__接受一个类型作为第一个参数,并且(通常)返回该类型的新实例。因此,它适用于可变类型和不可变类型。

__init__接受一个实例作为第一个参数,并修改该实例的属性。这不适用于不可变类型,因为它允许在创建后通过调用修改它们obj.__init__(*args)

比较的行为tuplelist

>>> x = (1, 2)
>>> x
(1, 2)
>>> x.__init__([3, 4])
>>> x # tuple.__init__ does nothing
(1, 2)
>>> y = [1, 2]
>>> y
[1, 2]
>>> y.__init__([3, 4])
>>> y # list.__init__ reinitialises the object
[3, 4]

关于它们为什么分开的原因(除了简单的历史原因):__new__方法需要一堆样板才能正确(最初的对象创建,然后记得最后返回对象)。__init__相比之下,方法非常简单,因为您只需设置需要设置的任何属性即可。

除了__init__更易于编写的方法以及上面提到的可变与不可变的区别外,还可以利用这种分离,__init__通过在中设置任何绝对必要的实例不变式,使在子类中调用父类成为可选的__new__。不过,这通常是一种可疑的做法-通常在需要时仅调用父类__init__方法会更清晰。

The difference mainly arises with mutable vs immutable types.

__new__ accepts a type as the first argument, and (usually) returns a new instance of that type. Thus it is suitable for use with both mutable and immutable types.

__init__ accepts an instance as the first argument and modifies the attributes of that instance. This is inappropriate for an immutable type, as it would allow them to be modified after creation by calling obj.__init__(*args).

Compare the behaviour of tuple and list:

>>> x = (1, 2)
>>> x
(1, 2)
>>> x.__init__([3, 4])
>>> x # tuple.__init__ does nothing
(1, 2)
>>> y = [1, 2]
>>> y
[1, 2]
>>> y.__init__([3, 4])
>>> y # list.__init__ reinitialises the object
[3, 4]

As to why they’re separate (aside from simple historical reasons): __new__ methods require a bunch of boilerplate to get right (the initial object creation, and then remembering to return the object at the end). __init__ methods, by contrast, are dead simple, since you just set whatever attributes you need to set.

Aside from __init__ methods being easier to write, and the mutable vs immutable distinction noted above, the separation can also be exploited to make calling the parent class __init__ in subclasses optional by setting up any absolutely required instance invariants in __new__. This is generally a dubious practice though – it’s usually clearer to just call the parent class __init__ methods as necessary.


回答 1

可能还有其他用途,__new__但有一个真正显而易见的用途:如果不使用,就不能继承不可变类型__new__。例如,假设您要创建一个元组的子类,该子类只能包含0到之间的整数值size

class ModularTuple(tuple):
    def __new__(cls, tup, size=100):
        tup = (int(x) % size for x in tup)
        return super(ModularTuple, cls).__new__(cls, tup)

你根本无法做到这一点__init__-如果你试图修改self__init__,解释器会抱怨你试图修改不可变对象。

There are probably other uses for __new__ but there’s one really obvious one: You can’t subclass an immutable type without using __new__. So for example, say you wanted to create a subclass of tuple that can contain only integral values between 0 and size.

class ModularTuple(tuple):
    def __new__(cls, tup, size=100):
        tup = (int(x) % size for x in tup)
        return super(ModularTuple, cls).__new__(cls, tup)

You simply can’t do this with __init__ — if you tried to modify self in __init__, the interpreter would complain that you’re trying to modify an immutable object.


回答 2

__new__()可以返回与其绑定的类不同类型的对象。__init__()仅初始化该类的现有实例。

>>> class C(object):
...   def __new__(cls):
...     return 5
...
>>> c = C()
>>> print type(c)
<type 'int'>
>>> print c
5

__new__() can return objects of types other than the class it’s bound to. __init__() only initializes an existing instance of the class.

>>> class C(object):
...   def __new__(cls):
...     return 5
...
>>> c = C()
>>> print type(c)
<type 'int'>
>>> print c
5

回答 3

这不是一个完整的答案,但也许可以说明差异。

__new__当必须创建一个对象时,它将总是被调用。在某些情况下__init__不会被呼叫。一个示例是,当您从pickle文件中解开对象时,它们将被分配(__new__)但未初始化(__init__)。

Not a complete answer but perhaps something that illustrates the difference.

__new__ will always get called when an object has to be created. There are some situations where __init__ will not get called. One example is when you unpickle objects from a pickle file, they will get allocated (__new__) but not initialised (__init__).


回答 4

只是想添加一个关于定义vs 的意图(与行为相反)的词__new____init__

当我试图理解定义类工厂的最佳方法时,我遇到了这个问题。我意识到,在__new__概念上与之不同的一种方式__init__是,这样的好处__new__恰恰是问题中所陈述的事实:

因此__new__方法的唯一好处是实例变量将从一个空字符串开始,而不是NULL。但是为什么这会有用呢,因为如果我们要确保实例变量被初始化为某个默认值,那么我们可以在__init__方法中做到这一点?

考虑到上述情况,当实例实际上是类本身时,我们关心实例变量的初始值。因此,如果我们在运行时动态创建一个类对象,并且需要定义/控制一些有关正在创建的类的后续实例的特殊操作,则可以在__new__元类的方法中定义这些条件/属性。

我一直对此感到困惑,直到我真正考虑到该概念的应用,而不仅仅是其含义。这是一个希望可以使区别清楚的示例:

a = Shape(sides=3, base=2, height=12)
b = Shape(sides=4, length=2)
print(a.area())
print(b.area())

# I want `a` and `b` to be an instances of either of 'Square' or 'Triangle'
# depending on number of sides and also the `.area()` method to do the right
# thing. How do I do that without creating a Shape class with all the
# methods having a bunch of `if`s ? Here is one possibility

class Shape:
    def __new__(cls, sides, *args, **kwargs):
        if sides == 3:
            return Triangle(*args, **kwargs)
        else:
            return Square(*args, **kwargs)

class Triangle:
    def __init__(self, base, height):
        self.base = base
        self.height = height

    def area(self):
        return (self.base * self.height) / 2

class Square:
    def __init__(self, length):
        self.length = length

    def area(self):
        return self.length*self.length

请注意,这只是一个示例。有多种方法可以获取解决方案,而无需借助上述的类工厂方法,即使我们确实选择以这种方式来实现该解决方案,为简洁起见也有一些注意事项(例如,明确声明元类) )

如果您要创建常规类(又称为非元类),那么__new__除非真正有特殊意义,例如ncoghlan答案中的可变与不可变方案(实际上是定义概念的更具体示例),否则这没有什么意义通过创建的类/类型的初始值/属性,__new__然后通过进行初始化__init__

Just want to add a word about the intent (as opposed to the behavior) of defining __new__ versus __init__.

I came across this question (among others) when I was trying to understand the best way to define a class factory. I realized that one of the ways in which __new__ is conceptually different from __init__ is the fact that the benefit of __new__ is exactly what was stated in the question:

So the only benefit of the __new__ method is that the instance variable will start out as an empty string, as opposed to NULL. But why is this ever useful, since if we cared about making sure our instance variables are initialized to some default value, we could have just done that in the __init__ method?

Considering the stated scenario, we care about the initial values of the instance variables when the instance is in reality a class itself. So, if we are dynamically creating a class object at runtime and we need to define/control something special about the subsequent instances of this class being created, we would define these conditions/properties in a __new__ method of a metaclass.

I was confused about this until I actually thought about the application of the concept rather than just the meaning of it. Here’s an example that would hopefully make the difference clear:

a = Shape(sides=3, base=2, height=12)
b = Shape(sides=4, length=2)
print(a.area())
print(b.area())

# I want `a` and `b` to be an instances of either of 'Square' or 'Triangle'
# depending on number of sides and also the `.area()` method to do the right
# thing. How do I do that without creating a Shape class with all the
# methods having a bunch of `if`s ? Here is one possibility

class Shape:
    def __new__(cls, sides, *args, **kwargs):
        if sides == 3:
            return Triangle(*args, **kwargs)
        else:
            return Square(*args, **kwargs)

class Triangle:
    def __init__(self, base, height):
        self.base = base
        self.height = height

    def area(self):
        return (self.base * self.height) / 2

class Square:
    def __init__(self, length):
        self.length = length

    def area(self):
        return self.length*self.length

Note this is just an demonstartive example. There are multiple ways to get a solution without resorting to a class factory approach like above and even if we do choose to implelent the solution in this manner, there are a little caveats left out for sake of brevity (for instance, declaring the metaclass explicitly)

If you are creating a regular class (a.k.a a non-metaclass), then __new__ doesn’t really make sense unless it is special case like the mutable versus immutable scenario in ncoghlan’s answer answer (which is essentially a more specific example of the concept of defining the initial values/properties of the class/type being created via __new__ to be then initialized via __init__).


与Python生成器模式等效的C ++

问题:与Python生成器模式等效的C ++

我有一些需要在C ++中模仿的示例Python代码。我不需要任何特定的解决方案(例如基于协同例程的收益解决方案,尽管它们也是可接受的答案),我只需要以某种方式重现语义。

Python

这是一个基本的序列生成器,显然太大了,无法存储实例化版本。

def pair_sequence():
    for i in range(2**32):
        for j in range(2**32):
            yield (i, j)

目标是维护上述序列的两个实例,并以半锁步的方式在块上进行迭代。在下面的示例中,first_pass使用对的序列来初始化缓冲区,然后second_pass重新生成相同的精确序列并再次处理缓冲区。

def run():
    seq1 = pair_sequence()
    seq2 = pair_sequence()

    buffer = [0] * 1000
    first_pass(seq1, buffer)
    second_pass(seq2, buffer)
    ... repeat ...

C ++

对于C ++解决方案,我唯一能找到的就是模仿yieldC ++协程,但是我还没有找到有关如何执行此操作的良好参考。我也对解决此问题的替代(非常规)解决方案感兴趣。我没有足够的内存预算来保留两次通过之间的序列副本。

I’ve got some example Python code that I need to mimic in C++. I do not require any specific solution (such as co-routine based yield solutions, although they would be acceptable answers as well), I simply need to reproduce the semantics in some manner.

Python

This is a basic sequence generator, clearly too large to store a materialized version.

def pair_sequence():
    for i in range(2**32):
        for j in range(2**32):
            yield (i, j)

The goal is to maintain two instances of the sequence above, and iterate over them in semi-lockstep, but in chunks. In the example below the first_pass uses the sequence of pairs to initialize the buffer, and the second_pass regenerates the same exact sequence and processes the buffer again.

def run():
    seq1 = pair_sequence()
    seq2 = pair_sequence()

    buffer = [0] * 1000
    first_pass(seq1, buffer)
    second_pass(seq2, buffer)
    ... repeat ...

C++

The only thing I can find for a solution in C++ is to mimic yield with C++ coroutines, but I haven’t found any good reference on how to do this. I’m also interested in alternative (non general) solutions for this problem. I do not have enough memory budget to keep a copy of the sequence between passes.


回答 0

生成器在C ++中存在,只是另外一个名称:Input Iterators。例如,从读取std::cin类似于使用的生成器char

您只需要了解生成器的功能:

  • 有大量数据:局部变量定义状态
  • 有一个初始化方法
  • 有一个“下一个”方法
  • 有一种信号终止的方法

在您的琐碎示例中,这很容易。从概念上讲:

struct State { unsigned i, j; };

State make();

void next(State&);

bool isDone(State const&);

当然,我们将其包装为适当的类:

class PairSequence:
    // (implicit aliases)
    public std::iterator<
        std::input_iterator_tag,
        std::pair<unsigned, unsigned>
    >
{
  // C++03
  typedef void (PairSequence::*BoolLike)();
  void non_comparable();
public:
  // C++11 (explicit aliases)
  using iterator_category = std::input_iterator_tag;
  using value_type = std::pair<unsigned, unsigned>;
  using reference = value_type const&;
  using pointer = value_type const*;
  using difference_type = ptrdiff_t;

  // C++03 (explicit aliases)
  typedef std::input_iterator_tag iterator_category;
  typedef std::pair<unsigned, unsigned> value_type;
  typedef value_type const& reference;
  typedef value_type const* pointer;
  typedef ptrdiff_t difference_type;

  PairSequence(): done(false) {}

  // C++11
  explicit operator bool() const { return !done; }

  // C++03
  // Safe Bool idiom
  operator BoolLike() const {
    return done ? 0 : &PairSequence::non_comparable;
  }

  reference operator*() const { return ij; }
  pointer operator->() const { return &ij; }

  PairSequence& operator++() {
    static unsigned const Max = std::numeric_limts<unsigned>::max();

    assert(!done);

    if (ij.second != Max) { ++ij.second; return *this; }
    if (ij.first != Max) { ij.second = 0; ++ij.first; return *this; }

    done = true;
    return *this;
  }

  PairSequence operator++(int) {
    PairSequence const tmp(*this);
    ++*this;
    return tmp;
  }

private:
  bool done;
  value_type ij;
};

所以,嗯…可能是C ++有点冗长:)

Generators exist in C++, just under another name: Input Iterators. For example, reading from std::cin is similar to having a generator of char.

You simply need to understand what a generator does:

  • there is a blob of data: the local variables define a state
  • there is an init method
  • there is a “next” method
  • there is a way to signal termination

In your trivial example, it’s easy enough. Conceptually:

struct State { unsigned i, j; };

State make();

void next(State&);

bool isDone(State const&);

Of course, we wrap this as a proper class:

class PairSequence:
    // (implicit aliases)
    public std::iterator<
        std::input_iterator_tag,
        std::pair<unsigned, unsigned>
    >
{
  // C++03
  typedef void (PairSequence::*BoolLike)();
  void non_comparable();
public:
  // C++11 (explicit aliases)
  using iterator_category = std::input_iterator_tag;
  using value_type = std::pair<unsigned, unsigned>;
  using reference = value_type const&;
  using pointer = value_type const*;
  using difference_type = ptrdiff_t;

  // C++03 (explicit aliases)
  typedef std::input_iterator_tag iterator_category;
  typedef std::pair<unsigned, unsigned> value_type;
  typedef value_type const& reference;
  typedef value_type const* pointer;
  typedef ptrdiff_t difference_type;

  PairSequence(): done(false) {}

  // C++11
  explicit operator bool() const { return !done; }

  // C++03
  // Safe Bool idiom
  operator BoolLike() const {
    return done ? 0 : &PairSequence::non_comparable;
  }

  reference operator*() const { return ij; }
  pointer operator->() const { return &ij; }

  PairSequence& operator++() {
    static unsigned const Max = std::numeric_limts<unsigned>::max();

    assert(!done);

    if (ij.second != Max) { ++ij.second; return *this; }
    if (ij.first != Max) { ij.second = 0; ++ij.first; return *this; }

    done = true;
    return *this;
  }

  PairSequence operator++(int) {
    PairSequence const tmp(*this);
    ++*this;
    return tmp;
  }

private:
  bool done;
  value_type ij;
};

So hum yeah… might be that C++ is a tad more verbose :)


回答 1

在C ++中有迭代器,但是实现迭代器并不容易:必须查阅迭代器概念并仔细设计新的迭代器类以实现它们。值得庆幸的是,Boost有一个iterator_facade模板,该模板应该有助于实现迭代器和兼容迭代器的生成器。

有时可以使用无堆栈协程来实现迭代器

PS另请参见 本文,其中同时提到了switchChristopher M. Kohlhoff 的黑客行为和Oliver Kowalke的Boost.Coroutine。Oliver Kowalke的工作 Giovanni P. Deretta 对Boost.Coroutine 的后续

PS我想你也可以用lambdas编写一种生成器:

std::function<int()> generator = []{
  int i = 0;
  return [=]() mutable {
    return i < 10 ? i++ : -1;
  };
}();
int ret = 0; while ((ret = generator()) != -1) std::cout << "generator: " << ret << std::endl;

或使用函子:

struct generator_t {
  int i = 0;
  int operator() () {
    return i < 10 ? i++ : -1;
  }
} generator;
int ret = 0; while ((ret = generator()) != -1) std::cout << "generator: " << ret << std::endl;

PS这是用Mordor协同程序实现的生成器:

#include <iostream>
using std::cout; using std::endl;
#include <mordor/coroutine.h>
using Mordor::Coroutine; using Mordor::Fiber;

void testMordor() {
  Coroutine<int> coro ([](Coroutine<int>& self) {
    int i = 0; while (i < 9) self.yield (i++);
  });
  for (int i = coro.call(); coro.state() != Fiber::TERM; i = coro.call()) cout << i << endl;
}

In C++ there are iterators, but implementing an iterator isn’t straightforward: one has to consult the iterator concepts and carefully design the new iterator class to implement them. Thankfully, Boost has an iterator_facade template which should help implementing the iterators and iterator-compatible generators.

Sometimes a stackless coroutine can be used to implement an iterator.

P.S. See also this article which mentions both a switch hack by Christopher M. Kohlhoff and Boost.Coroutine by Oliver Kowalke. Oliver Kowalke’s work is a followup on Boost.Coroutine by Giovanni P. Deretta.

P.S. I think you can also write a kind of generator with lambdas:

std::function<int()> generator = []{
  int i = 0;
  return [=]() mutable {
    return i < 10 ? i++ : -1;
  };
}();
int ret = 0; while ((ret = generator()) != -1) std::cout << "generator: " << ret << std::endl;

Or with a functor:

struct generator_t {
  int i = 0;
  int operator() () {
    return i < 10 ? i++ : -1;
  }
} generator;
int ret = 0; while ((ret = generator()) != -1) std::cout << "generator: " << ret << std::endl;

P.S. Here’s a generator implemented with the Mordor coroutines:

#include <iostream>
using std::cout; using std::endl;
#include <mordor/coroutine.h>
using Mordor::Coroutine; using Mordor::Fiber;

void testMordor() {
  Coroutine<int> coro ([](Coroutine<int>& self) {
    int i = 0; while (i < 9) self.yield (i++);
  });
  for (int i = coro.call(); coro.state() != Fiber::TERM; i = coro.call()) cout << i << endl;
}

回答 2

由于Boost.Coroutine2现在很好地支持了它(我找到它是因为我想解决完全相同的yield问题),所以我发布了符合您最初意图的C ++代码:

#include <stdint.h>
#include <iostream>
#include <memory>
#include <boost/coroutine2/all.hpp>

typedef boost::coroutines2::coroutine<std::pair<uint16_t, uint16_t>> coro_t;

void pair_sequence(coro_t::push_type& yield)
{
    uint16_t i = 0;
    uint16_t j = 0;
    for (;;) {
        for (;;) {
            yield(std::make_pair(i, j));
            if (++j == 0)
                break;
        }
        if (++i == 0)
            break;
    }
}

int main()
{
    coro_t::pull_type seq(boost::coroutines2::fixedsize_stack(),
                          pair_sequence);
    for (auto pair : seq) {
        print_pair(pair);
    }
    //while (seq) {
    //    print_pair(seq.get());
    //    seq();
    //}
}

在此示例中,pair_sequence不接受其他参数。如果需要,std::bind或者在将lambda push_type传递给coro_t::pull_type构造函数时,应使用lambda生成仅包含(of )个参数的函数对象。

Since Boost.Coroutine2 now supports it very well (I found it because I wanted to solve exactly the same yield problem), I am posting the C++ code that matches your original intention:

#include <stdint.h>
#include <iostream>
#include <memory>
#include <boost/coroutine2/all.hpp>

typedef boost::coroutines2::coroutine<std::pair<uint16_t, uint16_t>> coro_t;

void pair_sequence(coro_t::push_type& yield)
{
    uint16_t i = 0;
    uint16_t j = 0;
    for (;;) {
        for (;;) {
            yield(std::make_pair(i, j));
            if (++j == 0)
                break;
        }
        if (++i == 0)
            break;
    }
}

int main()
{
    coro_t::pull_type seq(boost::coroutines2::fixedsize_stack(),
                          pair_sequence);
    for (auto pair : seq) {
        print_pair(pair);
    }
    //while (seq) {
    //    print_pair(seq.get());
    //    seq();
    //}
}

In this example, pair_sequence does not take additional arguments. If it needs to, std::bind or a lambda should be used to generate a function object that takes only one argument (of push_type), when it is passed to the coro_t::pull_type constructor.


回答 3

所有涉及编写自己的迭代器的答案都是完全错误的。这样的答案完全错过了Python生成器的意义(该语言最大的独特功能之一)。生成器最重要的是执行从中断处开始执行。迭代器不会发生这种情况。取而代之的是,您必须手动存储状态信息,以便在重新调用operator ++或operator * 时,在下一个函数调用的最开始便有正确的信息。这就是为什么编写自己的C ++迭代器是一个巨大的痛苦。相反,生成器优雅,并且易于读写。

我认为本机C ++中没有适合Python生成器的良好模拟,至少目前还没有(有传言称yield将在C ++ 17中实现)。您可以借助第三方(例如,永伟的Boost建议)或自己动手做一些类似的事情。

我会说本机C ++中最接近的东西是线程。线程可以维护一组暂停的局部变量,并且可以在中断处继续执行,这与生成器非常相似,但是您需要使用一些附加的基础架构来支持生成器对象与其调用者之间的通信。例如

// Infrastructure

template <typename Element>
class Channel { ... };

// Application

using IntPair = std::pair<int, int>;

void yield_pairs(int end_i, int end_j, Channel<IntPair>* out) {
  for (int i = 0; i < end_i; ++i) {
    for (int j = 0; j < end_j; ++j) {
      out->send(IntPair{i, j});  // "yield"
    }
  }
  out->close();
}

void MyApp() {
  Channel<IntPair> pairs;
  std::thread generator(yield_pairs, 32, 32, &pairs);
  for (IntPair pair : pairs) {
    UsePair(pair);
  }
  generator.join();
}

但是,此解决方案有几个缺点:

  1. 线程是“昂贵的”。大多数人会认为这是对线程的“过度”使用,尤其是当生成器如此简单时。
  2. 您需要记住一些清理操作。这些可以是自动化的,但是您将需要更多的基础架构,而这些基础架构又可能被视为“过于奢侈”。无论如何,您需要进行的清理工作是:
    1. out-> close()
    2. generator.join()
  3. 这不允许您停止生成器。您可以进行一些修改以添加该功能,但是这会使代码混乱。它永远不会像Python的yield语句那么干净。
  4. 除2之外,每次您要“实例化”生成器对象时,还需要其他样板位:
    1. 通道*输出参数
    2. 主变量:对,生成器

All answers that involve writing your own iterator are completely wrong. Such answers entirely miss the point of Python generators (one of the language’s greatest and unique features). The most important thing about generators is that execution picks up where it left off. This does not happen to iterators. Instead, you must manually store state information such that when operator++ or operator* is called anew, the right information is in place at the very beginning of the next function call. This is why writing your own C++ iterator is a gigantic pain; whereas, generators are elegant, and easy to read+write.

I don’t think there is a good analog for Python generators in native C++, at least not yet (there is a rummor that yield will land in C++17). You can get something similarish by resorting to third-party (e.g. Yongwei’s Boost suggestion), or rolling your own.

I would say the closest thing in native C++ is threads. A thread can maintain a suspended set of local variables, and can continue execution where it left off, very much like generators, but you need to roll a little bit of additional infrastructure to support communication between the generator object and its caller. E.g.

// Infrastructure

template <typename Element>
class Channel { ... };

// Application

using IntPair = std::pair<int, int>;

void yield_pairs(int end_i, int end_j, Channel<IntPair>* out) {
  for (int i = 0; i < end_i; ++i) {
    for (int j = 0; j < end_j; ++j) {
      out->send(IntPair{i, j});  // "yield"
    }
  }
  out->close();
}

void MyApp() {
  Channel<IntPair> pairs;
  std::thread generator(yield_pairs, 32, 32, &pairs);
  for (IntPair pair : pairs) {
    UsePair(pair);
  }
  generator.join();
}

This solution has several downsides though:

  1. Threads are “expensive”. Most people would consider this to be an “extravagant” use of threads, especially when your generator is so simple.
  2. There are a couple of clean up actions that you need to remember. These could be automated, but you’d need even more infrastructure, which again, is likely to be seen as “too extravagant”. Anyway, the clean ups that you need are:
    1. out->close()
    2. generator.join()
  3. This does not allow you to stop generator. You could make some modifications to add that ability, but it adds clutter to the code. It would never be as clean as Python’s yield statement.
  4. In addition to 2, there are other bits of boilerplate that are needed each time you want to “instantiate” a generator object:
    1. Channel* out parameter
    2. Additional variables in main: pairs, generator

回答 4

您可能应该在Visual Studio 2015的std :: experimental中检查生成器,例如:https : //blogs.msdn.microsoft.com/vcblog/2014/11/12/resumable-functions-in-c/

我认为这正是您想要的。总体生成器应在C ++ 17中可用,因为这只是实验性的Microsoft VC功能。

You should probably check generators in std::experimental in Visual Studio 2015 e.g: https://blogs.msdn.microsoft.com/vcblog/2014/11/12/resumable-functions-in-c/

I think it’s exactly what you are looking for. Overall generators should be available in C++17 as this is only experimental Microsoft VC feature.


回答 5

如果只需要为相对较少的特定生成器执行此操作,则可以将每个实现生成为一个类,其中成员数据等效于Python生成器函数的局部变量。然后,您将具有一个next函数,该函数返回生成器将产生的下一个内容,并以此更新内部状态。

我相信,这基本上与Python生成器的实现方式相似。主要的区别是它们可以记住生成器函数的字节码中的偏移量,作为“内部状态”的一部分,这意味着可以将生成器写为包含yield的循环。您将不得不从上一个计算下一个值。在您的情况下pair_sequence,这是微不足道的。它可能不适用于复杂的生成器。

您还需要一些指示终止的方法。如果返回的是“类似指针的”,并且NULL不应为有效的yieldable值,则可以将NULL指针用作终止指示符。否则,您需要带外信号。

If you only need to do this for a relatively small number of specific generators, you can implement each as a class, where the member data is equivalent to the local variables of the Python generator function. Then you have a next function that returns the next thing the generator would yield, updating the internal state as it does so.

This is basically similar to how Python generators are implemented, I believe. The major difference being they can remember an offset into the bytecode for the generator function as part of the “internal state”, which means the generators can be written as loops containing yields. You would have to instead calculate the next value from the previous. In the case of your pair_sequence, that’s pretty trivial. It may not be for complex generators.

You also need some way of indicating termination. If what you’re returning is “pointer-like”, and NULL should not be a valid yieldable value you could use a NULL pointer as a termination indicator. Otherwise you need an out-of-band signal.


回答 6

这样的事情非常相似:

struct pair_sequence
{
    typedef pair<unsigned int, unsigned int> result_type;
    static const unsigned int limit = numeric_limits<unsigned int>::max()

    pair_sequence() : i(0), j(0) {}

    result_type operator()()
    {
        result_type r(i, j);
        if(j < limit) j++;
        else if(i < limit)
        {
          j = 0;
          i++;
        }
        else throw out_of_range("end of iteration");
    }

    private:
        unsigned int i;
        unsigned int j;
}

使用operator()只是要对该生成器执行的操作的一个问题,例如,您还可以将其构建为流,并确保它适合istream_iterator。

Something like this is very similar:

struct pair_sequence
{
    typedef pair<unsigned int, unsigned int> result_type;
    static const unsigned int limit = numeric_limits<unsigned int>::max()

    pair_sequence() : i(0), j(0) {}

    result_type operator()()
    {
        result_type r(i, j);
        if(j < limit) j++;
        else if(i < limit)
        {
          j = 0;
          i++;
        }
        else throw out_of_range("end of iteration");
    }

    private:
        unsigned int i;
        unsigned int j;
}

Using the operator() is only a question of what you want to do with this generator, you could also build it as a stream and make sure it adapts to an istream_iterator, for example.


回答 7

使用range-v3

#include <iostream>
#include <tuple>
#include <range/v3/all.hpp>

using namespace std;
using namespace ranges;

auto generator = [x = view::iota(0) | view::take(3)] {
    return view::cartesian_product(x, x);
};

int main () {
    for (auto x : generator()) {
        cout << get<0>(x) << ", " << get<1>(x) << endl;
    }

    return 0;
}

Using range-v3:

#include <iostream>
#include <tuple>
#include <range/v3/all.hpp>

using namespace std;
using namespace ranges;

auto generator = [x = view::iota(0) | view::take(3)] {
    return view::cartesian_product(x, x);
};

int main () {
    for (auto x : generator()) {
        cout << get<0>(x) << ", " << get<1>(x) << endl;
    }

    return 0;
}

回答 8

这样的东西:

使用示例:

using ull = unsigned long long;

auto main() -> int {
    for (ull val : range_t<ull>(100)) {
        std::cout << val << std::endl;
    }

    return 0;
}

将打印从0到99的数字

Something like this:

Example use:

using ull = unsigned long long;

auto main() -> int {
    for (ull val : range_t<ull>(100)) {
        std::cout << val << std::endl;
    }

    return 0;
}

Will print the numbers from 0 to 99


回答 9

好吧,今天我也在寻找在C ++ 11下实现轻松收集的实现。实际上,我很失望,因为我发现的一切都与python生成器或C#yield操作符等东西相距太远……或者过于复杂。

目的是使收集仅在需要时才发出其项目。

我希望它像这样:

auto emitter = on_range<int>(a, b).yield(
    [](int i) {
         /* do something with i */
         return i * 2;
    });

我发现这个职位,恕我直言,最好的回答是大约boost.coroutine2,通过永伟吴。由于它是最接近作者想要的东西。

值得学习加强日常护理。.而且我也许会在周末去做。但是到目前为止,我正在使用非常小的实现。希望它对其他人有帮助。

下面是使用示例,然后实现。

范例.cpp

#include <iostream>
#include "Generator.h"
int main() {
    typedef std::pair<int, int> res_t;

    auto emitter = Generator<res_t, int>::on_range(0, 3)
        .yield([](int i) {
            return std::make_pair(i, i * i);
        });

    for (auto kv : emitter) {
        std::cout << kv.first << "^2 = " << kv.second << std::endl;
    }

    return 0;
}

Generator.h

template<typename ResTy, typename IndexTy>
struct yield_function{
    typedef std::function<ResTy(IndexTy)> type;
};

template<typename ResTy, typename IndexTy>
class YieldConstIterator {
public:
    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;

    typedef YieldConstIterator<ResTy, IndexTy> mytype_t;
    typedef ResTy value_type;

    YieldConstIterator(index_t index, yield_function_t yieldFunction) :
            mIndex(index),
            mYieldFunction(yieldFunction) {}

    mytype_t &operator++() {
        ++mIndex;
        return *this;
    }

    const value_type operator*() const {
        return mYieldFunction(mIndex);
    }

    bool operator!=(const mytype_t &r) const {
        return mIndex != r.mIndex;
    }

protected:

    index_t mIndex;
    yield_function_t mYieldFunction;
};

template<typename ResTy, typename IndexTy>
class YieldIterator : public YieldConstIterator<ResTy, IndexTy> {
public:

    typedef YieldConstIterator<ResTy, IndexTy> parent_t;

    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;
    typedef ResTy value_type;

    YieldIterator(index_t index, yield_function_t yieldFunction) :
            parent_t(index, yieldFunction) {}

    value_type operator*() {
        return parent_t::mYieldFunction(parent_t::mIndex);
    }
};

template<typename IndexTy>
struct Range {
public:
    typedef IndexTy index_t;
    typedef Range<IndexTy> mytype_t;

    index_t begin;
    index_t end;
};

template<typename ResTy, typename IndexTy>
class GeneratorCollection {
public:

    typedef Range<IndexTy> range_t;

    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;
    typedef YieldIterator<ResTy, IndexTy> iterator;
    typedef YieldConstIterator<ResTy, IndexTy> const_iterator;

    GeneratorCollection(range_t range, const yield_function_t &yieldF) :
            mRange(range),
            mYieldFunction(yieldF) {}

    iterator begin() {
        return iterator(mRange.begin, mYieldFunction);
    }

    iterator end() {
        return iterator(mRange.end, mYieldFunction);
    }

    const_iterator begin() const {
        return const_iterator(mRange.begin, mYieldFunction);
    }

    const_iterator end() const {
        return const_iterator(mRange.end, mYieldFunction);
    }

private:
    range_t mRange;
    yield_function_t mYieldFunction;
};

template<typename ResTy, typename IndexTy>
class Generator {
public:
    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;

    typedef Generator<ResTy, IndexTy> mytype_t;
    typedef Range<IndexTy> parent_t;
    typedef GeneratorCollection<ResTy, IndexTy> finalized_emitter_t;
    typedef  Range<IndexTy> range_t;

protected:
    Generator(range_t range) : mRange(range) {}
public:
    static mytype_t on_range(index_t begin, index_t end) {
        return mytype_t({ begin, end });
    }

    finalized_emitter_t yield(yield_function_t f) {
        return finalized_emitter_t(mRange, f);
    }
protected:

    range_t mRange;
};      

Well, today I also was looking for easy collection implementation under C++11. Actually I was disappointed, because everything I found is too far from things like python generators, or C# yield operator… or too complicated.

The purpose is to make collection which will emit its items only when it is required.

I wanted it to be like this:

auto emitter = on_range<int>(a, b).yield(
    [](int i) {
         /* do something with i */
         return i * 2;
    });

I found this post, IMHO best answer was about boost.coroutine2, by Yongwei Wu. Since it is the nearest to what author wanted.

It is worth learning boost couroutines.. And I’ll perhaps do on weekends. But so far I’m using my very small implementation. Hope it helps to someone else.

Below is example of use, and then implementation.

Example.cpp

#include <iostream>
#include "Generator.h"
int main() {
    typedef std::pair<int, int> res_t;

    auto emitter = Generator<res_t, int>::on_range(0, 3)
        .yield([](int i) {
            return std::make_pair(i, i * i);
        });

    for (auto kv : emitter) {
        std::cout << kv.first << "^2 = " << kv.second << std::endl;
    }

    return 0;
}

Generator.h

template<typename ResTy, typename IndexTy>
struct yield_function{
    typedef std::function<ResTy(IndexTy)> type;
};

template<typename ResTy, typename IndexTy>
class YieldConstIterator {
public:
    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;

    typedef YieldConstIterator<ResTy, IndexTy> mytype_t;
    typedef ResTy value_type;

    YieldConstIterator(index_t index, yield_function_t yieldFunction) :
            mIndex(index),
            mYieldFunction(yieldFunction) {}

    mytype_t &operator++() {
        ++mIndex;
        return *this;
    }

    const value_type operator*() const {
        return mYieldFunction(mIndex);
    }

    bool operator!=(const mytype_t &r) const {
        return mIndex != r.mIndex;
    }

protected:

    index_t mIndex;
    yield_function_t mYieldFunction;
};

template<typename ResTy, typename IndexTy>
class YieldIterator : public YieldConstIterator<ResTy, IndexTy> {
public:

    typedef YieldConstIterator<ResTy, IndexTy> parent_t;

    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;
    typedef ResTy value_type;

    YieldIterator(index_t index, yield_function_t yieldFunction) :
            parent_t(index, yieldFunction) {}

    value_type operator*() {
        return parent_t::mYieldFunction(parent_t::mIndex);
    }
};

template<typename IndexTy>
struct Range {
public:
    typedef IndexTy index_t;
    typedef Range<IndexTy> mytype_t;

    index_t begin;
    index_t end;
};

template<typename ResTy, typename IndexTy>
class GeneratorCollection {
public:

    typedef Range<IndexTy> range_t;

    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;
    typedef YieldIterator<ResTy, IndexTy> iterator;
    typedef YieldConstIterator<ResTy, IndexTy> const_iterator;

    GeneratorCollection(range_t range, const yield_function_t &yieldF) :
            mRange(range),
            mYieldFunction(yieldF) {}

    iterator begin() {
        return iterator(mRange.begin, mYieldFunction);
    }

    iterator end() {
        return iterator(mRange.end, mYieldFunction);
    }

    const_iterator begin() const {
        return const_iterator(mRange.begin, mYieldFunction);
    }

    const_iterator end() const {
        return const_iterator(mRange.end, mYieldFunction);
    }

private:
    range_t mRange;
    yield_function_t mYieldFunction;
};

template<typename ResTy, typename IndexTy>
class Generator {
public:
    typedef IndexTy index_t;
    typedef ResTy res_t;
    typedef typename yield_function<res_t, index_t>::type yield_function_t;

    typedef Generator<ResTy, IndexTy> mytype_t;
    typedef Range<IndexTy> parent_t;
    typedef GeneratorCollection<ResTy, IndexTy> finalized_emitter_t;
    typedef  Range<IndexTy> range_t;

protected:
    Generator(range_t range) : mRange(range) {}
public:
    static mytype_t on_range(index_t begin, index_t end) {
        return mytype_t({ begin, end });
    }

    finalized_emitter_t yield(yield_function_t f) {
        return finalized_emitter_t(mRange, f);
    }
protected:

    range_t mRange;
};      

回答 10

这个答案适用于C语言(因此我认为也适用于C ++语言)

#include <stdio.h>

const uint64_t MAX = 1ll<<32;

typedef struct {
    uint64_t i, j;
} Pair;

Pair* generate_pairs()
{
    static uint64_t i = 0;
    static uint64_t j = 0;
    
    Pair p = {i,j};
    if(j++ < MAX)
    {
        return &p;
    }
        else if(++i < MAX)
    {
        p.i++;
        p.j = 0;
        j = 0;
        return &p;
    }
    else
    {
        return NULL;
    }
}

int main()
{
    while(1)
    {
        Pair *p = generate_pairs();
        if(p != NULL)
        {
            //printf("%d,%d\n",p->i,p->j);
        }
        else
        {
            //printf("end");
            break;
        }
    }
    return 0;
}

这是一种模拟生成器的简单,非面向对象的方法。这按我的预期工作。

This answer works in C (and hence i think works in c++ too)

#include <stdio.h>

const uint64_t MAX = 1ll<<32;

typedef struct {
    uint64_t i, j;
} Pair;

Pair* generate_pairs()
{
    static uint64_t i = 0;
    static uint64_t j = 0;
    
    Pair p = {i,j};
    if(j++ < MAX)
    {
        return &p;
    }
        else if(++i < MAX)
    {
        p.i++;
        p.j = 0;
        j = 0;
        return &p;
    }
    else
    {
        return NULL;
    }
}

int main()
{
    while(1)
    {
        Pair *p = generate_pairs();
        if(p != NULL)
        {
            //printf("%d,%d\n",p->i,p->j);
        }
        else
        {
            //printf("end");
            break;
        }
    }
    return 0;
}

This is simple, non object-oriented way to mimic a generator. This worked as expected for me.


回答 11

正如函数模拟堆栈的概念一样,生成器模拟队列的概念。剩下的就是语义。

附带说明,您始终可以通过使用操作堆栈而不是数据来模拟带有堆栈的队列。实际上,这意味着您可以通过返回一对来实现类似队列的行为,该对的第二个值要么具有要调用的下一个函数,要么表明我们没有值。但是,这比收益与收益的关系更为普遍。它允许模拟任何值的队列,而不是生成器期望的同类值,但是无需保留完整的内部队列。

更具体地说,由于C ++对队列没有自然的抽象,因此您需要使用在内部实现队列的构造。因此,使用迭代器给出示例的答案是该概念的不错实现。

这实际上意味着,如果您只想快速地进行操作,然后使用队列值,就可以使用生成器产生的值,那么您可以使用准系统队列功能来实现某些功能。

Just as a function simulates the concept of a stack, generators simulate the concept of a queue. The rest is semantics.

As a side note, you can always simulate a queue with a stack by using a stack of operations instead of data. What that practically means is that you can implement a queue-like behavior by returning a pair, the second value of which either has the next function to be called or indicates that we are out of values. But this is more general than what yield vs return does. It allows to simulate a queue of any values rather than homogeneous values that you expect from a generator, but without keeping a full internal queue.

More specifically, since C++ does not have a natural abstraction for a queue, you need to use constructs which implement a queue internally. So the answer which gave the example with iterators is a decent implementation of the concept.

What this practically means is that you can implement something with bare-bones queue functionality if you just want something quick and then consume queue’s values just as you would consume values yielded from a generator.


如何从命令行编译Visual Studio项目?

问题:如何从命令行编译Visual Studio项目?

我正在为使用MonotoneCMake,Visual Studio Express 2008和自定义测试的大型C ++解决方案编写结帐,构建,分发,测试和提交周期的脚本。

所有其他部分似乎都非常简单明了,但是我不明白如何在没有GUI的情况下编译Visual Studio解决方案。

该脚本是用Python编写的,但是给出的答案允许我仅调用os.system。

I’m scripting the checkout, build, distribution, test, and commit cycle for a large C++ solution that is using Monotone, CMake, Visual Studio Express 2008, and custom tests.

All of the other parts seem pretty straight-forward, but I don’t see how to compile the Visual Studio solution without getting the GUI.

The script is written in Python, but an answer that would allow me to just make a call to: os.system would do.


回答 0

我知道有两种方法可以做到。

方法1
第一种方法(我更喜欢)是使用msbuild

msbuild project.sln /Flags...

方法2
您还可以运行:

vcexpress project.sln /build /Flags...

vcexpress选项立即返回,并且不打印任何输出。我想这可能就是您想要的脚本。

请注意,DevEnv并未随Visual Studio Express 2008一起分发(我花了很多时间试图弄清第一次遇到类似问题的时间)。

因此,最终结果可能是:

os.system("msbuild project.sln /p:Configuration=Debug")

您还需要确保您的环境变量正确,因为默认情况下,系统路径上不包含msbuild和vcexpress。启动Visual Studio构建环境并从那里运行脚本,或者修改Python中的路径(使用os.putenv)。

I know of two ways to do it.

Method 1
The first method (which I prefer) is to use msbuild:

msbuild project.sln /Flags...

Method 2
You can also run:

vcexpress project.sln /build /Flags...

The vcexpress option returns immediately and does not print any output. I suppose that might be what you want for a script.

Note that DevEnv is not distributed with Visual Studio Express 2008 (I spent a lot of time trying to figure that out when I first had a similar issue).

So, the end result might be:

os.system("msbuild project.sln /p:Configuration=Debug")

You’ll also want to make sure your environment variables are correct, as msbuild and vcexpress are not by default on the system path. Either start the Visual Studio build environment and run your script from there, or modify the paths in Python (with os.putenv).


回答 1

MSBuild通常可以正常工作,但是我之前遇到过困难。你可能有更好的运气

devenv YourSolution.sln /Build 

MSBuild usually works, but I’ve run into difficulties before. You may have better luck with

devenv YourSolution.sln /Build 

回答 2

老实说,我必须加2美分。

您可以使用msbuild.exe来完成msbuild.exe的版本很多 。

C:\ Windows \ Microsoft.NET \ Framework64 \ v2.0.50727 \ msbuild.exe C:\ Windows \ Microsoft.NET \ Framework64 \ v3.5 \ msbuild.exe C:\ Windows \ Microsoft.NET \ Framework64 \ v4.0.30319 \ msbuild.exe
C:\ Windows \ Microsoft.NET \ Framework \ v2.0.50727 \ msbuild.exe C:\ Windows \ Microsoft.NET \ Framework \ v3.5 \ msbuild.exe C:\ Windows \ Microsoft.NET \ Framework \ v4.0.30319 \ msbuild.exe

使用您需要的版本。基本上,您必须使用最后一个。

C:\ Windows \ Microsoft.NET \ Framework64 \ v4.0.30319 \ msbuild.exe

那么怎么做。

  1. 运行命令窗口

  2. 输入msbuild.exe的路径

C:\ Windows \ Microsoft.NET \ Framework64 \ v4.0.30319 \ msbuild.exe

  1. 输入项目解决方案的路径,例如

“ C:\ Users \ Clark.Kent \ Documents \ visual studio 2012 \ Projects \ WpfApplication1 \ WpfApplication1.sln”

  1. 在解决方案路径后添加所需的任何标志。

  2. ENTER

请注意,您可以获得有关所有可能标记的帮助,例如

C:\ Windows \ Microsoft.NET \ Framework64 \ v4.0.30319 \ msbuild.exe / help

To be honest I have to add my 2 cents.

You can do it with msbuild.exe. There are many version of the msbuild.exe.

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\msbuild.exe C:\Windows\Microsoft.NET\Framework64\v3.5\msbuild.exe C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe
C:\Windows\Microsoft.NET\Framework\v2.0.50727\msbuild.exe C:\Windows\Microsoft.NET\Framework\v3.5\msbuild.exe C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe

Use version you need. Basically you have to use the last one.

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe

So how to do it.

  1. Run the COMMAND window

  2. Input the path to msbuild.exe

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe

  1. Input the path to the project solution like

“C:\Users\Clark.Kent\Documents\visual studio 2012\Projects\WpfApplication1\WpfApplication1.sln”

  1. Add any flags you need after the solution path.

  2. Press ENTER

Note you can get help about all possible flags like

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\msbuild.exe /help


回答 3

使用msbuild其他人指出的方法对我有用,但我需要做的不止于此。首先,msbuild需要访问编译器。这可以通过运行以下命令来完成:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"

然后msbuild不在我的$ PATH中,所以我不得不通过它的显式路径运行它:

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" myproj.sln

最后,我的项目使用了诸如的一些变量$(VisualStudioDir)。看来这些没有被设置,msbuild所以我不得不通过/property选项手动设置它们:

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" /property:VisualStudioDir="C:\Users\Administrator\Documents\Visual Studio 2013" myproj.sln

那行最终使我能够编译我的项目。

奖励:命令行工具使用30天后似乎不需要注册,就像基于GUI的“免费” Visual Studio Community版本一样。有了Microsoft注册要求,该版本几乎是免费的。如果有的话免费在Facebook上…

Using msbuild as pointed out by others worked for me but I needed to do a bit more than just that. First of all, msbuild needs to have access to the compiler. This can be done by running:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat"

Then msbuild was not in my $PATH so I had to run it via its explicit path:

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" myproj.sln

Lastly, my project was making use of some variables like $(VisualStudioDir). It seems those do not get set by msbuild so I had to set them manually via the /property option:

"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" /property:VisualStudioDir="C:\Users\Administrator\Documents\Visual Studio 2013" myproj.sln

That line then finally allowed me to compile my project.

Bonus: it seems that the command line tools do not require a registration after 30 days of using them like the “free” GUI-based Visual Studio Community edition does. With the Microsoft registration requirement in place, that version is hardly free. Free-as-in-facebook if anything…


回答 4

MSBuild是您的朋友。

msbuild "C:\path to solution\project.sln"

MSBuild is your friend.

msbuild "C:\path to solution\project.sln"

回答 5

DEVENV在许多情况下都能很好地工作,但是在WIXPROJ上构建我的WIX安装程序时,我得到的只是Out日志中的“ CATASTROPHIC”错误。

这有效:MSBUILD /Path/PROJECT.WIXPROJ / t:Build / p:Configuration =发布

DEVENV works well in many cases, but on a WIXPROJ to build my WIX installer, all I got is “CATASTROPHIC” error in the Out log.

This works: MSBUILD /Path/PROJECT.WIXPROJ /t:Build /p:Configuration=Release


基准测试(使用BLAS的python与c ++)和(numpy)

问题:基准测试(使用BLAS的python与c ++)和(numpy)

我想编写一个程序,该程序广泛使用BLAS和LAPACK线性代数功能。由于性能是一个问题,因此我做了一些基准测试,想知道我采用的方法是否合法。

可以说,我有三个参赛者,并希望通过一个简单的矩阵矩阵乘法来测试他们的表现。参赛者是:

  1. Numpy,仅使用的功能dot
  2. Python,通过共享对象调用BLAS功能。
  3. C ++,通过共享库调用BLAS功能。

情境

我为不同的尺寸实现了矩阵矩阵乘法ii为5的增量和matricies运行5-500 m1m2设置了这样的:

m1 = numpy.random.rand(i,i).astype(numpy.float32)
m2 = numpy.random.rand(i,i).astype(numpy.float32)

1.脾气暴躁

使用的代码如下所示:

tNumpy = timeit.Timer("numpy.dot(m1, m2)", "import numpy; from __main__ import m1, m2")
rNumpy.append((i, tNumpy.repeat(20, 1)))

2. Python,通过共享库调用BLAS

具有功能

_blaslib = ctypes.cdll.LoadLibrary("libblas.so")
def Mul(m1, m2, i, r):

    no_trans = c_char("n")
    n = c_int(i)
    one = c_float(1.0)
    zero = c_float(0.0)

    _blaslib.sgemm_(byref(no_trans), byref(no_trans), byref(n), byref(n), byref(n), 
            byref(one), m1.ctypes.data_as(ctypes.c_void_p), byref(n), 
            m2.ctypes.data_as(ctypes.c_void_p), byref(n), byref(zero), 
            r.ctypes.data_as(ctypes.c_void_p), byref(n))

测试代码如下:

r = numpy.zeros((i,i), numpy.float32)
tBlas = timeit.Timer("Mul(m1, m2, i, r)", "import numpy; from __main__ import i, m1, m2, r, Mul")
rBlas.append((i, tBlas.repeat(20, 1)))

3. c ++,通过共享库调用BLAS

现在,c ++代码自然会更长一些,因此我将信息减少到最低限度。
我用

void* handle = dlopen("libblas.so", RTLD_LAZY);
void* Func = dlsym(handle, "sgemm_");

我这样测量时间gettimeofday

gettimeofday(&start, NULL);
f(&no_trans, &no_trans, &dim, &dim, &dim, &one, A, &dim, B, &dim, &zero, Return, &dim);
gettimeofday(&end, NULL);
dTimes[j] = CalcTime(start, end);

这里j是运行20次的循环。我计算经过的时间

double CalcTime(timeval start, timeval end)
{
double factor = 1000000;
return (((double)end.tv_sec) * factor + ((double)end.tv_usec) - (((double)start.tv_sec) * factor + ((double)start.tv_usec))) / factor;
}

结果

结果如下图所示:

问题

  1. 您认为我的方法是否公平,还是可以避免一些不必要的开销?
  2. 您是否希望结果显示出c ++和python方法之间的巨大差异?两者都使用共享对象进行计算。
  3. 由于我宁愿在程序中使用python,在调用BLAS或LAPACK例程时该如何做才能提高性能?

下载

完整的基准可以在这里下载。(塞巴斯蒂安(JF Sebastian)使该链接成为可能^^)

I would like to write a program that makes extensive use of BLAS and LAPACK linear algebra functionalities. Since performance is an issue I did some benchmarking and would like know, if the approach I took is legitimate.

I have, so to speak, three contestants and want to test their performance with a simple matrix-matrix multiplication. The contestants are:

  1. Numpy, making use only of the functionality of dot.
  2. Python, calling the BLAS functionalities through a shared object.
  3. C++, calling the BLAS functionalities through a shared object.

Scenario

I implemented a matrix-matrix multiplication for different dimensions i. i runs from 5 to 500 with an increment of 5 and the matricies m1 and m2 are set up like this:

m1 = numpy.random.rand(i,i).astype(numpy.float32)
m2 = numpy.random.rand(i,i).astype(numpy.float32)

1. Numpy

The code used looks like this:

tNumpy = timeit.Timer("numpy.dot(m1, m2)", "import numpy; from __main__ import m1, m2")
rNumpy.append((i, tNumpy.repeat(20, 1)))

2. Python, calling BLAS through a shared object

With the function

_blaslib = ctypes.cdll.LoadLibrary("libblas.so")
def Mul(m1, m2, i, r):

    no_trans = c_char("n")
    n = c_int(i)
    one = c_float(1.0)
    zero = c_float(0.0)

    _blaslib.sgemm_(byref(no_trans), byref(no_trans), byref(n), byref(n), byref(n), 
            byref(one), m1.ctypes.data_as(ctypes.c_void_p), byref(n), 
            m2.ctypes.data_as(ctypes.c_void_p), byref(n), byref(zero), 
            r.ctypes.data_as(ctypes.c_void_p), byref(n))

the test code looks like this:

r = numpy.zeros((i,i), numpy.float32)
tBlas = timeit.Timer("Mul(m1, m2, i, r)", "import numpy; from __main__ import i, m1, m2, r, Mul")
rBlas.append((i, tBlas.repeat(20, 1)))

3. c++, calling BLAS through a shared object

Now the c++ code naturally is a little longer so I reduce the information to a minimum.
I load the function with

void* handle = dlopen("libblas.so", RTLD_LAZY);
void* Func = dlsym(handle, "sgemm_");

I measure the time with gettimeofday like this:

gettimeofday(&start, NULL);
f(&no_trans, &no_trans, &dim, &dim, &dim, &one, A, &dim, B, &dim, &zero, Return, &dim);
gettimeofday(&end, NULL);
dTimes[j] = CalcTime(start, end);

where j is a loop running 20 times. I calculate the time passed with

double CalcTime(timeval start, timeval end)
{
double factor = 1000000;
return (((double)end.tv_sec) * factor + ((double)end.tv_usec) - (((double)start.tv_sec) * factor + ((double)start.tv_usec))) / factor;
}

Results

The result is shown in the plot below:

Questions

  1. Do you think my approach is fair, or are there some unnecessary overheads I can avoid?
  2. Would you expect that the result would show such a huge discrepancy between the c++ and python approach? Both are using shared objects for their calculations.
  3. Since I would rather use python for my program, what could I do to increase the performance when calling BLAS or LAPACK routines?

Download

The complete benchmark can be downloaded here. (J.F. Sebastian made that link possible^^)


回答 0

我已经执行了您的基准测试。我的机器上C ++和numpy之间没有区别:

您认为我的方法是否公平,还是可以避免一些不必要的开销?

由于结果没有差异,因此看起来很公平。

您是否希望结果显示出c ++和python方法之间的巨大差异?两者都使用共享对象进行计算。

没有。

由于我宁愿在程序中使用python,在调用BLAS或LAPACK例程时该如何做才能提高性能?

确保numpy在系统上使用BLAS / LAPACK库的优化版本。

I’ve run your benchmark. There is no difference between C++ and numpy on my machine:

Do you think my approach is fair, or are there some unnecessary overheads I can avoid?

It seems fair due to there is no difference in results.

Would you expect that the result would show such a huge discrepancy between the c++ and python approach? Both are using shared objects for their calculations.

No.

Since I would rather use python for my program, what could I do to increase the performance when calling BLAS or LAPACK routines?

Make sure that numpy uses optimized version of BLAS/LAPACK libraries on your system.


回答 1

更新(30.07.2014):

我在新的HPC上重新运行基准测试。硬件和软件堆栈都与原始答案中的设置有所不同。

我将结果放在Google电子表格中(还包含原始答案的结果)。

硬件

我们的HPC有两个不同的节点,一个带有Intel Sandy Bridge CPU,一个带有较新的Ivy Bridge CPU:

桑迪(MKL,OpenBLAS,ATLAS):

  • CPU:2 x 16 Intel(R)Xeon(R)E2560 Sandy Bridge @ 2.00GHz(16核心)
  • 内存:64 GB

常春藤(MKL,OpenBLAS,ATLAS):

  • CPU:2.80GHz @ 2 x 20英特尔®至强®E2680 V2常春藤桥(20核,HT = 40核)
  • 内存:256 GB

软件

该软件堆栈用于两个节点的sam。代替GotoBLAS2OpenBLAS被使用并且也有一个多线程的ATLAS BLAS它被设置为8个线程(硬编码)。

  • 操作系统:Suse
  • 英特尔编译器:ictce-5.3.0
  • 脾气暴躁的: 1.8.0
  • OpenBLAS: 0.2.6
  • ATLAS:: 3.8.4

点产品基准

基准代码与以下相同。但是对于新机器,我还运行了50008000矩阵尺寸的基准测试。
下表包含原始答案的基准测试结果(重命名为:MKL-> Nehalem MKL,Netlib Blas-> Nehalem Netlib BLAS等)

单线程性能:

多线程性能(8个线程):

线程数与矩阵大小(Ivy Bridge MKL)

基准套件

单线程性能:

多线程(8个线程)性能:

结论

新的基准测试结果类似于原始答案中的结果。OpenBLASMKL的性能相同,但特征值测试除外。的特征值测试仅执行相当好上OpenBLAS单线程模式。在多线程模式下,性能较差。

“矩阵大小VS线程图表”也表明,虽然MKL以及OpenBLAS通常与核/线程的数量很好地扩展,这取决于基质的大小。对于较小的矩阵,添加更多内核不会大大提高性能。

Sandy BridgeIvy Bridge的性能也提高了大约30%,这可能是由于更高的时钟速率(+ 0.8 Ghz)和/或更好的体系结构所致。


原始答案(04.10.2011):

前段时间,我不得不优化一些使用numpy和BLAS用python编写的线性代数计算/算法,因此我对不同的numpy / BLAS配置进行了基准测试。

我专门测试了:

  • 用ATLAS调皮
  • Numpy与GotoBlas2(1.13)
  • 用MKL调皮(11.1 / 073)
  • Numpy with Accelerate Framework(Mac OS X)

我确实运行了两个不同的基准测试:

  1. 大小不同的矩阵的简单点积
  2. 基准套件可在此处找到。

这是我的结果:

机器

Linux(MKL,ATLAS,No-MKL,GotoBlas2):

  • 操作系统:Ubuntu Lucid 10.4 64 Bit。
  • CPU:2 x 4英特尔(R)至强(R)E5504 @ 2.00GHz(8核)
  • 内存:24 GB
  • 英特尔编译器:11.1 / 073
  • Scipy:0.8
  • 脾气暴躁的:1.5

Mac Book Pro(加速框架):

  • 操作系统:Mac OS X Snow Leopard(10.6)
  • CPU:1个Intel Core 2 Duo 2.93 Ghz(2个内核)
  • 内存:4 GB
  • 西皮:0.7
  • 脾气暴躁的:1.3

Mac Server(加速框架):

  • 操作系统:Mac OS X Snow Leopard Server(10.6)
  • CPU:4 X Intel(R)Xeon(R)E5520 @ 2.26 Ghz(8核)
  • 内存:4 GB
  • Scipy:0.8
  • 脾气暴躁的:1.5.1

点产品基准

代码

import numpy as np
a = np.random.random_sample((size,size))
b = np.random.random_sample((size,size))
%timeit np.dot(a,b)

结果

    系统| 大小= 1000 | 大小= 2000 | 大小= 3000 |
netlib BLAS | 1350毫秒| 10900毫秒| 39200毫秒|    
ATLAS(1 CPU)| 314毫秒| 2560毫秒| 8700毫秒|     
MKL(1 CPU)| 268毫秒| 2110毫秒| 7120毫秒|
MKL(2个CPU)| -| -| 3660毫秒|
MKL(8个CPU)| 39毫秒| 319毫秒| 1000毫秒|
GotoBlas2(1 CPU)| 266毫秒| 2100毫秒| 7280毫秒|
GotoBlas2(2个CPU)| 139毫秒| 1009毫秒| 3690毫秒|
GotoBlas2(8个CPU)| 54毫秒| 389毫秒| 1250毫秒|
Mac OS X(1个CPU)| 143毫秒| 1060毫秒| 3605毫秒|
Mac服务器(1个CPU)| 92毫秒| 714毫秒| 2130毫秒|

基准套件

代码
有关基准套件的更多信息,请参见此处

结果

    系统| 特征值| svd | det | inv | 点|
netlib BLAS | 1688毫秒| 13102毫秒| 438毫秒| 2155毫秒| 3522毫秒|
ATLAS(1 CPU)| 1210毫秒| 5897毫秒| 170毫秒| 560毫秒| 893毫秒|
MKL(1 CPU)| 691毫秒| 4475毫秒| 141毫秒| 450毫秒| 736毫秒|
MKL(2个CPU)| 552毫秒| 2718毫秒| 96毫秒| 267毫秒| 423毫秒|
MKL(8个CPU)| 525毫秒| 1679毫秒| 60毫秒| 137毫秒| 197毫秒|  
GotoBlas2(1 CPU)| 2124毫秒| 4636毫秒| 147毫秒| 456毫秒| 743毫秒|
GotoBlas2(2个CPU)| 1560毫秒| 3278毫秒| 116毫秒| 295毫秒| 460毫秒|
GotoBlas2(8个CPU)| 741毫秒| 2914毫秒| 82毫秒| 262毫秒| 192毫秒|
Mac OS X(1个CPU)| 948毫秒| 4339毫秒| 151毫秒| 318毫秒| 566毫秒|
Mac服务器(1个CPU)| 1033毫秒| 3645毫秒| 99毫秒| 232毫秒| 342毫秒|

安装

安装MKL包括安装完整的英特尔编译器套件,这是相当直截了当。但是,由于存在一些错误/问题,使用MKL支持配置和编译numpy有点麻烦。

GotoBlas2是一个小软件包,可以轻松地编译为共享库。但是,由于存在错误,您必须在构建共享库后重新创建共享库才能与numpy一起使用。
除了这种构建之外,由于某些原因,它无法用于多个目标平台。因此,我必须为每个平台都创建一个.so文件,我要为其提供优化的libgoto2.so文件。

如果您从Ubuntu的存储库中安装numpy,它将自动安装并配置numpy以使用ATLAS。从源代码安装ATLAS可能需要一些时间,并且需要一些其他步骤(fortran等)。

如果您在具有FinkMac Ports的Mac OS X机器上安装numpy,它将配置numpy以使用ATLASApple的Accelerate Framework。您可以通过在numpy.core._dotblas文件上运行ldd 或调用numpy.show_config()进行检查

结论

MKL紧随其后的是GotoBlas2
特征值测试中,GotoBlas2的表现令人惊讶地比预期的差。不知道为什么会这样。
Apple的Accelerate Framework的性能非常好,特别是在单线程模式下(与其他BLAS实现相比)。

GotoBlas2MKL可以很好地随线程数扩展。因此,如果您必须处理在多个线程上运行的大型矩阵,将会很有帮助。

无论如何都不要使用默认的netlib blas实现,因为它对于任何严肃的计算工作来说都太慢了。

在我们的集群上,我还安装了AMD的ACML,性能类似于MKLGotoBlas2。我没有任何强硬的数字。

我个人建议使用GotoBlas2,因为它更容易安装且免费。

如果您想用C ++ / C进行编码,还可以查看Eigen3,它在某些情况下应该胜过MKL / GotoBlas2,并且非常易于使用。

UPDATE (30.07.2014):

I re-run the the benchmark on our new HPC. Both the hardware as well as the software stack changed from the setup in the original answer.

I put the results in a google spreadsheet (contains also the results from the original answer).

Hardware

Our HPC has two different nodes one with Intel Sandy Bridge CPUs and one with the newer Ivy Bridge CPUs:

Sandy (MKL, OpenBLAS, ATLAS):

  • CPU: 2 x 16 Intel(R) Xeon(R) E2560 Sandy Bridge @ 2.00GHz (16 Cores)
  • RAM: 64 GB

Ivy (MKL, OpenBLAS, ATLAS):

  • CPU: 2 x 20 Intel(R) Xeon(R) E2680 V2 Ivy Bridge @ 2.80GHz (20 Cores, with HT = 40 Cores)
  • RAM: 256 GB

Software

The software stack is for both nodes the sam. Instead of GotoBLAS2, OpenBLAS is used and there is also a multi-threaded ATLAS BLAS that is set to 8 threads (hardcoded).

  • OS: Suse
  • Intel Compiler: ictce-5.3.0
  • Numpy: 1.8.0
  • OpenBLAS: 0.2.6
  • ATLAS:: 3.8.4

Dot-Product Benchmark

Benchmark-code is the same as below. However for the new machines I also ran the benchmark for matrix sizes 5000 and 8000.
The table below includes the benchmark results from the original answer (renamed: MKL –> Nehalem MKL, Netlib Blas –> Nehalem Netlib BLAS, etc)

Single threaded performance:

Multi threaded performance (8 threads):

Threads vs Matrix size (Ivy Bridge MKL):

Benchmark Suite

Single threaded performance:

Multi threaded (8 threads) performance:

Conclusion

The new benchmark results are similar to the ones in the original answer. OpenBLAS and MKL perform on the same level, with the exception of Eigenvalue test. The Eigenvalue test performs only reasonably well on OpenBLAS in single threaded mode. In multi-threaded mode the performance is worse.

The “Matrix size vs threads chart” also show that although MKL as well as OpenBLAS generally scale well with number of cores/threads,it depends on the size of the matrix. For small matrices adding more cores won’t improve performance very much.

There is also approximately 30% performance increase from Sandy Bridge to Ivy Bridge which might be either due to higher clock rate (+ 0.8 Ghz) and/or better architecture.


Original Answer (04.10.2011):

Some time ago I had to optimize some linear algebra calculations/algorithms which were written in python using numpy and BLAS so I benchmarked/tested different numpy/BLAS configurations.

Specifically I tested:

  • Numpy with ATLAS
  • Numpy with GotoBlas2 (1.13)
  • Numpy with MKL (11.1/073)
  • Numpy with Accelerate Framework (Mac OS X)

I did run two different benchmarks:

  1. simple dot product of matrices with different sizes
  2. Benchmark suite which can be found here.

Here are my results:

Machines

Linux (MKL, ATLAS, No-MKL, GotoBlas2):

  • OS: Ubuntu Lucid 10.4 64 Bit.
  • CPU: 2 x 4 Intel(R) Xeon(R) E5504 @ 2.00GHz (8 Cores)
  • RAM: 24 GB
  • Intel Compiler: 11.1/073
  • Scipy: 0.8
  • Numpy: 1.5

Mac Book Pro (Accelerate Framework):

  • OS: Mac OS X Snow Leopard (10.6)
  • CPU: 1 Intel Core 2 Duo 2.93 Ghz (2 Cores)
  • RAM: 4 GB
  • Scipy: 0.7
  • Numpy: 1.3

Mac Server (Accelerate Framework):

  • OS: Mac OS X Snow Leopard Server (10.6)
  • CPU: 4 X Intel(R) Xeon(R) E5520 @ 2.26 Ghz (8 Cores)
  • RAM: 4 GB
  • Scipy: 0.8
  • Numpy: 1.5.1

Dot product benchmark

Code:

import numpy as np
a = np.random.random_sample((size,size))
b = np.random.random_sample((size,size))
%timeit np.dot(a,b)

Results:

    System        |  size = 1000  | size = 2000 | size = 3000 |
netlib BLAS       |  1350 ms      |   10900 ms  |  39200 ms   |    
ATLAS (1 CPU)     |   314 ms      |    2560 ms  |   8700 ms   |     
MKL (1 CPUs)      |   268 ms      |    2110 ms  |   7120 ms   |
MKL (2 CPUs)      |    -          |       -     |   3660 ms   |
MKL (8 CPUs)      |    39 ms      |     319 ms  |   1000 ms   |
GotoBlas2 (1 CPU) |   266 ms      |    2100 ms  |   7280 ms   |
GotoBlas2 (2 CPUs)|   139 ms      |    1009 ms  |   3690 ms   |
GotoBlas2 (8 CPUs)|    54 ms      |     389 ms  |   1250 ms   |
Mac OS X (1 CPU)  |   143 ms      |    1060 ms  |   3605 ms   |
Mac Server (1 CPU)|    92 ms      |     714 ms  |   2130 ms   |

Benchmark Suite

Code:
For additional information about the benchmark suite see here.

Results:

    System        | eigenvalues   |    svd   |   det  |   inv   |   dot   |
netlib BLAS       |  1688 ms      | 13102 ms | 438 ms | 2155 ms | 3522 ms |
ATLAS (1 CPU)     |   1210 ms     |  5897 ms | 170 ms |  560 ms |  893 ms |
MKL (1 CPUs)      |   691 ms      |  4475 ms | 141 ms |  450 ms |  736 ms |
MKL (2 CPUs)      |   552 ms      |  2718 ms |  96 ms |  267 ms |  423 ms |
MKL (8 CPUs)      |   525 ms      |  1679 ms |  60 ms |  137 ms |  197 ms |  
GotoBlas2 (1 CPU) |  2124 ms      |  4636 ms | 147 ms |  456 ms |  743 ms |
GotoBlas2 (2 CPUs)|  1560 ms      |  3278 ms | 116 ms |  295 ms |  460 ms |
GotoBlas2 (8 CPUs)|   741 ms      |  2914 ms |  82 ms |  262 ms |  192 ms |
Mac OS X (1 CPU)  |   948 ms      |  4339 ms | 151 ms |  318 ms |  566 ms |
Mac Server (1 CPU)|  1033 ms      |  3645 ms |  99 ms |  232 ms |  342 ms |

Installation

Installation of MKL included installing the complete Intel Compiler Suite which is pretty straight forward. However because of some bugs/issues configuring and compiling numpy with MKL support was a bit of a hassle.

GotoBlas2 is a small package which can be easily compiled as a shared library. However because of a bug you have to re-create the shared library after building it in order to use it with numpy.
In addition to this building it for multiple target plattform didn’t work for some reason. So I had to create an .so file for each platform for which i want to have an optimized libgoto2.so file.

If you install numpy from Ubuntu’s repository it will automatically install and configure numpy to use ATLAS. Installing ATLAS from source can take some time and requires some additional steps (fortran, etc).

If you install numpy on a Mac OS X machine with Fink or Mac Ports it will either configure numpy to use ATLAS or Apple’s Accelerate Framework. You can check by either running ldd on the numpy.core._dotblas file or calling numpy.show_config().

Conclusions

MKL performs best closely followed by GotoBlas2.
In the eigenvalue test GotoBlas2 performs surprisingly worse than expected. Not sure why this is the case.
Apple’s Accelerate Framework performs really good especially in single threaded mode (compared to the other BLAS implementations).

Both GotoBlas2 and MKL scale very well with number of threads. So if you have to deal with big matrices running it on multiple threads will help a lot.

In any case don’t use the default netlib blas implementation because it is way too slow for any serious computational work.

On our cluster I also installed AMD’s ACML and performance was similar to MKL and GotoBlas2. I don’t have any numbers tough.

I personally would recommend to use GotoBlas2 because it’s easier to install and it’s free.

If you want to code in C++/C also check out Eigen3 which is supposed to outperform MKL/GotoBlas2 in some cases and is also pretty easy to use.


回答 2

这是另一个基准测试(在Linux上,只需输入make):http : //dl.dropbox.com/u/5453551/blas_call_benchmark.zip

http://dl.dropbox.com/u/5453551/blas_call_benchmark.png

我看不到大型矩阵的不同方法,Numpy,Ctypes和Fortran之间的任何区别。(Fortran而不是C ++ —如果这很重要,则您的基准可能已损坏。)

CalcTime在C ++中的函数似乎有符号错误。... + ((double)start.tv_usec))应该代替... - ((double)start.tv_usec))也许您的基准测试还存在其他错误,例如,在不同的BLAS库之间进行比较,或在不同的BLAS设置(例如线程数)之间进行比较,或者在实时与CPU时间之间进行比较?

编辑:无法计算CalcTime函数中的花括号-可以。

作为准则:如果进行基准测试,请始终将所有代码发布到某个地方。在没有完整代码的情况下对基准进行注释,尤其是在令人惊讶的情况下,通常是无效的。


要找出链接到哪个BLAS Numpy,请执行以下操作:

$Python
Python 2.7.2+(默认值,2011年8月16日,07:24:41) 
linux2上的[GCC 4.6.1]
键入“帮助”,“版权”,“信用”或“许可证”以获取更多信息。
>>>导入numpy.core._dotblas
>>> numpy.core._dotblas .__ file__
'/usr/lib/pymodules/python2.7/numpy/core/_dotblas.so'
>>> 
$ ldd /usr/lib/pymodules/python2.7/numpy/core/_dotblas.so
    linux-vdso.so.1 =>(0x00007fff5ebff000)
    libblas.so.3gf => /usr/lib/libblas.so.3gf(0x00007fbe618b3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6(0x00007fbe61514000)

更新:如果您无法导入numpy.core._dotblas,则您的Numpy正在使用其内部的BLAS后备副本,该副本速度较慢,并且不能用于性能计算!下面来自@Woltan的答复表明,这是他/她在Numpy与Ctypes + BLAS中看到的差异的解释。

要解决这种情况,您需要ATLAS或MKL —查看以下说明:http : //scipy.org/Installing_SciPy/Linux 大多数Linux发行版都随ATLAS一起提供,因此最好的选择是安装其libatlas-dev软件包(名称可能有所不同) 。

Here’s another benchmark (on Linux, just type make): http://dl.dropbox.com/u/5453551/blas_call_benchmark.zip

http://dl.dropbox.com/u/5453551/blas_call_benchmark.png

I do not see essentially any difference between the different methods for large matrices, between Numpy, Ctypes and Fortran. (Fortran instead of C++ — and if this matters, your benchmark is probably broken.)

Your CalcTime function in C++ seems to have a sign error. ... + ((double)start.tv_usec)) should be instead ... - ((double)start.tv_usec)). Perhaps your benchmark also has other bugs, e.g., comparing between different BLAS libraries, or different BLAS settings such as number of threads, or between real time and CPU time?

EDIT: failed to count the braces in the CalcTime function — it’s OK.

As a guideline: if you do a benchmark, please always post all the code somewhere. Commenting on benchmarks, especially when surprising, without having the full code is usually not productive.


To find out which BLAS Numpy is linked against, do:

$ python
Python 2.7.2+ (default, Aug 16 2011, 07:24:41) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy.core._dotblas
>>> numpy.core._dotblas.__file__
'/usr/lib/pymodules/python2.7/numpy/core/_dotblas.so'
>>> 
$ ldd /usr/lib/pymodules/python2.7/numpy/core/_dotblas.so
    linux-vdso.so.1 =>  (0x00007fff5ebff000)
    libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007fbe618b3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe61514000)

UPDATE: If you can’t import numpy.core._dotblas, your Numpy is using its internal fallback copy of BLAS, which is slower, and not meant to be used in performance computing! The reply from @Woltan below indicates that this is the explanation for the difference he/she sees in Numpy vs. Ctypes+BLAS.

To fix the situation, you need either ATLAS or MKL — check these instructions: http://scipy.org/Installing_SciPy/Linux Most Linux distributions ship with ATLAS, so the best option is to install their libatlas-dev package (name may vary).


回答 3

考虑到您对分析的严格要求,迄今为止的结果令我感到惊讶。我将其作为“答案”,但这仅是因为评论时间太长并且确实提供了可能性(尽管我希望您已经考虑过)。

我本来认为numpy / python方法不会为合理复杂度的矩阵增加太多开销,因为随着复杂度的增加,python参与的比例应该很小。我对图右侧的结果更感兴趣,但显示出数量级差异会令人不安。

我想知道您是否正在使用numpy可以利用的最佳算法。从Linux的编译指南中:

“构建FFTW(3.1.2):SciPy版本> = 0.7和Numpy> = 1.2:由于许可证,配置和维护问题,在SciPy> = 0.7和NumPy> = 1.2的版本中,不再支持FFTW。现在使用fftpack的内置版本。如果需要进行分析,有几种方法可以利用FFTW的速度;降级到包含支持的Numpy / Scipy版本;安装或创建自己的FFTW包装器。请参阅http: //developer.berlios.de/projects/pyfftw/作为未经认可的示例。”

你用mkl编译numpy吗?(http://software.intel.com/zh-cn/articles/intel-mkl/)。如果您在Linux上运行,则使用mkl编译numpy的说明如下:http : //www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974(尽管有url)。关键部分是:

[mkl]
library_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/include
mkl_libs = mkl_intel_lp64,mkl_intel_thread,mkl_core 

如果您使用的是Windows,则可以通过以下网址使用mkl获得编译的二进制文件(并且还可以获取pyfftw和许多其他相关算法):http ://www.lfd.uci.edu/~gohlke/pythonlibs/ ,其中包含感谢UC Irvine荧光动力学实验室的Christoph Gohlke。

需要注意的是,无论哪种情况,都有许多许可问题等需要注意的地方,但是intel页面对此进行了解释。同样,我想您已经考虑了这一点,但是如果满足许可要求(在Linux上很容易做到),相对于使用简单的自动构建(甚至不使用FFTW),这将大大加快numpy的工作。我将有兴趣关注这个话题,看看其他人的想法。无论如何,都非常严格,也有很好的问题。感谢您发布。

Given the rigor you’ve shown with your analysis, I’m surprised by the results thus far. I put this as an ‘answer’ but only because it’s too long for a comment and does provide a possibility (though I expect you’ve considered it).

I would’ve thought the numpy/python approach wouldn’t add much overhead for a matrix of reasonable complexity, since as the complexity increases, the proportion that python participates in should be small. I’m more interested in the results on the right hand side of the graph, but orders of magnitude discrepancy shown there would be disturbing.

I wonder if you’re using the best algorithms that numpy can leverage. From the compilation guide for linux:

“Build FFTW (3.1.2): SciPy Versions >= 0.7 and Numpy >= 1.2: Because of license, configuration, and maintenance issues support for FFTW was removed in versions of SciPy >= 0.7 and NumPy >= 1.2. Instead now uses a built-in version of fftpack. There are a couple ways to take advantage of the speed of FFTW if necessary for your analysis. Downgrade to a Numpy/Scipy version that includes support. Install or create your own wrapper of FFTW. See http://developer.berlios.de/projects/pyfftw/ as an un-endorsed example.”

Did you compile numpy with mkl? (http://software.intel.com/en-us/articles/intel-mkl/). If you’re running on linux, the instructions for compiling numpy with mkl are here: http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974 (in spite of url). The key part is:

[mkl]
library_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/include
mkl_libs = mkl_intel_lp64,mkl_intel_thread,mkl_core 

If you’re on windows, you can obtain a compiled binary with mkl, (and also obtain pyfftw, and many other related algorithms) at: http://www.lfd.uci.edu/~gohlke/pythonlibs/, with a debt of gratitude to Christoph Gohlke at the Laboratory for Fluorescence Dynamics, UC Irvine.

Caveat, in either case, there are many licensing issues and so on to be aware of, but the intel page explains those. Again, I imagine you’ve considered this, but if you meet the licensing requirements (which on linux is very easy to do), this would speed up the numpy part a great deal relative to using a simple automatic build, without even FFTW. I’ll be interested to follow this thread and see what others think. Regardless, excellent rigor and excellent question. Thanks for posting it.


将Python程序转换为C / C ++代码?[关闭]

问题:将Python程序转换为C / C ++代码?[关闭]

是否可以将Python程序转换为C / C ++?

我需要实现一些算法,而且我不确定性能差距是否足够大,足以证明我在C / C ++中做的所有痛苦(我不擅长)。我考虑过要编写一个简单的算法,并针对这种转换后的解决方案进行基准测试。如果仅此一项比Python版本要快得多,那么除了在C / C ++中做到这一点,我别无选择。

is it possible to convert a Python program to C/C++?

I need to implement a couple of algorithms, and I’m not sure if the performance gap is big enough to justify all the pain I’d go through when doing it in C/C++ (which I’m not good at). I thought about writing one simple algorithm and benchmark it against such a converted solution. If that alone is significantly faster than the Python version, then I’ll have no other choice than doing it in C/C++.


回答 0

是。看赛顿。它就是这样做的:将Python转换为C以加快速度。

Yes. Look at Cython. It does just that: Converts Python to C for speedups.


回答 1

如果C变体所需的时间减少了x个小时,那么我会花时间让算法更长或更长时间地运行

“投资”在这里不是正确的词。

  1. 用Python构建有效的实现。在完成C版本之前,您将完成此工作很久。

  2. 使用Python分析器衡量性能。解决您发现的所有问题。根据实际需要更改数据结构和算法。您需要完成很长时间才能完成C中的第一个版本。

  3. 如果仍然太慢,请手动将精心设计和精心构建的Python转换为C。

    由于事后观察的工作方式,从现有的Python(使用现有的单元测试和现有的分析数据)执行第二版仍比尝试从头开始执行C代码要快。

这句话很重要。

汤普森首次望远镜制造商的规则制作
四英寸镜然后制作六英寸镜比制造六英寸镜要快。

比尔·麦肯南·
旺研究所

If the C variant needs x hours less, then I’d invest that time in letting the algorithms run longer/again

“invest” isn’t the right word here.

  1. Build a working implementation in Python. You’ll finish this long before you’d finish a C version.

  2. Measure performance with the Python profiler. Fix any problems you find. Change data structures and algorithms as necessary to really do this properly. You’ll finish this long before you finish the first version in C.

  3. If it’s still too slow, manually translate the well-designed and carefully constructed Python into C.

    Because of the way hindsight works, doing the second version from existing Python (with existing unit tests, and with existing profiling data) will still be faster than trying to do the C code from scratch.

This quote is important.

Thompson’s Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.

Bill McKeenan
Wang Institute


回答 2

Shed Skin是“(受限制的)Python到C ++的编译器”。

Shed Skin is “a (restricted) Python-to-C++ compiler”.


回答 3

刚刚在黑客新闻中遇到了这个新工具。

在他们的页面上-“ Nuitka是Python解释器的很好的替代品,它可以编译CPython 2.6、2.7、3.2和3.3提供的每个构造。它将Python转换为C ++程序,然后使用“ libpython”以与以下相同的方式执行CPython以一种非常兼容的方式做到了。”

Just came across this new tool in hacker news.

From their page – “Nuitka is a good replacement for the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.2 and 3.3 offer. It translates the Python into a C++ program that then uses “libpython” to execute in the same way as CPython does, in a very compatible way.”


回答 4

Pythran是除Shed Skin之外还可以转换为C ++的另一种选择。

引用Micha Gorelick和Ian Ozsvald的高性能Python

Pythran是一个Python到C ++的编译器,用于部分numpy支持的Python子集。它的行为有点像Numba和Cython,您可以对函数的参数进行注释,然后由进一步的类型注释和代码专门化来接管。它利用了矢量化可能性和基于OpenMP的并行化可能性。它仅使用Python 2.7运行。

Pythran的一个非常有趣的功能是它将尝试自动发现并行化机会(例如,如果您使用map),并将其转换为并行代码,而无需您付出额外的努力。您也可以使用pragma omp >指令指定并行节;在这方面,它与Cython的OpenMP支持非常相似。

在幕后,Pythran将同时使用普通的Python和numpy代码,并试图将它们积极地编译成非常快的C ++,甚至比Cython的结果还要快。

您应该注意,这个项目还很年轻,并且可能会遇到bug。您还应该注意,开发团队非常友好,往往会在几个小时内修复错误。

Another option – to convert to C++ besides Shed Skin – is Pythran.

To quote High Performance Python by Micha Gorelick and Ian Ozsvald:

Pythran is a Python-to-C++ compiler for a subset of Python that includes partial numpy support. It acts a little like Numba and Cython—you annotate a function’s arguments, and then it takes over with further type annotation and code specialization. It takes advantage of vectorization possibilities and of OpenMP-based parallelization possibilities. It runs using Python 2.7 only.

One very interesting feature of Pythran is that it will attempt to automatically spot parallelization opportunities (e.g., if you’re using a map), and turn this into parallel code without requiring extra effort from you. You can also specify parallel sections using pragma omp > directives; in this respect, it feels very similar to Cython’s OpenMP support.

Behind the scenes, Pythran will take both normal Python and numpy code and attempt to aggressively compile them into very fast C++—even faster than the results of Cython.

You should note that this project is young, and you may encounter bugs; you should also note that the development team are very friendly and tend to fix bugs in a matter of hours.


回答 5

我知道这是一个较旧的主题,但是我想提供我认为是有用的信息。

我个人使用PyPy,使用pip真的很容易安装。我可以互换地使用Python / PyPy解释器,您根本不需要更改代码,我发现它比标准python解释器(Python 2x或3x)快40倍左右。我使用pyCharm Community Edition管理我的代码,我喜欢它。

我喜欢用python编写代码,因为我认为它可以使您更多地专注于任务而不是语言,这对我来说是一个巨大的优势。而且,如果您需要更快的速度,则始终可以将其编译为适用于Windows,Linux或Mac的二​​进制文件(不是直接的,但可以使用其他工具)。根据我的经验,编译时我的速度是PyPy的3.5倍,这比python快140倍。PyPy适用于Python 3x和2x代码,如果使用像PyCharm这样的IDE,则可以很容易地在PyPy,Cython和Python之间互换(尽管需要一些初始学习和设置)。

有人可能在这一点上与我参数,但我发现PyPy比Cython快。但是它们都是不错的选择。

编辑:我想对编译做个简短的说明:编译时,生成的二进制文件比python脚本大得多,因为它在其中建立了所有依赖关系,等等。但是随后您得到了一些明显的好处:速度!,现在,该应用程序可以在没有Python或库的任何机器上运行(取决于所编译的操作系统(如果不是全部,则取决于.lol)),而无需使用Python或库,它还会使您的代码变得模糊,并且在某种程度上从技术上来说已经准备好进行生产了。一些编译器还会生成C代码,我并没有真正查看或查看它是否有用或只是乱码。祝好运。

希望有帮助。

I know this is an older thread but I wanted to give what I think to be helpful information.

I personally use PyPy which is really easy to install using pip. I interchangeably use Python/PyPy interpreter, you don’t need to change your code at all and I’ve found it to be roughly 40x faster than the standard python interpreter (Either Python 2x or 3x). I use pyCharm Community Edition to manage my code and I love it.

I like writing code in python as I think it lets you focus more on the task than the language, which is a huge plus for me. And if you need it to be even faster, you can always compile to a binary for Windows, Linux, or Mac (not straight forward but possible with other tools). From my experience, I get about 3.5x speedup over PyPy when compiling, meaning 140x faster than python. PyPy is available for Python 3x and 2x code and again if you use an IDE like PyCharm you can interchange between say PyPy, Cython, and Python very easily (takes a little of initial learning and setup though).

Some people may argue with me on this one, but I find PyPy to be faster than Cython. But they’re both great choices though.

Edit: I’d like to make another quick note about compiling: when you compile, the resulting binary is much bigger than your python script as it builds all dependencies into it, etc. But then you get a few distinct benefits: speed!, now the app will work on any machine (depending on which OS you compiled for, if not all. lol) without Python or libraries, it also obfuscates your code and is technically ‘production’ ready (to a degree). Some compilers also generate C code, which I haven’t really looked at or seen if it’s useful or just gibberish. Good luck.

Hope that helps.


回答 6

我意识到缺少一个全新解决方案的答案。如果在代码中使用了Numpy,我建议尝试使用Pythran:

http://pythran.readthedocs.io/

对于我尝试过的功能,Pythran提供了非常好的结果。所产生的功能与已编写好的Fortran代码一样快(或稍慢),并且比(相当优化的)Cython解决方案快一点。

与Cython相比,优点是您只需要在针对Numpy优化的Python函数上使用Pythran,这意味着您不必扩展循环并为循环中的所有变量添加类型。Pythran花费时间分析代码,因此可以理解上的操作numpy.ndarray

与Numba或其他基于即时编译的项目相比,这也是一个巨大的优势(据我所知),对于这些项目,您必须扩展循环才能真正有效。然后仅使用CPython和Numpy,带有循环的代码效率非常低下…

Pythran的缺点:没有类!但是由于只需要编译真正需要优化的功能,所以它并不是很烦人。

另一点:Pythran支持(非常容易)OpenMP并行性。但是我不支持mpi4py …

I realize that an answer on a quite new solution is missing. If Numpy is used in the code, I would advice to try Pythran:

http://pythran.readthedocs.io/

For the functions I tried, Pythran gives extremely good results. The resulting functions are as fast as well written Fortran code (or only slightly slower) and a little bit faster than the (quite optimized) Cython solution.

The advantage compared to Cython is that you just have to use Pythran on the Python function optimized for Numpy, meaning that you do not have to expand the loops and add types for all variables in the loop. Pythran takes its time to analyse the code so it understands the operations on numpy.ndarray.

It is also a huge advantage compared to Numba or other projects based on just-in-time compilation for which (to my knowledge), you have to expand the loops to be really efficient. And then the code with the loops becomes very very inefficient using only CPython and Numpy…

A drawback of Pythran: no classes! But since only the functions that really need to be optimized have to be compiled, it is not very annoying.

Another point: Pythran supports well (and very easily) OpenMP parallelism. But I don’t think mpi4py is supported…


回答 7

http://code.google.com/p/py2c/似乎很可能-他们还在自己的网站上提到:Cython,Shedskin和RPython,并确认他们正在将Python代码转换为比C快得多的纯C / C ++。 / C ++充满了Python API调用。注意:我还没有尝试过,但是我要去做。

http://code.google.com/p/py2c/ looks like a possibility – they also mention on their site: Cython, Shedskin and RPython and confirm that they are converting Python code to pure C/C++ which is much faster than C/C++ riddled with Python API calls. Note: I haven’t tried it but I am going to..


TensorFlow,为什么选择python语言?

问题:TensorFlow,为什么选择python语言?

我最近开始研究深度学习和其他ML技术,并开始寻找简化构建网络并对其进行培训的框架,然后我发现TensorFlow在该领域经验不足,对我来说,速度似乎是如果与深度学习一起工作,那么使大型机器学习系统变得更大的重要因素,那么为什么Google选择python来制造TensorFlow?用一种可以编译且无法解释的语言来编写代码会更好吗?

使用Python而不是像C ++这样的语言进行机器学习有什么优势?

I recently started studying deep learning and other ML techniques, and I started searching for frameworks that simplify the process of build a net and training it, then I found TensorFlow, having little experience in the field, for me, it seems that speed is a big factor for making a big ML system even more if working with deep learning, so why python was chosen by Google to make TensorFlow? Wouldn’t it be better to make it over an language that can be compiled and not interpreted?

What are the advantages of using Python over a language like C++ for machine learning?


回答 0

关于TensorFlow的最重要的认识是,在大多数情况下,内核不是用Python编写的:它是由高度优化的C ++和CUDA(Nvidia用于GPU编程的语言)结合而成。反过来,大多数情况是通过使用Eigen(高性能C ++和CUDA数值库)和NVidia的cuDNN(为NVidia GPU进行了非常优化的DNN库,用于诸如卷积之类的功能)而发生的。

TensorFlow的模型是程序员使用“某种语言”(很可能是Python!)来表达模型。该模型以TensorFlow构造编写,例如:

h1 = tf.nn.relu(tf.matmul(l1, W1) + b1)
h2 = ...

在运行Python时实际上并未执行。相反,实际创建的是一个数据流图,该表示接受特定的输入,应用特定的操作,将结果作为输入提供给其他操作,等等。 该模型由快速的C ++代码执行,并且在大多数情况下,操作之间传递的数据永远不会复制回Python代码

然后,程序员通过拉上节点来“驱动”该模型的执行-通常在Python中进行训练,有时在Python中甚至在原始C ++中进行服务:

sess.run(eval_results)

这个Python(或C ++函数调用)使用对C ++的进程内调用或针对分布式版本的RPC来调用C ++ TensorFlow服务器以使其执行,然后将结果复制回去。

因此,话虽如此,让我们重新表述一下问题:为什么TensorFlow为什么选择Python作为表达和控制模型训练的第一种得到良好支持的语言?

答案很简单:对于许多数据科学家和机器学习专家来说,Python可能最舒适的语言,它易于集成并可以控制C ++后端,同时在内部和外部也广泛使用。和开放源代码。鉴于使用TensorFlow的基本模型,Python的性能并不那么重要,因此很自然。NumPy的巨大优势还在于它可以在Python中轻松进行预处理-同时具有高性能-在将其输入TensorFlow进行真正占用大量CPU的处理之前。

表示执行模型时不使用的模型也有很多复杂性-形状推断(例如,如果您做matmul(A,B),结果数据的形状是什么?)和自动梯度计算。事实证明,能够用Python表达这些内容真是太好了,尽管从长远来看,我认为它们可能会转移到C ++后端以使添加其他语言变得更加容易。

(当然,希望是将来支持其他语言来创建和表达模型。使用其他几种语言来运行推理已经非常简单了-C ++现在可以工作了,Facebook的某人贡献了Go绑定,我们现在正在对其进行审查。等)

The most important thing to realize about TensorFlow is that, for the most part, the core is not written in Python: It’s written in a combination of highly-optimized C++ and CUDA (Nvidia’s language for programming GPUs). Much of that happens, in turn, by using Eigen (a high-performance C++ and CUDA numerical library) and NVidia’s cuDNN (a very optimized DNN library for NVidia GPUs, for functions such as convolutions).

The model for TensorFlow is that the programmer uses “some language” (most likely Python!) to express the model. This model, written in the TensorFlow constructs such as:

h1 = tf.nn.relu(tf.matmul(l1, W1) + b1)
h2 = ...

is not actually executed when the Python is run. Instead, what’s actually created is a dataflow graph that says to take particular inputs, apply particular operations, supply the results as the inputs to other operations, and so on. This model is executed by fast C++ code, and for the most part, the data going between operations is never copied back to the Python code.

Then the programmer “drives” the execution of this model by pulling on nodes — for training, usually in Python, and for serving, sometimes in Python and sometimes in raw C++:

sess.run(eval_results)

This one Python (or C++ function call) uses either an in-process call to C++ or an RPC for the distributed version to call into the C++ TensorFlow server to tell it to execute, and then copies back the results.

So, with that said, let’s re-phrase the question: Why did TensorFlow choose Python as the first well-supported language for expressing and controlling the training of models?

The answer to that is simple: Python is probably the most comfortable language for a large range of data scientists and machine learning experts that’s also that easy to integrate and have control a C++ backend, while also being general, widely-used both inside and outside of Google, and open source. Given that with the basic model of TensorFlow, the performance of Python isn’t that important, it was a natural fit. It’s also a huge plus that NumPy makes it easy to do pre-processing in Python — also with high performance — before feeding it in to TensorFlow for the truly CPU-heavy things.

There’s also a bunch of complexity in expressing the model that isn’t used when executing it — shape inference (e.g., if you do matmul(A, B), what is the shape of the resulting data?) and automatic gradient computation. It turns out to have been nice to be able to express those in Python, though I think in the long term they’ll probably move to the C++ backend to make adding other languages easier.

(The hope, of course, is to support other languages in the future for creating and expressing models. It’s already quite straightforward to run inference using several other languages — C++ works now, someone from Facebook contributed Go bindings that we’re reviewing now, etc.)


回答 1

TF不是用python编写的。它是用C ++编写的(并使用高性能的数字CUDA代码),您可以通过查看他们的github进行检查。因此,核心不是用python编写的,而是TF提供了许多其他语言(python,C ++,Java,Go)的接口

如果您来自数据分析领域,则可以像numpy(不是用python编写,但提供了Python的接口)那样考虑它,或者如果您是Web开发人员,则可以将其视为数据库(PostgreSQL,MySQL,可以从Java,Python,PHP调用)


由于许多 原因, Python前端(人们使用TF编写模型的语言)最受欢迎。在我看来,主要原因是历史原因:大多数ML用户已经在使用它(另一个流行的选择是R),因此,如果您不提供python的接口,那么您的库很可能注定会变得晦涩难懂。


但是用python编写并不意味着您的模型是用python执行的。相反,如果您以正确的方式编写模型,则在评估TF图期间绝不会执行Python(tf.py_func()除外,该存在于调试中,应在实际模型中避免使用,因为它是在Python方面)。

例如,这与numpy不同。例如,如果您这样做np.linalg.eig(np.matmul(A, np.transpose(A))(是eig(AA')),则该操作将以某种快速语言(C ++或fortran)计算转置,将其返回给python,将其与python一起从python中取出,并以某种快速语言计算一个乘法并将其返回给python,然后计算特征值并将其返回给python。因此,尽管有效地计算了诸如matmul和eig之类的昂贵操作,但您仍然需要通过将结果移回python并强制执行来浪费时间。TF不会这样做,一旦定义了图,张量就不会在python中,而是在C ++ / CUDA /其他地方流动。

TF is not written in python. It is written in C++ (and uses high-performant numerical libraries and CUDA code) and you can check this by looking at their github. So the core is written not in python but TF provide an interface to many other languages (python, C++, Java, Go)

If you come from a data analysis world, you can think about it like numpy (not written in python, but provides an interface to Python) or if you are a web-developer – think about it as a database (PostgreSQL, MySQL, which can be invoked from Java, Python, PHP)


Python frontend (the language in which people write models in TF) is the most popular due to many reasons. In my opinion the main reason is historical: majority of ML users already use it (another popular choice is R) so if you will not provide an interface to python, your library is most probably doomed to obscurity.


But being written in python does not mean that your model is executed in python. On the contrary, if you written your model in the right way Python is never executed during the evaluation of the TF graph (except of tf.py_func(), which exists for debugging and should be avoided in real model exactly because it is executed on Python’s side).

This is different from for example numpy. For example if you do np.linalg.eig(np.matmul(A, np.transpose(A)) (which is eig(AA')), the operation will compute transpose in some fast language (C++ or fortran), return it to python, take it from python together with A, and compute a multiplication in some fast language and return it to python, then compute eigenvalues and return it to python. So nonetheless expensive operations like matmul and eig are calculated efficiently, you still lose time by moving the results to python back and force. TF does not do it, once you defined the graph your tensors flow not in python but in C++/CUDA/something else.


回答 2

Python允许您使用C和C ++创建扩展模块,与本机代码接口,并且仍然获得Python给您的优势。

TensorFlow使用Python,是的,但是它也包含大量的C ++

这样就可以使用更简单的界面进行实验,从而减少了用Python进行的人工操作,并通过对C ++中最重要的部分进行编程来提高性能。

Python allows you to create extension modules using C and C++, interfacing with native code, and still getting the advantages that Python gives you.

TensorFlow uses Python, yes, but it also contains large amounts of C++.

This allows a simpler interface for experimentation with less human-thought overhead with Python, and add performance by programming the most important parts in C++.


回答 3

您可以从此处查看的最新比率显示TensorFlow C ++内部需要约50%的代码,而Python需要约40%的代码。

C ++和Python都是Google的官方语言,所以也难怪为什么会这样。如果我必须在存在C ++和Python的地方提供快速回归…

C ++在计算代数内部,Python用于其他所有方面,包括测试。知道今天的测试无处不在,难怪Python代码对TF做出了如此大的贡献。

The latest ratio you can check from here shows inside TensorFlow C++ takes ~50% of code, and Python takes ~40% of code.

Both C++ and Python are the official languages at Google so there is no wonder why this is so. If I would have to provide fast regression where C++ and Python are present…

C++ is inside the computational algebra, and Python is used for everything else including for the testing. Knowing how ubiquitous the testing is today it is no wonder why Python code contributes that much to TF.


使用if-return-return或if-else-return更有效吗?

问题:使用if-return-return或if-else-return更有效吗?

假设我有一个if带有的语句return。从效率的角度来看,我应该使用

if(A > B):
    return A+1
return A-1

要么

if(A > B):
    return A+1
else:
    return A-1

使用编译语言(C)或脚本化语言(Python)时,我应该选择一种还是另一种?

Suppose I have an if statement with a return. From the efficiency perspective, should I use

if(A > B):
    return A+1
return A-1

or

if(A > B):
    return A+1
else:
    return A-1

Should I prefer one or another when using a compiled language (C) or a scripted one (Python)?


回答 0

由于该return语句终止了当前函数的执行,因此两种形式是等效的(尽管第二种形式比第一种更具可读性)。

两种形式的效率都相当,如果if条件为假,则基础机器代码必须执行跳转。

请注意,Python支持一种语法,该语法仅允许您使用一种return情况:

return A+1 if A > B else A-1

Since the return statement terminates the execution of the current function, the two forms are equivalent (although the second one is arguably more readable than the first).

The efficiency of both forms is comparable, the underlying machine code has to perform a jump if the if condition is false anyway.

Note that Python supports a syntax that allows you to use only one return statement in your case:

return A+1 if A > B else A-1

回答 1

根据Chromium的风格指南:

返回后请勿使用其他:

# Bad
if (foo)
  return 1
else
  return 2

# Good
if (foo)
  return 1
return 2

return 1 if foo else 2

From Chromium’s style guide:

Don’t use else after return:

# Bad
if (foo)
  return 1
else
  return 2

# Good
if (foo)
  return 1
return 2

return 1 if foo else 2

回答 2

关于编码风格:

无论哪种语言,大多数编码标准都禁止从单个函数中使用多个返回语句,这是一种不好的做法。

(尽管我个人会说在某些情况下多个返回语句确实有意义:文本/数据协议解析器,具有大量错误处理的功能等)

所有这些行业编码标准的共识是,该表达式应写为:

int result;

if(A > B)
{
  result = A+1;
}
else
{
  result = A-1;
}
return result;

关于效率:

上面的示例和问题中的两个示例在效率方面都完全等效。在所有这些情况下,机器码都必须比较A> B,然后跳转到A + 1或A-1计算,然后将结果存储在CPU寄存器或堆栈中。

编辑:

资料来源:

  • MISRA-C:2004规则14.7,依次引用…:
  • IEC 61508-3。第3部分,表B.9。
  • IEC 61508-7。C.2.9。

Regarding coding style:

Most coding standards no matter language ban multiple return statements from a single function as bad practice.

(Although personally I would say there are several cases where multiple return statements do make sense: text/data protocol parsers, functions with extensive error handling etc)

The consensus from all those industry coding standards is that the expression should be written as:

int result;

if(A > B)
{
  result = A+1;
}
else
{
  result = A-1;
}
return result;

Regarding efficiency:

The above example and the two examples in the question are all completely equivalent in terms of efficiency. The machine code in all these cases have to compare A > B, then branch to either the A+1 or the A-1 calculation, then store the result of that in a CPU register or on the stack.

EDIT :

Sources:

  • MISRA-C:2004 rule 14.7, which in turn cites…:
  • IEC 61508-3. Part 3, table B.9.
  • IEC 61508-7. C.2.9.

回答 3

对于任何明智的编译器,您都应该观察到没有区别。它们应该被编译为相同的机器代码,因为它们是等效的。

With any sensible compiler, you should observe no difference; they should be compiled to identical machine code as they’re equivalent.


回答 4

因为口译员不在乎,所以这是一个风格(或偏好)问题。就我个人而言,我尽量不要对以函数基础以外的缩进级别返回值的函数做最终声明。示例1中的else会(即使只是稍微)掩盖了函数的结束位置。

根据偏好,我使用:

return A+1 if (A > B) else A-1

因为它遵循了将单个return语句作为函数中的最后一条语句的良好约定(如已提到的那样)以及避免命令式中间结果的良好的函数编程范例。

对于更复杂的功能,我更喜欢将功能分解为多个子功能,以避免可能的过早返回。否则,我将恢复使用称为rval的命令式样式变量。我尽量不要使用多个return语句,除非该函数是微不足道的,或者在结束之前的return语句是由于错误导致的。过早返回会突出显示您无法继续前进的事实。对于旨在分解为多个子功能的复杂功能,我尝试将它们编码为case语句(例如,由dict驱动)。

一些海报提到了运行速度。对于我来说,运行时的速度是次要的,因为如果您需要执行速度,那么Python并不是最好的语言。我将Python用作对我很重要的编码效率(即编写无错误代码)。

This is a question of style (or preference) since the interpreter does not care. Personally I would try not to make the final statement of a function which returns a value at an indent level other than the function base. The else in example 1 obscures, if only slightly, where the end of the function is.

By preference I use:

return A+1 if (A > B) else A-1

As it obeys both the good convention of having a single return statement as the last statement in the function (as already mentioned) and the good functional programming paradigm of avoiding imperative style intermediate results.

For more complex functions I prefer to break the function into multiple sub-functions to avoid premature returns if possible. Otherwise I revert to using an imperative style variable called rval. I try not to use multiple return statements unless the function is trivial or the return statement before the end is as a result of an error. Returning prematurely highlights the fact that you cannot go on. For complex functions that are designed to branch off into multiple subfunctions I try to code them as case statements (driven by a dict for instance).

Some posters have mentioned speed of operation. Speed of Run-time is secondary for me since if you need speed of execution Python is not the best language to use. I use Python as its the efficiency of coding (i.e. writing error free code) that matters to me.


回答 5

我个人else尽可能避免阻塞。参见反假宣传运动

另外,他们不收取“额外”费用,你知道:p

“简单胜于复杂”和“可读性为王”

delta = 1 if (A > B) else -1
return A + delta

I personally avoid else blocks when possible. See the Anti-if Campaign

Also, they don’t charge ‘extra’ for the line, you know :p

“Simple is better than complex” & “Readability is king”

delta = 1 if (A > B) else -1
return A + delta

回答 6

版本A更简单,这就是我要使用它的原因。

而且,如果您打开Java中的所有编译器警告,您将在第二个版本上收到警告,因为它是不必要的,并且增加了代码复杂度。

Version A is simpler and that’s why I would use it.

And if you turn on all compiler warnings in Java you will get a warning on the second Version because it is unnecesarry and turns up code complexity.


回答 7

我知道这个问题被标记为python,但是它提到了动态语言,因此我想我应该提到在ruby中if语句实际上具有一个返回类型,因此您可以执行以下操作

def foo
  rv = if (A > B)
         A+1
       else
         A-1
       end
  return rv 
end

或者因为它也有隐式的回报

def foo 
  if (A>B)
    A+1
  else 
    A-1
  end
end

解决了没有很好的多次收益的样式问题。

I know the question is tagged python, but it mentions dynamic languages so thought I should mention that in ruby the if statement actually has a return type so you can do something like

def foo
  rv = if (A > B)
         A+1
       else
         A-1
       end
  return rv 
end

Or because it also has implicit return simply

def foo 
  if (A>B)
    A+1
  else 
    A-1
  end
end

which gets around the style issue of not having multiple returns quite nicely.


寻求澄清有关弱类型语言的明显矛盾

问题:寻求澄清有关弱类型语言的明显矛盾

我想我了解强类型,但是每次我寻找弱类型的示例时,我最终都会找到简单地自动强制转换类型的编程语言示例。

例如,在这篇名为“ 打字:强vs.弱”,“静态vs.动态 ”的文章中,Python是强类型的,因为如果尝试执行以下操作,则会得到异常:

Python

1 + "1"
Traceback (most recent call last):
File "", line 1, in ? 
TypeError: unsupported operand type(s) for +: 'int' and 'str'

但是,在Java和C#中这种事情是可能的,因此我们不认为它们只是弱类型的。

爪哇

  int a = 10;
  String b = "b";
  String result = a + b;
  System.out.println(result);

C#

int a = 10;
string b = "b";
string c = a + b;
Console.WriteLine(c);

在另一篇名为弱类型语言的文章中,作者说Perl弱类型仅仅是因为我可以将字符串连接成数字,反之亦然,而无需任何显式转换。

佩尔

$a=10;
$b="a";
$c=$a.$b;
print $c; #10a

因此,同一示例使Perl的类型较弱,但Java和C#?的类型却没有。

真是的

作者似乎暗示一种阻止对不同类型的值执行某些操作的语言是强类型的,而相反的意思是弱类型。

因此,在某些时候,我感到被提示相信,如果一种语言在类型之间提供大量自动转换或强制转换(例如perl),最终可能会被认为是弱类型,而其他仅提供少量转换的语言可能最终会被视为弱类型。被认为是强类型的。

但是,我倾向于相信,在这种相互交流中我一定是错的,我只是不知道为什么或如何解释它。

因此,我的问题是:

  • 语言真正弱键入到底意味着什么?
  • 您能否提及与该语言完成的自动转换/自动强制无关的弱类型的任何好例子?
  • 语言可以同时弱输入和强输入吗?

I think I understand strong typing, but every time I look for examples for what is weak typing I end up finding examples of programming languages that simply coerce/convert types automatically.

For instance, in this article named Typing: Strong vs. Weak, Static vs. Dynamic says that Python is strongly typed because you get an exception if you try to:

Python

1 + "1"
Traceback (most recent call last):
File "", line 1, in ? 
TypeError: unsupported operand type(s) for +: 'int' and 'str'

However, such thing is possible in Java and in C#, and we do not consider them weakly typed just for that.

Java

  int a = 10;
  String b = "b";
  String result = a + b;
  System.out.println(result);

C#

int a = 10;
string b = "b";
string c = a + b;
Console.WriteLine(c);

In this another article named Weakly Type Languages the author says that Perl is weakly typed simply because I can concatenate a string to a number and viceversa without any explicit conversion.

Perl

$a=10;
$b="a";
$c=$a.$b;
print $c; #10a

So the same example makes Perl weakly typed, but not Java and C#?.

Gee, this is confusing

The authors seem to imply that a language that prevents the application of certain operations on values of different types is strongly typed and the contrary means weakly typed.

Therefore, at some point I have felt prompted to believe that if a language provides a lot of automatic conversions or coercion between types (as perl) may end up being considered weakly typed, whereas other languages that provide only a few conversions may end up being considered strongly typed.

I am inclined to believe, though, that I must be wrong in this interepretation, I just do not know why or how to explain it.

So, my questions are:

  • What does it really mean for a language to be truly weakly typed?
  • Could you mention any good examples of weakly typing that are not related to automatic conversion/automatic coercion done by the language?
  • Can a language be weakly typed and strongly typed at the same time?

回答 0

更新:这个问题是我在2012年10月15日发布的博客的主题。感谢您提出的伟大问题!


语言“弱类型化”的真正含义是什么?

它的意思是“这种语言使用的类型系统令人讨厌”。相比之下,“强类型”语言是具有令人愉悦的类型系统的语言。

这些术语本质上是没有意义的,您应该避免使用它们。维基百科列出了“强类型”的十一种不同含义,其中有几种是矛盾的。这表明在涉及术语“强类型”或“弱类型”的任何对话中,造成混乱的可能性很高。

您真正可以肯定地说的是,正在讨论的“强类型”语言在类型系统上有一些其他限制,无论是在运行时还是编译时,都缺乏在讨论中的“弱类型”语言。没有进一步的上下文,就无法确定该限制是什么。

不应使用“强类型”和“弱类型”,而应详细描述您所指的类型安全。例如,C#在大多数情况下静态类型的语言,类型安全的语言和内存安全的语言。C#允许违反所有三种“强”类型的输入形式。强制转换运算符违反静态类型;它对编译器说:“我比您更了解此表达式的运行时类型”。如果开发人员错误,则运行时将抛出异常以保护类型安全。如果开发人员希望破坏类型安全性或存储安全性,则可以通过制作“不安全”块来关闭类型安全性系统来做到这一点。在不安全的块中,您可以使用指针魔术来将int视为浮点型(违反类型安全性)或写入您不拥有的内存。(破坏内存安全。)

C#施加了在编译时和运行时都进行检查的类型限制,因此与进行较少的编译时检查或较少的运行时检查的语言相比,C#使其成为“强类型”语言。C#还允许您在特殊情况下绕这些限制进行最终运行,与不允许您进行此类最终运行的语言相比,它成为“弱类型”语言。

到底是什么 很难说。这取决于说话者的观点及其对各种语言功能的态度。

UPDATE: This question was the subject of my blog on the 15th of October, 2012. Thanks for the great question!


What does it really mean for a language to be “weakly typed”?

It means “this language uses a type system that I find distasteful”. A “strongly typed” language by contrast is a language with a type system that I find pleasant.

The terms are essentially meaningless and you should avoid them. Wikipedia lists eleven different meanings for “strongly typed”, several of which are contradictory. This indicates that the odds of confusion being created are high in any conversation involving the term “strongly typed” or “weakly typed”.

All that you can really say with any certainty is that a “strongly typed” language under discussion has some additional restriction in the type system, either at runtime or compile time, that a “weakly typed” language under discussion lacks. What that restriction might be cannot be determined without further context.

Instead of using “strongly typed” and “weakly typed”, you should describe in detail what kind of type safety you mean. For example, C# is a statically typed language and a type safe language and a memory safe language, for the most part. C# allows all three of those forms of “strong” typing to be violated. The cast operator violates static typing; it says to the compiler “I know more about the runtime type of this expression than you do”. If the developer is wrong, then the runtime will throw an exception in order to protect type safety. If the developer wishes to break type safety or memory safety, they can do so by turning off the type safety system by making an “unsafe” block. In an unsafe block you can use pointer magic to treat an int as a float (violating type safety) or to write to memory you do not own. (Violating memory safety.)

C# imposes type restrictions that are checked at both compile-time and at runtime, thereby making it a “strongly typed” language compared to languages that do less compile-time checking or less runtime checking. C# also allows you to in special circumstances do an end-run around those restrictions, making it a “weakly typed” language compared with languages which do not allow you to do such an end-run.

Which is it really? It is impossible to say; it depends on the point of view of the speaker and their attitude towards the various language features.


回答 1

正如其他人指出的那样,术语“强类型”和“弱类型”具有许多不同的含义,因此您的问题没有一个答案。但是,由于您在问题中特别提到了Perl,因此让我尝试解释Perl弱键入的含义。

关键是,在Perl中,没有“整数变量”,“浮点变量”,“字符串变量”或“布尔变量”之类的东西。实际上,据用户所知(通常),甚至没有整数,浮点数,字符串或布尔:您所拥有的都是“标量”,它们同时是所有这些东西。因此,您可以例如编写:

$foo = "123" + "456";           # $foo = 579
$bar = substr($foo, 2, 1);      # $bar = 9
$bar .= " lives";               # $bar = "9 lives"
$foo -= $bar;                   # $foo = 579 - 9 = 570

当然,正如您正确指出的那样,所有这些都可以看作是强制类型。但是关键是,在Perl中,类型始终是强制的。实际上,用户很难说出变量的内部“类型”是什么:在我上面的示例的第2行,询问变量的值$bar是字符串"9"还是数字9几乎没有意义,因为就Perl而言,它们是同一回事。实际上,Perl标量甚至有可能在内部同时具有字符串和数字值,例如$foo上面第2行之后的情况。

不利的一面是,由于Perl变量是无类型的(或者,不向用户公开其内部类型),因此不能重载运算符以对不同类型的参数执行不同的操作。您不能只说“此运算符将对数字执行X,对字符串执行Y”,因为该运算符无法(不会)告诉其参数是哪种类型的值。

因此,例如,Perl同时具有并且需要数字加法运算符(+)和字符串连接运算符(.):如上所述,添加字符串("1" + "2" == "3")或连接数字(1 . 2 == 12)很好。同样,数字比较操作符==!=<><=>=<=>比较它们的参数的数值,而字符串比较操作符eqneltgtlegecmp字典顺序比较它们为字符串。所以2 < 10,但是2 gt 10(但是"02" lt 10,虽然"02" == 2)。(请注意,某些其他语言(例如JavaScript)会尝试容纳类似Perl的弱类型做运算符重载。这通常会导致丑陋,例如失去与+。)的关联性。

(美中不足的是,由于历史原因,Perl 5确实有一些极端情况,例如按位逻辑运算符,其行为取决于其参数的内部表示。通常认为这是令人讨厌的设计缺陷,因为内部表述可能会由于令人惊讶的原因而发生变化,因此仅预测那些操作员在给定情况下的操作可能很棘手。)

综上所述,可以说Perl 确实具有强类型。它们不是您可能期望的那种类型。具体来说,除了上面讨论的“标量”类型外,Perl还具有两种结构化类型:“数组”和“哈希”。这些是非常从标量不同,到了那里的Perl变量具有不同的点印记,指示它们的类型($用于标量,@数组,%对于散列)1。有这些类型之间的强制规则,这样你就可以写例如%foo = @bar,但其中不少是相当有损耗:例如,$foo = @bar分配长度的数组 @bar$foo,而不是其内容。(此外,还有其他一些奇怪的类型,例如typeglob和I / O句柄,您通常不会看到它们是公开的。)

同样,在这种出色的设计中,有一个小缺点是引用类型的存在,它们是一种特殊的标量(可以使用ref运算符将其与普通标量区分开)。可以将引用用作普通标量,但是它们的字符串/数字值并不是特别有用,并且如果您使用普通标量操作对其进行修改,它们往往会失去其特殊的引用性。同样,任何Perl变量2都可以作为范例。通常的意见是,如果您发现自己在Perl中检查了对象的类,则说明您做错了什么。bless编入一个类,将其变成该类的对象。Perl中的OO类系统在某种程度上与上述原始类型(或无类型性)系统正交,尽管它在遵循鸭子类型的意义上也是“弱”的


1实际上,印记表示被访问的值的类型,以使得例如在阵列中的第一标@foo$foo[0]。有关更多详细信息,请参见perlfaq4

2(通常)通过引用访问Perl中的对象,但实际上得到的bless是引用所指向的(可能是匿名的)变量。但是,祝福实际上是变量的属性,而不是变量的值,因此,例如,将实际的祝福变量分配给另一个变量,只会给您一个浅浅的,没有祝福的副本。有关更多详细信息,请参见perlobj

As others have noted, the terms “strongly typed” and “weakly typed” have so many different meanings that there’s no single answer to your question. However, since you specifically mentioned Perl in your question, let me try to explain in what sense Perl is weakly typed.

The point is that, in Perl, there is no such thing as an “integer variable”, a “float variable”, a “string variable” or a “boolean variable”. In fact, as far as the user can (usually) tell, there aren’t even integer, float, string or boolean values: all you have are “scalars”, which are all of these things at the same time. So you can, for example, write:

$foo = "123" + "456";           # $foo = 579
$bar = substr($foo, 2, 1);      # $bar = 9
$bar .= " lives";               # $bar = "9 lives"
$foo -= $bar;                   # $foo = 579 - 9 = 570

Of course, as you correctly note, all of this can be seen as just type coercion. But the point is that, in Perl, types are always coerced. In fact, it’s quite hard for a user to tell what the internal “type” of a variable might be: at line 2 in my example above, asking whether the value of $bar is the string "9" or the number 9 is pretty much meaningless, since, as far as Perl is concerned, those are the same thing. Indeed, it’s even possible for a Perl scalar to internally have both a string and a numeric value at the same time, as is e.g. the case for $foo after line 2 above.

The flip side of all this is that, since Perl variables are untyped (or, rather, don’t expose their internal type to the user), operators cannot be overloaded to do different things for different types of arguments; you can’t just say “this operator will do X for numbers and Y for strings”, because the operator can’t (won’t) tell which kind of values its arguments are.

Thus, for example, Perl has and needs both a numeric addition operator (+) and a string concatenation operator (.): as you saw above, it’s perfectly fine to add strings ("1" + "2" == "3") or to concatenate numbers (1 . 2 == 12). Similarly, the numeric comparison operators ==, !=, <, >, <=, >= and <=> compare the numeric values of their arguments, while the string comparison operators eq, ne, lt, gt, le, ge and cmp compare them lexicographically as strings. So 2 < 10, but 2 gt 10 (but "02" lt 10, while "02" == 2). (Mind you, certain other languages, like JavaScript, try to accommodate Perl-like weak typing while also doing operator overloading. This often leads to ugliness, like the loss of associativity for +.)

(The fly in the ointment here is that, for historical reasons, Perl 5 does have a few corner cases, like the bitwise logical operators, whose behavior depends on the internal representation of their arguments. Those are generally considered an annoying design flaw, since the internal representation can change for surprising reasons, and so predicting just what those operators do in a given situation can be tricky.)

All that said, one could argue that Perl does have strong types; they’re just not the kind of types you might expect. Specifically, in addition to the “scalar” type discussed above, Perl also has two structured types: “array” and “hash”. Those are very distinct from scalars, to the point where Perl variables have different sigils indicating their type ($ for scalars, @ for arrays, % for hashes)1. There are coercion rules between these types, so you can write e.g. %foo = @bar, but many of them are quite lossy: for example, $foo = @bar assigns the length of the array @bar to $foo, not its contents. (Also, there are a few other strange types, like typeglobs and I/O handles, that you don’t often see exposed.)

Also, a slight chink in this nice design is the existence of reference types, which are a special kind of scalars (and which can be distinguished from normal scalars, using the ref operator). It’s possible to use references as normal scalars, but their string/numeric values are not particularly useful, and they tend to lose their special reference-ness if you modify them using normal scalar operations. Also, any Perl variable2 can be blessed to a class, turning it into an object of that class; the OO class system in Perl is somewhat orthogonal to the primitive type (or typelessness) system described above, although it’s also “weak” in the sense of following the duck typing paradigm. The general opinion is that, if you find yourself checking the class of an object in Perl, you’re doing something wrong.


1 Actually, the sigil denotes the type of the value being accessed, so that e.g. the first scalar in the array @foo is denoted $foo[0]. See perlfaq4 for more details.

2 Objects in Perl are (normally) accessed through references to them, but what actually gets blessed is the (possibly anonymous) variable the reference points to. However, the blessing is indeed a property of the variable, not of its value, so e.g. that assigning the actual blessed variable to another one just gives you a shallow, unblessed copy of it. See perlobj for more details.


回答 2

除了Eric所说的以外,请考虑以下C代码:

void f(void* x);

f(42);
f("hello");

与诸如Python,C#,Java或其他语言之类的语言相比,上面的类型是弱类型的,因为我们会丢失类型信息。Eric正确指出,在C#中,我们可以通过强制转换来绕过编译器,有效地告诉它“我比您更了解此变量的类型”。

但是即使那样,运行时仍会检查类型!如果强制转换无效,则运行时系统将对其进行捕获并引发异常。

使用类型擦除不会发生这种情况–类型信息会被丢弃。强制转换void*为C可以做到这一点。在这方面,以上内容与C#方法声明(例如)从根本上有所不同void f(Object x)

(从技术上讲,C#还允许通过不安全的代码或编组来擦除类型。)

是尽可能弱的类型。其他的一切只是一个静态还是动态类型检查,即时间的事一个类型被选中。

In addition to what Eric has said, consider the following C code:

void f(void* x);

f(42);
f("hello");

In contrast to languages such as Python, C#, Java or whatnot, the above is weakly typed because we lose type information. Eric correctly pointed out that in C# we can circumvent the compiler by casting, effectively telling it “I know more about the type of this variable than you”.

But even then, the runtime will still check the type! If the cast is invalid, the runtime system will catch it and throw an exception.

With type erasure, this doesn’t happen – type information is thrown away. A cast to void* in C does exactly that. In this regard, the above is fundamentally different from a C# method declaration such as void f(Object x).

(Technically, C# also allows type erasure through unsafe code or marshalling.)

This is as weakly typed as it gets. Everything else is just a matter of static vs. dynamic type checking, i.e. of the time when a type is checked.


回答 3

Wikipedia的“强类型”文章就是一个很好的例子:

通常,强类型意味着编程语言对允许发生的混合进行了严格的限制。

弱打字

a = 2
b = "2"

concatenate(a, b) # returns "22"
add(a, b) # returns 4

强类型

a = 2
b = "2"

concatenate(a, b) # Type Error
add(a, b) # Type Error
concatenate(str(a), b) #Returns "22"
add(a, int(b)) # Returns 4

请注意,弱类型的语言可以混合不同类型而不会出错。强类型语言要求输入类型为预期类型。在强类型语言中,可以转换类型(str(a)将整数转换为字符串)或强制转换(int(b))。

这一切都取决于键入的解释。

A perfect example comes from the wikipedia article of Strong Typing:

Generally strong typing implies that the programming language places severe restrictions on the intermixing that is permitted to occur.

Weak Typing

a = 2
b = "2"

concatenate(a, b) # returns "22"
add(a, b) # returns 4

Strong Typing

a = 2
b = "2"

concatenate(a, b) # Type Error
add(a, b) # Type Error
concatenate(str(a), b) #Returns "22"
add(a, int(b)) # Returns 4

Notice that a weak typing language can intermix different types without errors. A strong type language requires the input types to be the expected types. In a strong type language a type can be converted (str(a) converts an integer to a string) or cast (int(b)).

This all depends on the interpretation of typing.


回答 4

我想通过自己对这个问题的研究来为讨论做出贡献,正如其他人评论并做出贡献一样,我一直在阅读他们的答案并遵循他们的参考文献,并且发现了有趣的信息。如建议的那样,在程序员论坛中可能会更好地讨论其中的大多数内容,因为它似乎是理论上的而非实际的。

从理论的角度来看,我认为Luca Cardelli和Peter Wegner撰写的关于理解类型,数据抽象和多态性的文章是我所读过的最好的论据之一。

一种类型可以看作是一套衣服(或盔甲),可以保护基础的无类型表示形式免受任意使用或非预期使用。它提供了一个保护性遮盖物,该遮盖物隐藏了底层表示并限制了对象与其他对象交互的方式。在无类型的系统中,无类型的对象是裸露 的,其基础表示形式公开给所有人看。违反字体系统需要脱下防护服并直接在裸露的衣服上操作。

该说法似乎表明,弱类型输入将使我们能够访问类型的内部结构并像对待其他类型(另一种类型)一样对其进行操作。也许我们可以用不安全的代码(由Eric提及)或由Konrad提及的用c类型擦除的指针来做。

文章继续…

所有表达式类型一致的语言称为强类型语言。如果语言是强类型的,则其编译器可以保证所接受的程序将在没有类型错误的情况下执行。通常,我们应该努力实现强类型化,并在可能的情况下采用静态类型。请注意,每种静态类型的语言都是强类型的,但相反不一定是正确的。

因此,强类型表示没有类型错误,我只能假设弱类型意味着相反:可能存在类型错误。在运行时还是编译时?在这里似乎无关紧要。

有趣的是,按照该定义,具有强大类型强制性的语言(如Perl)将被视为强类型化的,因​​为系统不会失败,但是它通过将类型强制为适当的和定义良好的对等来处理类型。

另一方面,我是否可以说ClassCastExceptionArrayStoreException(在Java中)和InvalidCastExceptionArrayTypeMismatchException(在C#中)的允许表示至少在编译时处于弱类型的水平?埃里克的答案似乎同意这一点。

在此问题的答案之一提供的参考文献之一中提供的第二篇名为类型化编程的文章中,Luca Cardelli深入研究了类型冲突的概念:

大多数系统编程语言都允许任意类型的违反,有些是不加区分的,有些仅在程序的受限部分中。涉及类型冲突的操作称为不健全。类型违规分为几类[我们可以提到]:

基本值强制:包括整数,布尔值,字符,集合等之间的转换。这里不需要类型冲突,因为可以提供内置接口来以类型健全的方式执行强制。

这样,像操作员提供的那样的类型强制可以被认为是类型冲突,但是除非它们破坏了类型系统的一致性,否则我们可以说它们不会导致弱类型系统。

基于此,Python,Perl,Java或C#都不是弱类型。

Cardelli提到了两个类型错误,我很好地考虑了真正弱类型的情况:

地址算术。如有必要,应该有一个内置的(不健全的)接口,提供对地址和类型转换的适当操作。各种情况都涉及到堆的指针(对于重定位收集器非常危险),指向堆栈的指针,指向静态区域的指针以及指向其他地址空间的指针。有时,数组索引可以代替地址算法。 内存映射。这涉及将内存区域视为非结构化数组,尽管它包含结构化数据。这是内存分配器和收集器的典型特征。

诸如C(由Konrad提及)或.Net中不安全代码(由Eric提及)之类的语言中可能发生的这类事情实际上暗示着弱键入。

我相信到目前为止,最好的答案是埃里克(Eric),因为这个概念的定义是非常理论性的,当涉及到特定语言时,对所有这些概念的解释可能会得出不同的结论。

I would like to contribute to the discussion with my own research on the subject, as others comment and contribute I have been reading their answers and following their references and I have found interesting information. As suggested, it is probable that most of this would be better discussed in the Programmers forum, since it appears to be more theoretical than practical.

From a theoretical standpoint, I think the article by Luca Cardelli and Peter Wegner named On Understanding Types, Data Abstraction and Polymorphism has one of the best arguments I have read.

A type may be viewed as a set of clothes (or a suit of armor) that protects an underlying untyped representation from arbitrary or unintended use. It provides a protective covering that hides the underlying representation and constrains the way objects may interact with other objects. In an untyped system untyped objects are naked in that the underlying representation is exposed for all to see. Violating the type system involves removing the protective set of clothing and operating directly on the naked representation.

This statement seems to suggest that weakly typing would let us access the inner structure of a type and manipulate it as if it was something else (another type). Perhaps what we could do with unsafe code (mentioned by Eric) or with c type-erased pointers mentioned by Konrad.

The article continues…

Languages in which all expressions are type-consistent are called strongly typed languages. If a language is strongly typed its compiler can guarantee that the programs it accepts will execute without type errors. In general, we should strive for strong typing, and adopt static typing whenever possible. Note that every statically typed language is strongly typed but the converse is not necessarily true.

As such, strong typing means the absence of type errors, I can only assume that weak typing means the contrary: the likely presence of type errors. At runtime or compile time? Seems irrelevant here.

Funny thing, as per this definition, a language with powerful type coercions like Perl would be considered strongly typed, because the system is not failing, but it is dealing with the types by coercing them into appropriate and well defined equivalences.

On the other hand, could I say than the allowance of ClassCastException and ArrayStoreException (in Java) and InvalidCastException, ArrayTypeMismatchException (in C#) would indicate a level of weakly typing, at least at compile time? Eric’s answer seems to agree with this.

In a second article named Typeful Programming provided in one of the references provided in one of the answers in this question, Luca Cardelli delves into the concept of type violations:

Most system programming languages allow arbitrary type violations, some indiscriminately, some only in restricted parts of a program. Operations that involve type violations are called unsound. Type violations fall in several classes [among which we can mention]:

Basic-value coercions: These include conversions between integers, booleans, characters, sets, etc. There is no need for type violations here, because built-in interfaces can be provided to carry out the coercions in a type-sound way.

As such, type coercions like those provided by operators could be considered type violations, but unless they break the consistency of the type system, we might say that they do not lead to a weakly typed system.

Based on this neither Python, Perl, Java or C# are weakly typed.

Cardelli mentions two type vilations that I very well consider cases of truly weak typing:

Address arithmetic. If necessary, there should be a built-in (unsound) interface, providing the adequate operations on addresses and type conversions. Various situations involve pointers into the heap (very dangerous with relocating collectors), pointers to the stack, pointers to static areas, and pointers into other address spaces. Sometimes array indexing can replace address arithmetic. Memory mapping. This involves looking at an area of memory as an unstructured array, although it contains structured data. This is typical of memory allocators and collectors.

This kind of things possible in languages like C (mentioned by Konrad) or through unsafe code in .Net (mentioned by Eric) would truly imply weakly typing.

I believe the best answer so far is Eric’s, because the definition of this concepts is very theoretical, and when it comes to a particular language, the interpretations of all these concepts may lead to different debatable conclusions.


回答 5

弱类型的确确实意味着可以隐式强制转换很大比例的类型,试图猜测编码器的意图。

强类型意味着没有强制类型,或者至少没有强制类型。

静态类型意味着您的变量类型在编译时确定。

最近,许多人将“明显键入”与“强烈键入”混淆。“清单式输入”是指您明确声明变量的类型。

Python通常是强类型的,尽管您可以在布尔上下文中使用几乎所有内容,布尔值可以在整数上下文中使用,并且您可以在浮点上下文中使用整数。它没有明显的类型,因为您不需要声明您的类型(Cython除外,尽管它很有趣,但它并不完全是python)。它也不是静态类型的。

C和C ++是明显类型化,静态类型化和某种程度强类型化的,因​​为您声明类型,类型是在编译时确定的,并且可以混合使用整数和指针,整数和双精度,甚至将指向一种类型的指针转​​换为指向另一种类型的指针。

Haskell是一个有趣的示例,因为它不是显式键入的,而是静态和强类型的。

Weak typing does indeed mean that a high percentage of types can be implicitly coerced, attempting to guess what the coder intended.

Strong typing means that types are not coerced, or at least coerced less.

Static typing means your variables’ types are determined at compile time.

Many people have recently been confusing “manifestly typed” with “strongly typed”. “Manifestly typed” means that you declare your variables’ types explicitly.

Python is mostly strongly typed, though you can use almost anything in a boolean context, and booleans can be used in an integer context, and you can use an integer in a float context. It is not manifestly typed, because you don’t need to declare your types (except for Cython, which isn’t entirely python, albeit interesting). It is also not statically typed.

C and C++ are manifestly typed, statically typed, and somewhat strongly typed, because you declare your types, types are determined at compile time, and you can mix integers and pointers, or integers and doubles, or even cast a pointer to one type into a pointer to another type.

Haskell is an interesting example, because it is not manifestly typed, but it’s also statically and strongly typed.


回答 6

强<=>弱类型不仅是关于一种数据类型的语言将语言自动将多少值强制转换为另一种数据的连续性,还涉及到对实际的强弱程度。在Python和Java中,大多数情况下在C#中,值的类型设置为固定。在Perl中,不是那么多-实际上只有少数几个不同的值类型可以存储在变量中。

让我们一一打开案例。


Python

在Python示例中1 + "1"+运算符调用__add__for类型int,将字符串"1"作为参数,但是这会导致NotImplemented:

>>> (1).__add__('1')
NotImplemented

接下来,解释器尝试__radd__str的:

>>> '1'.__radd__(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__radd__'

由于失败,+操作员将结果与失败TypeError: unsupported operand type(s) for +: 'int' and 'str'。这样,该异常并不能说明强类型,但是该运算符+ 不会自动将其参数强制转换为同一类型,这说明了Python不是连续体中最弱类型的语言。

另一方面,在Python 'a' * 5 实现了:

>>> 'a' * 5
'aaaaa'

那是,

>>> 'a'.__mul__(5)
'aaaaa'

操作不同的事实需要强类型化-但是,*在乘法之前将值强制转换为数字的相反情况并不一定会使值弱类型化。


爪哇

Java示例String result = "1" + 1;之所以起作用,仅是因为为方便起见,运算符+被字符串重载。Java +运算符使用创建一个替换序列StringBuilder(请参阅参考资料):

String result = a + b;
// becomes something like
String result = new StringBuilder().append(a).append(b).toString()

这是一个非常静态的键入的示例,没有实际的强制性- StringBuilder有一种append(Object)专门用于此的方法。该文档说:

追加Object参数的字符串表示形式。

总体效果就好像参数已由方法转换为String.valueOf(Object)字符串,然后将该字符串的字符附加到此字符序列。

String.valueOf

返回Object参数的字符串表示形式。[返回]如果参数为null,则字符串等于"null"; 否则,obj.toString()返回的值。

因此,这种情况绝对不会被语言强制-将所有问题都委派给对象本身。


C#

根据此处Jon Skeet答案,该类+甚至都不会重载运算符string-类似于Java,这归功于静态和强类型化,这是编译器生成的便利。


佩尔

正如perldata解释的那样,

Perl具有三种内置数据类型:标量,标量数组和标量的关联数组,称为“哈希”。标量是单个字符串(任何大小,仅受可用内存限制),数字或对某物的引用(将在perlref中进行讨论)。普通数组是按数字索引的标量的有序列表,从0开始。哈希是通过其关联的字符串键索引的无序标量值的集合。

但是,Perl没有用于数字,布尔值,字符串,空值,undefineds,对其他对象的引用等的单独数据类型-它仅具有一种用于所有这些的类型,即标量类型。0是“ 0”的标量值。设置为字符串的标量变量实际上可以变成数字,并且从此开始,如果在数字上下文中访问则其行为就不同于“只是字符串”。标量可以在Perl中容纳任何内容,它与系统中存在的对象一样多。而在Python中,名称仅指对象,而在Perl中,名称中的标量值是可变对象。此外,基于对象的类型系统还基于此:perl中只有3种数据类型-标量,列表和哈希。Perl中的用户定义对象是对包的引用(指向之前3个中的任何一个的指针)bless-您可以获取任何此类值,并在需要的任何时候将其祝福给任何类。

Perl甚至允许您一时兴起地更改值的类-在Python中这是不可能的,在Python中创建某些类的值时,您需要显式构造具有该类object.__new__或类似值的该类的值。在Python中,创建后实际上不能更改对象的本质,在Perl中,您可以做很多事情:

package Foo;
package Bar;

my $val = 42;
# $val is now a scalar value set from double
bless \$val, Foo;
# all references to $val now belong to class Foo
my $obj = \$val;
# now $obj refers to the SV stored in $val
# thus this prints: Foo=SCALAR(0x1c7d8c8)
print \$val, "\n"; 
# all references to $val now belong to class Bar
bless \$val, Bar;
# thus this prints Bar=SCALAR(0x1c7d8c8)
print \$val, "\n";
# we change the value stored in $val from number to a string
$val = 'abc';
# yet still the SV is blessed: Bar=SCALAR(0x1c7d8c8)
print \$val, "\n";
# and on the course, the $obj now refers to a "Bar" even though
# at the time of copying it did refer to a "Foo".
print $obj, "\n";

因此,类型标识弱绑定到变量,并且可以通过任何引用即时更改它。实际上,如果您这样做

my $another = $val;

\$another没有类标识,即使仍然\$val会提供祝福的引用。


TL; DR

对于Perl而言,弱类型不仅仅是自动强制,还有很多更多的是,值的类型本身并没有固定不变,这与Python是动态但非常强类型的语言不同。这Python给人TypeError1 + "1"是一种迹象表明,语言是强类型,即使做一些有用的一个相反,如Java或C#不排除他们是强类型语言。

The strong <=> weak typing is not only about the continuum on how much or how little of the values are coerced automatically by the language for one datatype to another, but how strongly or weakly the actual values are typed. In Python and Java, and mostly in C#, the values have their types set in stone. In Perl, not so much – there are really only a handful of different valuetypes to store in a variable.

Let’s open the cases one by one.


Python

In Python example 1 + "1", + operator calls the __add__ for type int giving it the string "1" as an argument – however, this results in NotImplemented:

>>> (1).__add__('1')
NotImplemented

Next, the interpreter tries the __radd__ of str:

>>> '1'.__radd__(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__radd__'

As it fails, the + operator fails with the the result TypeError: unsupported operand type(s) for +: 'int' and 'str'. As such, the exception does not say much about strong typing, but the fact that the operator + does not coerce its arguments automatically to the same type, is a pointer to the fact that Python is not the most weakly typed language in the continuum.

On the other hand, in Python 'a' * 5 is implemented:

>>> 'a' * 5
'aaaaa'

That is,

>>> 'a'.__mul__(5)
'aaaaa'

The fact that the operation is different requires some strong typing – however the opposite of * coercing the values to numbers before multiplying still would not necessarily make the values weakly typed.


Java

The Java example, String result = "1" + 1; works only because as a fact of convenience, the operator + is overloaded for strings. The Java + operator replaces the sequence with creating a StringBuilder (see this):

String result = a + b;
// becomes something like
String result = new StringBuilder().append(a).append(b).toString()

This is rather an example of very static typing, without no actual coercion – StringBuilder has a method append(Object) that is specifically used here. The documentation says the following:

Appends the string representation of the Object argument.

The overall effect is exactly as if the argument were converted to a string by the method String.valueOf(Object), and the characters of that string were then appended to this character sequence.

Where String.valueOf then

Returns the string representation of the Object argument. [Returns] if the argument is null, then a string equal to "null"; otherwise, the value of obj.toString() is returned.

Thus this is a case of absolutely no coercion by the language – delegating every concern to the objects itself.


C#

According to the Jon Skeet answer here, operator + is not even overloaded for the string class – akin to Java, this is just convenience generated by the compiler, thanks to both static and strong typing.


Perl

As the perldata explains,

Perl has three built-in data types: scalars, arrays of scalars, and associative arrays of scalars, known as “hashes”. A scalar is a single string (of any size, limited only by the available memory), number, or a reference to something (which will be discussed in perlref). Normal arrays are ordered lists of scalars indexed by number, starting with 0. Hashes are unordered collections of scalar values indexed by their associated string key.

Perl however does not have a separate data type for numbers, booleans, strings, nulls, undefineds, references to other objects etc – it just has one type for these all, the scalar type; 0 is a scalar value as much as is “0”. A scalar variable that was set as a string can really change into a number, and from there on behave differently from “just a string”, if it is accessed in a numerical context. The scalar can hold anything in Perl, it is as much the object as it exists in the system. whereas in Python the names just refers to the objects, in Perl the scalar values in the names are changeable objects. Furthermore, the Object Oriented Type system is glued on top of this: there are just 3 datatypes in perl – scalars, lists and hashes. A user defined object in Perl is a reference (that is a pointer to any of the 3 previous) blessed to a package – you can take any such value and bless it to any class at any instant you want.

Perl even allows you to change the classes of values at whim – this is not possible in Python where to create a value of some class you need to explicitly construct the value belonging to that class with object.__new__ or similar. In Python you cannot really change the essence of the object after the creation, in Perl you can do much anything:

package Foo;
package Bar;

my $val = 42;
# $val is now a scalar value set from double
bless \$val, Foo;
# all references to $val now belong to class Foo
my $obj = \$val;
# now $obj refers to the SV stored in $val
# thus this prints: Foo=SCALAR(0x1c7d8c8)
print \$val, "\n"; 
# all references to $val now belong to class Bar
bless \$val, Bar;
# thus this prints Bar=SCALAR(0x1c7d8c8)
print \$val, "\n";
# we change the value stored in $val from number to a string
$val = 'abc';
# yet still the SV is blessed: Bar=SCALAR(0x1c7d8c8)
print \$val, "\n";
# and on the course, the $obj now refers to a "Bar" even though
# at the time of copying it did refer to a "Foo".
print $obj, "\n";

thus the type identity is weakly bound to the variable, and it can be changed through any reference on the fly. In fact, if you do

my $another = $val;

\$another does not have the class identity, even though \$val will still give the blessed reference.


TL;DR

There are much more about weak typing to Perl than just automatic coercions, and it is more about that the types of the values themselves are not set into stone, unlike the Python which is dynamically yet very strongly typed language. That python gives TypeError on 1 + "1" is an indication that the language is strongly typed, even though the contrary one of doing something useful, as in Java or C# does not preclude them being strongly typed languages.


回答 7

正如许多其他人所表示的那样,“强”键入与“弱”键入的整个概念都是有问题的。

作为一个原型,Smalltalk是非常强类型的- 如果两个对象之间的操作不兼容,它将始终引发异常。但是,我怀疑此列表中很少有人将Smalltalk称为强类型语言,因为它是动态类型。

我发现“静态”与“动态”打字的概念比“强”与“弱”的打字更有用。静态类型的语言具有在编译时确定的所有类型,否则程序员必须明确声明。

与动态类型语言相反,后者是在运行时执行键入的。这通常是多态语言的要求,因此程序员不必事先确定关于两个对象之间的操作是否合法的决定。

在多态,动态类型的语言(如Smalltalk和Ruby)中,将“类型”视为“符合协议”更为有用。如果一个对象遵循与另一个对象相同的协议(即使两个对象不共享任何继承,混合或其他伏都教),则在运行系统中它们被视为相同的“类型”。更正确地说,此类系统中的对象是自治的,并且可以决定响应任何引用特定参数的特定消息是否有意义。

是否需要一个对象,该对象可以使用描述蓝色的对象参数对消息“ +”做出有意义的响应?您可以在动态类型的语言中执行此操作,但是在静态类型的语言中则很麻烦。

As many others have expressed, the entire notion of “strong” vs “weak” typing is problematic.

As a archetype, Smalltalk is very strongly typed — it will always raise an exception if an operation between two objects is incompatible. However, I suspect few on this list would call Smalltalk a strongly-typed language, because it is dynamically typed.

I find the notion of “static” versus “dynamic” typing more useful than “strong” versus “weak.” A statically-typed language has all the types figured out at compile-time, and the programmer has to explicitly declare if otherwise.

Contrast with a dynamically-typed language, where typing is performed at run-time. This is typically a requirement for polymorphic languages, so that decisions about whether an operation between two objects is legal does not have to be decided by the programmer in advance.

In polymorphic, dynamically-typed languages (like Smalltalk and Ruby), it’s more useful to think of a “type” as a “conformance to protocol.” If an object obeys a protocol the same way another object does — even if the two objects do not share any inheritance or mixins or other voodoo — they are considered the same “type” by the run-time system. More correctly, an object in such systems is autonomous, and can decide if it makes sense to respond to any particular message referring to any particular argument.

Want an object that can make some meaningful response to the message “+” with an object argument that describes the colour blue? You can do that in dynamically-typed languages, but it is a pain in statically-typed languages.


回答 8

我喜欢@Eric Lippert的答案,但要解决这个问题-强类型语言通常在程序的每个点都具有变量类型的显式知识。弱类型语言不会,因此它们可以尝试执行某种特定类型可能无法执行的操作。它认为最简单的方法是在函数中。C ++:

void func(string a) {...}

a已知该变量的类型为字符串,任何不兼容的操作都将在编译时捕获。

Python:

def func(a)
  ...

该变量a可以是任何东西,我们可以拥有调用无效方法的代码,该方法只会在运行时被捕获。

I like @Eric Lippert’s answer, but to address the question – strongly typed languages typically have explicit knowledge of the types of variables at each point of the program. Weakly typed languages do not, so they can attempt to perform an operation that may not be possible for a particular type. It think the easiest way to see this is in a function. C++:

void func(string a) {...}

The variable a is known to be of type string and any incompatible operation will be caught at compile time.

Python:

def func(a)
  ...

The variable a could be anything and we can have code that calls an invalid method, which will only get caught at runtime.