标签归档:genfromtxt

如何将CSV数据读入NumPy中的记录数组?

问题:如何将CSV数据读入NumPy中的记录数组?

我不知道是否有一个CSV文件的内容导入到一个记录阵列直接的方式,很多的方式是R的read.table()read.delim()read.csv()家庭的进口数据与R的数据帧?

还是使用csv.reader()然后应用类似内容的最佳方法numpy.core.records.fromrecords()

I wonder if there is a direct way to import the contents of a CSV file into a record array, much in the way that R’s read.table(), read.delim(), and read.csv() family imports data to R’s data frame?

Or is the best way to use csv.reader() and then apply something like numpy.core.records.fromrecords()?


回答 0

您可以genfromtxt()通过将delimiterkwarg 设置为逗号来使用Numpy的方法。

from numpy import genfromtxt
my_data = genfromtxt('my_file.csv', delimiter=',')

有关该功能的更多信息,请参见其相应的文档

You can use Numpy’s genfromtxt() method to do so, by setting the delimiter kwarg to a comma.

from numpy import genfromtxt
my_data = genfromtxt('my_file.csv', delimiter=',')

More information on the function can be found at its respective documentation.


回答 1

我会read_csvpandas库中推荐该功能:

import pandas as pd
df=pd.read_csv('myfile.csv', sep=',',header=None)
df.values
array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

这提供了一个熊猫DataFrame-允许许多有用的数据操作功能,而numpy记录数组无法直接使用这些功能

DataFrame是二维标记的数据结构,具有可能不同类型的列。您可以将其视为电子表格或SQL表…


我也建议genfromtxt。但是,由于该问题要求记录数组,而不是普通数组,因此dtype=None需要将参数添加到genfromtxt调用中:

给定一个输入文件,myfile.csv

1.0, 2, 3
4, 5.5, 6

import numpy as np
np.genfromtxt('myfile.csv',delimiter=',')

给出一个数组:

array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

np.genfromtxt('myfile.csv',delimiter=',',dtype=None)

给出一个记录数组:

array([(1.0, 2.0, 3), (4.0, 5.5, 6)], 
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<i4')])

这样的优点是可以轻松导入具有多种数据类型(包括字符串)的文件。

I would recommend the read_csv function from the pandas library:

import pandas as pd
df=pd.read_csv('myfile.csv', sep=',',header=None)
df.values
array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

This gives a pandas DataFrame – allowing many useful data manipulation functions which are not directly available with numpy record arrays.

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table…


I would also recommend genfromtxt. However, since the question asks for a record array, as opposed to a normal array, the dtype=None parameter needs to be added to the genfromtxt call:

Given an input file, myfile.csv:

1.0, 2, 3
4, 5.5, 6

import numpy as np
np.genfromtxt('myfile.csv',delimiter=',')

gives an array:

array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

and

np.genfromtxt('myfile.csv',delimiter=',',dtype=None)

gives a record array:

array([(1.0, 2.0, 3), (4.0, 5.5, 6)], 
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<i4')])

This has the advantage that file with multiple data types (including strings) can be easily imported.


回答 2

我定时了

from numpy import genfromtxt
genfromtxt(fname = dest_file, dtype = (<whatever options>))

import csv
import numpy as np
with open(dest_file,'r') as dest_f:
    data_iter = csv.reader(dest_f,
                           delimiter = delimiter,
                           quotechar = '"')
    data = [data for data in data_iter]
data_array = np.asarray(data, dtype = <whatever options>)

在460万行,约70列的数据上,发现NumPy路径花费了2分16秒,而csv-list理解方法花费了13秒。

我建议使用csv-list理解方法,因为它很可能依赖于预编译的库,而不像NumPy那样依赖于解释器。我怀疑pandas方法会有类似的解释器开销。

I timed the

from numpy import genfromtxt
genfromtxt(fname = dest_file, dtype = (<whatever options>))

versus

import csv
import numpy as np
with open(dest_file,'r') as dest_f:
    data_iter = csv.reader(dest_f,
                           delimiter = delimiter,
                           quotechar = '"')
    data = [data for data in data_iter]
data_array = np.asarray(data, dtype = <whatever options>)

on 4.6 million rows with about 70 columns and found that the NumPy path took 2 min 16 secs and the csv-list comprehension method took 13 seconds.

I would recommend the csv-list comprehension method as it is most likely relies on pre-compiled libraries and not the interpreter as much as NumPy. I suspect the pandas method would have similar interpreter overhead.


回答 3

您也可以尝试使用recfromcsv()哪种方法可以猜测数据类型并返回格式正确的记录数组。

You can also try recfromcsv() which can guess data types and return a properly formatted record array.


回答 4

当我尝试使用NumPy和Pandas两种方式时,使用Pandas有很多优点:

  • 快点
  • 减少CPU使用率
  • 与NumPy genfromtxt相比1/3的RAM使用量

这是我的测试代码:

$ for f in test_pandas.py test_numpy_csv.py ; do  /usr/bin/time python $f; done
2.94user 0.41system 0:03.05elapsed 109%CPU (0avgtext+0avgdata 502068maxresident)k
0inputs+24outputs (0major+107147minor)pagefaults 0swaps

23.29user 0.72system 0:23.72elapsed 101%CPU (0avgtext+0avgdata 1680888maxresident)k
0inputs+0outputs (0major+416145minor)pagefaults 0swaps

test_numpy_csv.py

from numpy import genfromtxt
train = genfromtxt('/home/hvn/me/notebook/train.csv', delimiter=',')

test_pandas.py

from pandas import read_csv
df = read_csv('/home/hvn/me/notebook/train.csv')

资料档案:

du -h ~/me/notebook/train.csv
 59M    /home/hvn/me/notebook/train.csv

使用NumPy和pandas版本:

$ pip freeze | egrep -i 'pandas|numpy'
numpy==1.13.3
pandas==0.20.2

As I tried both ways using NumPy and Pandas, using pandas has a lot of advantages:

  • Faster
  • Less CPU usage
  • 1/3 RAM usage compared to NumPy genfromtxt

This is my test code:

$ for f in test_pandas.py test_numpy_csv.py ; do  /usr/bin/time python $f; done
2.94user 0.41system 0:03.05elapsed 109%CPU (0avgtext+0avgdata 502068maxresident)k
0inputs+24outputs (0major+107147minor)pagefaults 0swaps

23.29user 0.72system 0:23.72elapsed 101%CPU (0avgtext+0avgdata 1680888maxresident)k
0inputs+0outputs (0major+416145minor)pagefaults 0swaps

test_numpy_csv.py

from numpy import genfromtxt
train = genfromtxt('/home/hvn/me/notebook/train.csv', delimiter=',')

test_pandas.py

from pandas import read_csv
df = read_csv('/home/hvn/me/notebook/train.csv')

Data file:

du -h ~/me/notebook/train.csv
 59M    /home/hvn/me/notebook/train.csv

With NumPy and pandas at versions:

$ pip freeze | egrep -i 'pandas|numpy'
numpy==1.13.3
pandas==0.20.2

回答 5

您可以使用以下代码将CSV文件数据发送到数组中:

import numpy as np
csv = np.genfromtxt('test.csv', delimiter=",")
print(csv)

You can use this code to send CSV file data into an array:

import numpy as np
csv = np.genfromtxt('test.csv', delimiter=",")
print(csv)

回答 6

使用 numpy.loadtxt

一个非常简单的方法。但这要求所有元素都是浮点数(int等)

import numpy as np 
data = np.loadtxt('c:\\1.csv',delimiter=',',skiprows=0)  

Using numpy.loadtxt

A quite simple method. But it requires all the elements being float (int and so on)

import numpy as np 
data = np.loadtxt('c:\\1.csv',delimiter=',',skiprows=0)  

回答 7

这是最简单的方法:

import csv with open('testfile.csv', newline='') as csvfile: data = list(csv.reader(csvfile))

现在,数据中的每个条目都是一条记录,表示为一个数组。因此,您拥有一个2D阵列。它节省了我很多时间。

This is the easiest way:

import csv with open('testfile.csv', newline='') as csvfile: data = list(csv.reader(csvfile))

Now each entry in data is a record, represented as an array. So you have a 2D array. It saved me so much time.


回答 8

我尝试了这个:

import pandas as p
import numpy as n

closingValue = p.read_csv("<FILENAME>", usecols=[4], dtype=float)
print(closingValue)

I tried this:

import pandas as p
import numpy as n

closingValue = p.read_csv("<FILENAME>", usecols=[4], dtype=float)
print(closingValue)

回答 9

我建议使用表格(pip3 install tables)。您可以将.csv文件保存为.h5使用熊猫(pip3 install pandas),

import pandas as pd
data = pd.read_csv("dataset.csv")
store = pd.HDFStore('dataset.h5')
store['mydata'] = data
store.close()

然后,您可以轻松地以较少的时间(即使是处理大量数据)将数据加载到NumPy数组中

import pandas as pd
store = pd.HDFStore('dataset.h5')
data = store['mydata']
store.close()

# Data in NumPy format
data = data.values

I would suggest using tables (pip3 install tables). You can save your .csv file to .h5 using pandas (pip3 install pandas),

import pandas as pd
data = pd.read_csv("dataset.csv")
store = pd.HDFStore('dataset.h5')
store['mydata'] = data
store.close()

You can then easily, and with less time even for huge amount of data, load your data in a NumPy array.

import pandas as pd
store = pd.HDFStore('dataset.h5')
data = store['mydata']
store.close()

# Data in NumPy format
data = data.values

回答 10

这项工作令人着迷…

import csv
with open("data.csv", 'r') as f:
    data = list(csv.reader(f, delimiter=";"))

import numpy as np
data = np.array(data, dtype=np.float)

This work as a charm…

import csv
with open("data.csv", 'r') as f:
    data = list(csv.reader(f, delimiter=";"))

import numpy as np
data = np.array(data, dtype=np.float)