Python 实用宝典

Question 1

I have data which is being accessed via http request and is sent back by the server in a comma separated format, I have the following code :

site= 'www.example.com'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
soup = soup.get_text()
text=str(soup)

The content of text is as follows:

april,2,5,7
may,3,5,8
june,4,7,3
july,5,6,9

How can I save this data into a CSV file. I know I can do something along the lines of the following to iterate line by line:

import StringIO
s = StringIO.StringIO(text)
for line in s:

But i’m unsure how to now properly write each line to CSV

EDIT—> Thanks for the feedback as suggested the solution was rather simple and can be seen below.

Solution:

import StringIO
s = StringIO.StringIO(text)
with open('fileName.csv', 'w') as f:
    for line in s:
        f.write(line)

Question 2

General way:

##text=List of strings to be written to file
with open('csvfile.csv','wb') as file:
    for line in text:
        file.write(line)
        file.write('\n')

OR

Using CSV writer :

import csv
with open(<path to output_csv>, "wb") as csv_file:
        writer = csv.writer(csv_file, delimiter=',')
        for line in data:
            writer.writerow(line)

OR

Simplest way:

f = open('csvfile.csv','w')
f.write('hi there\n') #Give your csv text here.
## Python will convert \n to os.linesep
f.close()

Question 3

You could just write to the file as you would write any normal file.

with open('csvfile.csv','wb') as file:
    for l in text:
        file.write(l)
        file.write('\n')

If just in case, it is a list of lists, you could directly use built-in csv module

import csv

with open("csvfile.csv", "wb") as file:
    writer = csv.writer(file)
    writer.writerows(text)

Question 4

I would simply write each line to a file, since it’s already in a CSV format:

write_file = "output.csv"
with open(write_file, "w") as output:
    for line in text:
        output.write(line + '\n')

I can’t recall how to write lines with line-breaks at the moment, though :p

Also, you might like to take a look at this answer about write(), writelines(), and '\n'.

Question 5

To complement the previous answers, I whipped up a quick class to write to CSV files. It makes it easier to manage and close open files and achieve consistency and cleaner code if you have to deal with multiple files.

class CSVWriter():

    filename = None
    fp = None
    writer = None

    def __init__(self, filename):
        self.filename = filename
        self.fp = open(self.filename, 'w', encoding='utf8')
        self.writer = csv.writer(self.fp, delimiter=';', quotechar='"', quoting=csv.QUOTE_ALL, lineterminator='\n')

    def close(self):
        self.fp.close()

    def write(self, elems):
        self.writer.writerow(elems)

    def size(self):
        return os.path.getsize(self.filename)

    def fname(self):
        return self.filename

Example usage:

mycsv = CSVWriter('/tmp/test.csv')
mycsv.write((12,'green','apples'))
mycsv.write((7,'yellow','bananas'))
mycsv.close()
print("Written %d bytes to %s" % (mycsv.size(), mycsv.fname()))

Have fun

Question 6

What about this:

with open("your_csv_file.csv", "w") as f:
    f.write("\n".join(text))

str.join() Return a string which is the concatenation of the strings in iterable. The separator between elements is the string providing this method.

Question 7

Is there a built-in way to use read_csv to read only the first n lines of a file without knowing the length of the lines ahead of time? I have a large file that takes a long time to read, and occasionally only want to use the first, say, 20 lines to get a sample of it (and prefer not to load the full thing and take the head of it).

If I knew the total number of lines I could do something like footer_lines = total_lines - n and pass this to the skipfooter keyword arg. My current solution is to manually grab the first n lines with python and StringIO it to pandas:

import pandas as pd
from StringIO import StringIO

n = 20
with open('big_file.csv', 'r') as f:
    head = ''.join(f.readlines(n))

df = pd.read_csv(StringIO(head))

It’s not that bad, but is there a more concise, ‘pandasic’ (?) way to do it with keywords or something?

Question 8

I think you can use the nrows parameter. From the docs:

nrows : int, default None

    Number of rows of file to read. Useful for reading pieces of large files

which seems to work. Using one of the standard large test files (988504479 bytes, 5344499 lines):

In [1]: import pandas as pd

In [2]: time z = pd.read_csv("P00000001-ALL.csv", nrows=20)
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s

In [3]: len(z)
Out[3]: 20

In [4]: time z = pd.read_csv("P00000001-ALL.csv")
CPU times: user 27.63 s, sys: 1.92 s, total: 29.55 s
Wall time: 30.23 s

Question 9

我正在使用pandas库读取一些CSV数据。在我的数据中，某些列包含字符串。该字符串"nan"是一个可能的值，一个空字符串也可以。我设法让大熊猫将“ nan”读取为字符串，但是我不知道如何获取不读取空值的NaN。这是示例数据和输出

One,Two,Three
a,1,one
b,2,two
,3,three
d,4,nan
e,5,five
nan,6,
g,7,seven

>>> pandas.read_csv('test.csv', na_values={'One': [], "Three": []})
    One  Two  Three
0    a    1    one
1    b    2    two
2  NaN    3  three
3    d    4    nan
4    e    5   five
5  nan    6    NaN
6    g    7  seven

它正确地写着“男”为字符串“南”，但仍读取空单元格作为NaN的。我想传递str的converters参数read_csv（带converters={'One': str})），但它仍然读取空单元格作为NaN的。

我意识到我可以在读取后使用fillna填充值，但是真的没有办法告诉熊猫特定CSV列中的空单元格应被读取为空字符串而不是NaN吗？

Question 10

I’m using the pandas library to read in some CSV data. In my data, certain columns contain strings. The string "nan" is a possible value, as is an empty string. I managed to get pandas to read “nan” as a string, but I can’t figure out how to get it not to read an empty value as NaN. Here’s sample data and output

One,Two,Three
a,1,one
b,2,two
,3,three
d,4,nan
e,5,five
nan,6,
g,7,seven

>>> pandas.read_csv('test.csv', na_values={'One': [], "Three": []})
    One  Two  Three
0    a    1    one
1    b    2    two
2  NaN    3  three
3    d    4    nan
4    e    5   five
5  nan    6    NaN
6    g    7  seven

It correctly reads “nan” as the string “nan’, but still reads the empty cells as NaN. I tried passing in str in the converters argument to read_csv (with converters={'One': str})), but it still reads the empty cells as NaN.

I realize I can fill the values after reading, with fillna, but is there really no way to tell pandas that an empty cell in a particular CSV column should be read as an empty string instead of NaN?

Question 11

我添加了票证以在此处添加某种选项：

https://github.com/pydata/pandas/issues/1450

同时，result.fillna('')应该做你想做的

编辑：在开发版本中（最终为0.8.0），如果您指定的空列表na_values，则空字符串将在结果中保留空字符串

Question 12

I added a ticket to add an option of some sort here:

https://github.com/pydata/pandas/issues/1450

In the meantime, result.fillna('') should do what you want

EDIT: in the development version (to be 0.8.0 final) if you specify an empty list of na_values, empty strings will stay empty strings in the result

Question 13

阅读其他答案和评论后，我仍然感到困惑。但是，现在的答案似乎更简单，因此您可以开始操作。

从Pandas 0.9版（自2012年起）开始，您只需设置keep_default_na=False以下内容，即可读取解释为空字符串的空单元格的csv ：

pd.read_csv('test.csv', keep_default_na=False)

此问题在以下内容中有更清楚的说明

read_csv中更一致的na_values处理·问题＃1657·pandas-dev / pandas

该问题已于2012年8月19日在Pandas 0.9版中修复

错误：更一致的na_values＃1657·pandas-dev / pandas @ d9abf68

Question 14

I was still confused after reading the other answers and comments. But the answer now seems simpler, so here you go.

Since Pandas version 0.9 (from 2012), you can read your csv with empty cells interpreted as empty strings by simply setting keep_default_na=False:

pd.read_csv('test.csv', keep_default_na=False)

This issue is more clearly explained in

More consistent na_values handling in read_csv · Issue #1657 · pandas-dev/pandas

That was fixed on on Aug 19, 2012 for Pandas version 0.9 in

BUG: more consistent na_values #1657 · pandas-dev/pandas@d9abf68

Question 15

我们在Pandas read_csv中有一个简单的参数：

使用：

df = pd.read_csv('test.csv', na_filter= False)

熊猫的文档清楚地解释了上述论点是如何工作的。

链接

Question 16

We have a simple argument in Pandas read_csv for this:

Use:

df = pd.read_csv('test.csv', na_filter= False)

Pandas documentation clearly explains how the above argument works.

Link

Question 17

I have a long list of lists of the following form —

a = [[1.2,'abc',3],[1.2,'werew',4],........,[1.4,'qew',2]]

i.e. the values in the list are of different types — float,int, strings.How do I write it into a csv file so that my output csv file looks like

1.2,abc,3
1.2,werew,4
.
.
.
1.4,qew,2

Question 18

Python’s built-in CSV module can handle this easily:

import csv

with open("output.csv", "wb") as f:
    writer = csv.writer(f)
    writer.writerows(a)

This assumes your list is defined as a, as it is in your question. You can tweak the exact format of the output CSV via the various optional parameters to csv.writer() as documented in the library reference page linked above.

Update for Python 3

import csv

with open("out.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerows(a)

Question 19

You could use pandas:

In [1]: import pandas as pd

In [2]: a = [[1.2,'abc',3],[1.2,'werew',4],[1.4,'qew',2]]

In [3]: my_df = pd.DataFrame(a)

In [4]: my_df.to_csv('my_csv.csv', index=False, header=False)

Question 20

import csv
with open(file_path, 'a') as outcsv:   
    #configure writer to write standard csv file
    writer = csv.writer(outcsv, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL, lineterminator='\n')
    writer.writerow(['number', 'text', 'number'])
    for item in list:
        #Write item to outcsv
        writer.writerow([item[0], item[1], item[2]])

official docs: http://docs.python.org/2/library/csv.html

Question 21

If for whatever reason you wanted to do it manually (without using a module like csv,pandas,numpy etc.):

with open('myfile.csv','w') as f:
    for sublist in mylist:
        for item in sublist:
            f.write(item + ',')
        f.write('\n')

Of course, rolling your own version can be error-prone and inefficient … that’s usually why there’s a module for that. But sometimes writing your own can help you understand how they work, and sometimes it’s just easier.

Question 22

Using csv.writer in my very large list took quite a time. I decided to use pandas, it was faster and more easy to control and understand:

 import pandas

 yourlist = [[...],...,[...]]
 pd = pandas.DataFrame(yourlist)
 pd.to_csv("mylist.csv")

The good part you can change somethings to make a better csv file:

 yourlist = [[...],...,[...]]
 columns = ["abcd","bcde","cdef"] #a csv with 3 columns
 index = [i[0] for i in yourlist] #first element of every list in yourlist
 not_index_list = [i[1:] for i in yourlist]
 pd = pandas.DataFrame(not_index_list, columns = columns, index = index)

 #Now you have a csv with columns and index:
 pd.to_csv("mylist.csv")

Question 23

Ambers’s solution also works well for numpy arrays:

from pylab import *
import csv

array_=arange(0,10,1)
list_=[array_,array_*2,array_*3]
with open("output.csv", "wb") as f:
    writer = csv.writer(f)
    writer.writerows(list_)

Question 24

If you don’t want to import csv module for that, you can write a list of lists to a csv file using only Python built-ins

with open("output.csv", "w") as f:
    for row in a:
        f.write("%s\n" % ','.join(str(col) for col in row))

Question 25

Make sure to indicate lineterinator='\n' when create the writer; otherwise, an extra empty line might be written into file after each data line when data sources are from other csv file…

Here is my solution:

with open('csvfile', 'a') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter='    ',quotechar='|', quoting=csv.QUOTE_MINIMAL, lineterminator='\n')
for i in range(0, len(data)):
    spamwriter.writerow(data[i])

Question 26

How about dumping the list of list into pickle and restoring it with pickle module? It’s quite convenient.

>>> import pickle
>>> 
>>> mylist = [1, 'foo', 'bar', {1, 2, 3}, [ [1,4,2,6], [3,6,0,10]]]
>>> with open('mylist', 'wb') as f:
...     pickle.dump(mylist, f) 


>>> with open('mylist', 'rb') as f:
...      mylist = pickle.load(f)
>>> mylist
[1, 'foo', 'bar', {1, 2, 3}, [[1, 4, 2, 6], [3, 6, 0, 10]]]
>>>

Question 27

I got an error message when following the examples with a newline parameter in the csv.writer function. The following code worked for me.

 with open(strFileName, "w") as f:
    writer = csv.writer(f, delimiter=',',  quoting=csv.QUOTE_MINIMAL)
    writer.writerows(result)

Question 28

I am reading a csv file into pandas. This csv file constists of four columns and some rows, but does not have a header row, which I want to add. I have been trying the following:

Cov = pd.read_csv("path/to/file.txt", sep='\t')
Frame=pd.DataFrame([Cov], columns = ["Sequence", "Start", "End", "Coverage"])
Frame.to_csv("path/to/file.txt", sep='\t')

But when I apply the code, I get the following Error:

ValueError: Shape of passed values is (1, 1), indices imply (4, 1)

What exactly does the error mean? And what would be a clean way in python to add a header row to my csv file/pandas df?

Question 29

You can use names directly in the read_csv

names : array-like, default None List of column names to use. If file contains no header row, then you should explicitly pass header=None

Cov = pd.read_csv("path/to/file.txt", 
                  sep='\t', 
                  names=["Sequence", "Start", "End", "Coverage"])

Question 30

Alternatively you could read you csv with header=None and then add it with df.columns:

Cov = pd.read_csv("path/to/file.txt", sep='\t', header=None)
Cov.columns = ["Sequence", "Start", "End", "Coverage"]

Question 31

col_Names=["Sequence", "Start", "End", "Coverage"]
my_CSV_File= pd.read_csv("yourCSVFile.csv",names=col_Names)

having done this, just check it with[well obviously I know, u know that. But still…

my_CSV_File.head()

Hope it helps … Cheers

Question 32

To fix your code you can simply change [Cov] to Cov.values, the first parameter of pd.DataFrame will become a multi-dimensional numpy array:

Cov = pd.read_csv("path/to/file.txt", sep='\t')
Frame=pd.DataFrame(Cov.values, columns = ["Sequence", "Start", "End", "Coverage"])
Frame.to_csv("path/to/file.txt", sep='\t')

But the smartest solution still is use pd.read_excel with header=None and names=columns_list.

Question 33

I have a list of dictionaries that looks something like this:

toCSV = [{'name':'bob','age':25,'weight':200},{'name':'jim','age':31,'weight':180}]

What should I do to convert this to a csv file that looks something like this:

name,age,weight
bob,25,200
jim,31,180

Question 34

import csv
toCSV = [{'name':'bob','age':25,'weight':200},
         {'name':'jim','age':31,'weight':180}]
keys = toCSV[0].keys()
with open('people.csv', 'w', newline='')  as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)

EDIT: My prior solution doesn’t handle the order. As noted by Wilduck, DictWriter is more appropriate here.

Question 35

this is when you have one dictionary list:

import csv
with open('names.csv', 'w') as csvfile:
    fieldnames = ['first_name', 'last_name']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})

Question 36

In python 3 things are a little different, but way simpler and less error prone. It’s a good idea to tell the CSV your file should be opened with utf8 encoding, as it makes that data more portable to others (assuming you aren’t using a more restrictive encoding, like latin1)

import csv
toCSV = [{'name':'bob','age':25,'weight':200},
         {'name':'jim','age':31,'weight':180}]
with open('people.csv', 'w', encoding='utf8', newline='') as output_file:
    fc = csv.DictWriter(output_file, 
                        fieldnames=toCSV[0].keys(),

                       )
    fc.writeheader()
    fc.writerows(toCSV)

Note that csv in python 3 needs the newline='' parameter, otherwise you get blank lines in your CSV when opening in excel/opencalc.

Alternatively: I prefer use to the csv handler in the pandas module. I find it is more tolerant of encoding issues, and pandas will automatically convert string numbers in CSVs into the correct type (int,float,etc) when loading the file.

import pandas
dataframe = pandas.read_csv(filepath)
list_of_dictionaries = dataframe.to_dict('records')
dataframe.to_csv(filepath)

Note:

pandas will take care of opening the file for you if you give it a path, and will default to utf8 in python3, and figure out headers too.
a dataframe is not the same structure as what CSV gives you, so you add one line upon loading to get the same thing: dataframe.to_dict('records')
pandas also makes it much easier to control the order of columns in your csv file. By default, they’re alphabetical, but you can specify the column order. With vanilla csv module, you need to feed it an OrderedDict or they’ll appear in a random order (if working in python < 3.5). See: Preserving column order in Python Pandas DataFrame for more.

Question 37

Because @User and @BiXiC asked for help with UTF-8 here a variation of the solution by @Matthew. (I’m not allowed to comment, so I’m answering.)

import unicodecsv as csv
toCSV = [{'name':'bob','age':25,'weight':200},
         {'name':'jim','age':31,'weight':180}]
keys = toCSV[0].keys()
with open('people.csv', 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)

Question 38

import csv

with open('file_name.csv', 'w') as csv_file:
    writer = csv.writer(csv_file)
    writer.writerow(('colum1', 'colum2', 'colum3'))
    for key, value in dictionary.items():
        writer.writerow([key, value[0], value[1]])

This would be the simplest way to write data to .csv file

Question 39

Here is another, more general solution assuming you don’t have a list of rows (maybe they don’t fit in memory) or a copy of the headers (maybe the write_csv function is generic):

def gen_rows():
    yield OrderedDict(name='bob', age=25, weight=200)
    yield OrderedDict(name='jim', age=31, weight=180)

def write_csv():
    it = genrows()
    first_row = it.next()  # __next__ in py3
    with open("people.csv", "w") as outfile:
        wr = csv.DictWriter(outfile, fieldnames=list(first_row))
        wr.writeheader()
        wr.writerow(first_row)
        wr.writerows(it)

Note: the OrderedDict constructor used here only preserves order in python >3.4. If order is important, use the OrderedDict([('name', 'bob'),('age',25)]) form.

Question 40

import csv
toCSV = [{'name':'bob','age':25,'weight':200},
         {'name':'jim','age':31,'weight':180}]
header=['name','age','weight']     
try:
   with open('output'+str(date.today())+'.csv',mode='w',encoding='utf8',newline='') as output_to_csv:
       dict_csv_writer = csv.DictWriter(output_to_csv, fieldnames=header,dialect='excel')
       dict_csv_writer.writeheader()
       dict_csv_writer.writerows(toCSV)
   print('\nData exported to csv succesfully and sample data')
except IOError as io:
    print('\n',io)

Question 41

Sample.csv contains the following:

NAME    Id   No  Dept
Tom     1    12   CS
Hendry  2    35   EC
Bahamas 3    21   IT
Frank   4    61   EE

And the Python file contains the following code:

import csv
ifile  = open('sample.csv', "rb")
read = csv.reader(ifile)
for row in read :
    print (row)

When I run the above code in Python, I get the following exception:

File “csvformat.py”, line 4, in for row in read : _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

How can I fix it?

Question 42

You open the file in text mode.

More specifically:

ifile  = open('sample.csv', "rt", encoding=<theencodingofthefile>)

Good guesses for encoding is “ascii” and “utf8”. You can also leave the encoding off, and it will use the system default encoding, which tends to be UTF8, but may be something else.

Question 43

The reason it is throwing that exception is because you have the argument rb, which opens the file in binary mode. Change that to r, which will by default open the file in text mode.

Your code:

import csv
ifile  = open('sample.csv', "rb")
read = csv.reader(ifile)
for row in read :
    print (row)

New code:

import csv
ifile  = open('sample.csv', "r")
read = csv.reader(ifile)
for row in read :
    print (row)

Question 44

Your problem is you have the b in the open flag. The flag rt (read, text) is the default, so, using the context manager, simply do this:

with open('sample.csv') as ifile:
    read = csv.reader(ifile) 
    for row in read:
        print (row)

The context manager means you don’t need generic error handling (without which you may get stuck with the file open, especially in an interpreter), because it will automatically close the file on an error, or on exiting the context.

The above is the same as:

with open('sample.csv', 'r') as ifile:
    ...

or

with open('sample.csv', 'rt') as ifile:
    ...

Question 45

In Python3, csv.reader expects, that passed iterable returns strings, not bytes. Here is one more solution to this problem, that uses codecs module:

import csv
import codecs
ifile  = open('sample.csv', "rb")
read = csv.reader(codecs.iterdecode(ifile, 'utf-8'))
for row in read :
    print (row)

Question 46

I had this error when running an old python script developped with Python 2.6.4

When updating to 3.6.2, I had to remove all ‘rb’ parameters from open calls in order to fix this csv reading error.

Question 47

I am trying to create a dictionary from a csv file. The first column of the csv file contains unique keys and the second column contains values. Each row of the csv file represents a unique key, value pair within the dictionary. I tried to use the csv.DictReader and csv.DictWriter classes, but I could only figure out how to generate a new dictionary for each row. I want one dictionary. Here is the code I am trying to use:

import csv

with open('coors.csv', mode='r') as infile:
    reader = csv.reader(infile)
    with open('coors_new.csv', mode='w') as outfile:
    writer = csv.writer(outfile)
    for rows in reader:
        k = rows[0]
        v = rows[1]
        mydict = {k:v for k, v in rows}
    print(mydict)

When I run the above code I get a ValueError: too many values to unpack (expected 2). How do I create one dictionary from a csv file? Thanks.

Question 48

I believe the syntax you were looking for is as follows:

import csv

with open('coors.csv', mode='r') as infile:
    reader = csv.reader(infile)
    with open('coors_new.csv', mode='w') as outfile:
        writer = csv.writer(outfile)
        mydict = {rows[0]:rows[1] for rows in reader}

Alternately, for python <= 2.7.1, you want:

mydict = dict((rows[0],rows[1]) for rows in reader)

Question 49

Open the file by calling open and then csv.DictReader.

input_file = csv.DictReader(open("coors.csv"))

You may iterate over the rows of the csv file dict reader object by iterating over input_file.

for row in input_file:
    print(row)

OR To access first line only

dictobj = csv.DictReader(open('coors.csv')).next()

UPDATE In python 3+ versions, this code would change a little:

reader = csv.DictReader(open('coors.csv'))
dictobj = next(reader)

Question 50

import csv
reader = csv.reader(open('filename.csv', 'r'))
d = {}
for row in reader:
   k, v = row
   d[k] = v

Question 51

This isn’t elegant but a one line solution using pandas.

import pandas as pd
pd.read_csv('coors.csv', header=None, index_col=0, squeeze=True).to_dict()

If you want to specify dtype for your index (it can’t be specified in read_csv if you use the index_col argument because of a bug):

import pandas as pd
pd.read_csv('coors.csv', header=None, dtype={0: str}).set_index(0).squeeze().to_dict()

Question 52

You have to just convert csv.reader to dict:

~ >> cat > 1.csv
key1, value1
key2, value2
key2, value22
key3, value3

~ >> cat > d.py
import csv
with open('1.csv') as f:
    d = dict(filter(None, csv.reader(f)))

print(d)

~ >> python d.py
{'key3': ' value3', 'key2': ' value22', 'key1': ' value1'}

Question 53

You can also use numpy for this.

from numpy import loadtxt
key_value = loadtxt("filename.csv", delimiter=",")
mydict = { k:v for k,v in key_value }

Question 54

I’d suggest adding if rows in case there is an empty line at the end of the file

import csv
with open('coors.csv', mode='r') as infile:
    reader = csv.reader(infile)
    with open('coors_new.csv', mode='w') as outfile:
        writer = csv.writer(outfile)
        mydict = dict(row[:2] for row in reader if row)

Question 55

One-liner solution

import pandas as pd

dict = {row[0] : row[1] for _, row in pd.read_csv("file.csv").iterrows()}

Question 56

If you are OK with using the numpy package, then you can do something like the following:

import numpy as np

lines = np.genfromtxt("coors.csv", delimiter=",", dtype=None)
my_dict = dict()
for i in range(len(lines)):
   my_dict[lines[i][0]] = lines[i][1]

Question 57

For simple csv files, such as the following

id,col1,col2,col3
row1,r1c1,r1c2,r1c3
row2,r2c1,r2c2,r2c3
row3,r3c1,r3c2,r3c3
row4,r4c1,r4c2,r4c3

You can convert it to a Python dictionary using only built-ins

with open(csv_file) as f:
    csv_list = [[val.strip() for val in r.split(",")] for r in f.readlines()]

(_, *header), *data = csv_list
csv_dict = {}
for row in data:
    key, *values = row   
    csv_dict[key] = {key: value for key, value in zip(header, values)}

This should yield the following dictionary

{'row1': {'col1': 'r1c1', 'col2': 'r1c2', 'col3': 'r1c3'},
 'row2': {'col1': 'r2c1', 'col2': 'r2c2', 'col3': 'r2c3'},
 'row3': {'col1': 'r3c1', 'col2': 'r3c2', 'col3': 'r3c3'},
 'row4': {'col1': 'r4c1', 'col2': 'r4c2', 'col3': 'r4c3'}}

Note: Python dictionaries have unique keys, so if your csv file has duplicate ids you should append each row to a list.

for row in data:
    key, *values = row

    if key not in csv_dict:
            csv_dict[key] = []

    csv_dict[key].append({key: value for key, value in zip(header, values)})

Question 58

You can use this, it is pretty cool:

import dataconverters.commas as commas
filename = 'test.csv'
with open(filename) as f:
      records, metadata = commas.parse(f)
      for row in records:
            print 'this is row in dictionary:'+rowenter code here

Question 59

Many solutions have been posted and I’d like to contribute with mine, which works for a different number of columns in the CSV file. It creates a dictionary with one key per column, and the value for each key is a list with the elements in such column.

    input_file = csv.DictReader(open(path_to_csv_file))
    csv_dict = {elem: [] for elem in input_file.fieldnames}
    for row in input_file:
        for key in csv_dict.keys():
            csv_dict[key].append(row[key])

Question 60

with pandas, it is much easier, for example. assuming you have the following data as CSV and let’s call it test.txt / test.csv (you know CSV is a sort of text file )

a,b,c,d
1,2,3,4
5,6,7,8

now using pandas

import pandas as pd
df = pd.read_csv("./text.txt")
df_to_doct = df.to_dict()

for each row, it would be

df.to_dict(orient='records')

and that’s it.

Question 61

Try to use a defaultdict and DictReader.

import csv
from collections import defaultdict
my_dict = defaultdict(list)

with open('filename.csv', 'r') as csv_file:
    csv_reader = csv.DictReader(csv_file)
    for line in csv_reader:
        for key, value in line.items():
            my_dict[key].append(value)

It returns:

{'key1':[value_1, value_2, value_3], 'key2': [value_a, value_b, value_c], 'Key3':[value_x, Value_y, Value_z]}

Question 62

I have a situation wherein sometimes when I read a csv from df I get an unwanted index-like column named unnamed:0.

file.csv

,A,B,C
0,1,2,3
1,4,5,6
2,7,8,9

The CSV is read with this:

pd.read_csv('file.csv')

   Unnamed: 0  A  B  C
0           0  1  2  3
1           1  4  5  6
2           2  7  8  9

This is very annoying! Does anyone have an idea on how to get rid of this?

Question 63

It’s the index column, pass index=False to not write it out, see the docs

Example:

In [37]:
df = pd.DataFrame(np.random.randn(5,3), columns=list('abc'))
pd.read_csv(io.StringIO(df.to_csv()))

Out[37]:
   Unnamed: 0         a         b         c
0           0  0.109066 -1.112704 -0.545209
1           1  0.447114  1.525341  0.317252
2           2  0.507495  0.137863  0.886283
3           3  1.452867  1.888363  1.168101
4           4  0.901371 -0.704805  0.088335

compare with:

In [38]:
pd.read_csv(io.StringIO(df.to_csv(index=False)))

Out[38]:
          a         b         c
0  0.109066 -1.112704 -0.545209
1  0.447114  1.525341  0.317252
2  0.507495  0.137863  0.886283
3  1.452867  1.888363  1.168101
4  0.901371 -0.704805  0.088335

You could also optionally tell read_csv that the first column is the index column by passing index_col=0:

In [40]:
pd.read_csv(io.StringIO(df.to_csv()), index_col=0)

Out[40]:
          a         b         c
0  0.109066 -1.112704 -0.545209
1  0.447114  1.525341  0.317252
2  0.507495  0.137863  0.886283
3  1.452867  1.888363  1.168101
4  0.901371 -0.704805  0.088335

Question 64

This issue most likely manifests because your CSV was saved along with its RangeIndex (which usually doesn’t have a name). The fix would actually need to be done when saving the DataFrame, but this isn’t always an option.

Avoiding the Problem: `read_csv` with `index_col` argument

IMO, the simplest solution would be to read the unnamed column as the index. Specify an index_col=[0] argument to pd.read_csv, this reads in the first column as the index.

df = pd.DataFrame('x', index=range(5), columns=list('abc'))
df

   a  b  c
0  x  x  x
1  x  x  x
2  x  x  x
3  x  x  x
4  x  x  x

# Save DataFrame to CSV.
df.to_csv('file.csv')

pd.read_csv('file.csv')

   Unnamed: 0  a  b  c
0           0  x  x  x
1           1  x  x  x
2           2  x  x  x
3           3  x  x  x
4           4  x  x  x

# Now try this again, with the extra argument.
pd.read_csv('file.csv', index_col=[0])

   a  b  c
0  x  x  x
1  x  x  x
2  x  x  x
3  x  x  x
4  x  x  x

Note
You could have avoided this in the first place by using index=False when creating the output CSV, if your DataFrame does not have an index to begin with.
df.to_csv('file.csv', index=False)
But as mentioned above, this isn’t always an option.

Stopgap Solution: Filtering with `str.match`

If you cannot modify the code to read/write the CSV file, you can just remove the column by filtering with str.match:

df 

   Unnamed: 0  a  b  c
0           0  x  x  x
1           1  x  x  x
2           2  x  x  x
3           3  x  x  x
4           4  x  x  x

df.columns
# Index(['Unnamed: 0', 'a', 'b', 'c'], dtype='object')

df.columns.str.match('Unnamed')
# array([ True, False, False, False])

df.loc[:, ~df.columns.str.match('Unnamed')]

   a  b  c
0  x  x  x
1  x  x  x
2  x  x  x
3  x  x  x
4  x  x  x

问题：Python逐行写入CSV

回答 0

回答 1

回答 2

回答 3

回答 4

问题：读取pandas数据框的前几行的方法

回答 0

问题：获取pandas.read_csv以将空值读取为空字符串而不是nan

回答 0

回答 1

回答 2

问题：将列表的Python列表写入csv文件

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

问题：如何将标题行添加到Pandas DataFrame

回答 0

回答 1

回答 2

回答 3

问题：如何将此字典列表转换为csv文件？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：csv.Error：迭代器应返回字符串，而不是字节

回答 0

回答 1

回答 2

回答 3

回答 4

问题：从CSV文件创建字典？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

问题：如何摆脱熊猫DataFrame中的“未命名：0”列？

回答 0

回答 1

避免问题：read_csv带有index_col 参数

权宜之计解决方案：过滤 str.match

Avoiding the Problem: read_csv with index_col argument

Stopgap Solution: Filtering with str.match

回答 2

回答 3

回答 4

问题：写入Excel电子表格

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

避免问题：`read_csv`带有`index_col` 参数

权宜之计解决方案：过滤 `str.match`

Avoiding the Problem: `read_csv` with `index_col` argument

Stopgap Solution: Filtering with `str.match`