Python 实用宝典

Question 1

As I know, %debug magic can do debug within one cell.

However, I have function calls across multiple cells.

For example,

In[1]: def fun1(a)
           def fun2(b)
               # I want to set a breakpoint for the following line #
               return do_some_thing_about(b)

       return fun2(a)

In[2]: import multiprocessing as mp
       pool=mp.Pool(processes=2)
       results=pool.map(fun1, 1.0)
       pool.close()
       pool.join

What I tried:

I tried to set %debug in the first line of cell-1. But it enter into debug mode immediately, even before executing cell-2.
I tried to add %debug in the line right before the code return do_some_thing_about(b). But then the code runs forever, never stops.

What is the right way to set a break point within the ipython notebook?

Question 2

Use ipdb

Install it via

pip install ipdb

Usage:

In[1]: def fun1(a):
   def fun2(a):
       import ipdb; ipdb.set_trace() # debugging starts here
       return do_some_thing_about(b)
   return fun2(a)
In[2]: fun1(1)

For executing line by line use n and for step into a function use s and to exit from debugging prompt use c.

For complete list of available commands: https://appletree.or.kr/quick_reference_cards/Python/Python%20Debugger%20Cheatsheet.pdf

Question 3

You can use ipdb inside jupyter with:

from IPython.core.debugger import Tracer; Tracer()()

Edit: the functions above are deprecated since IPython 5.1. This is the new approach:

from IPython.core.debugger import set_trace

Add set_trace() where you need a breakpoint. Type help for ipdb commands when the input field appears.

Question 4

Your return function is in line of def function(main function), you must give one tab to it. And Use

%%debug

instead of

%debug

to debug the whole cell not only line. Hope, maybe this will help you.

Question 5

You can always add this in any cell:

import pdb; pdb.set_trace()

and the debugger will stop on that line. For example:

In[1]: def fun1(a):
           def fun2(a):
               import pdb; pdb.set_trace() # debugging starts here
           return fun2(a)

In[2]: fun1(1)

Question 6

In Python 3.7 you can use breakpoint() function. Just enter

breakpoint()

wherever you would like runtime to stop and from there you can use the same pdb commands (r, c, n, …) or evaluate your variables.

Question 7

Just type import pdb in jupyter notebook, and then use this cheatsheet to debug. It’s very convenient.

c –> continue, s –> step, b 12 –> set break point at line 12 and so on.

Some useful links: Python Official Document on pdb, Python pdb debugger examples for better understanding how to use the debugger commands.

Some useful screenshots:

Question 8

After you get an error, in the next cell just run %debug and that’s it.

Question 9

The %pdb magic command is good to use as well. Just say %pdb on and subsequently the pdb debugger will run on all exceptions, no matter how deep in the call stack. Very handy.

If you have a particular line that you want to debug, just raise an exception there (often you already are!) or use the %debug magic command that other folks have been suggesting.

Question 10

I just discovered PixieDebugger. Even thought I have not yet had the time to test it, it really seems the most similar way to debug the way we’re used in ipython with ipdb

It also has an “evaluate” tab

Question 11

A native debugger is being made available as an extension to JupyterLab. Released a few weeks ago, this can be installed by getting the relevant extension, as well as xeus-python kernel (which notably comes without the magics well-known to ipykernel users):

jupyter labextension install @jupyterlab/debugger
conda install xeus-python -c conda-forge

This enables a visual debugging experience well-known from other IDEs.

Source: A visual debugger for Jupyter

Question 12

I use pandas to write to excel file in the following fashion:

import pandas

writer = pandas.ExcelWriter('Masterfile.xlsx') 

data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])

writer.save()

Masterfile.xlsx already consists of number of different tabs. However, it does not yet contain “Main”.

Pandas correctly writes to “Main” sheet, unfortunately it also deletes all other tabs.

Question 13

Pandas docs says it uses openpyxl for xlsx files. Quick look through the code in ExcelWriter gives a clue that something like this might work out:

import pandas
from openpyxl import load_workbook

book = load_workbook('Masterfile.xlsx')
writer = pandas.ExcelWriter('Masterfile.xlsx', engine='openpyxl') 
writer.book = book

## ExcelWriter for some reason uses writer.sheets to access the sheet.
## If you leave it empty it will not know that sheet Main is already there
## and will create a new sheet.

writer.sheets = dict((ws.title, ws) for ws in book.worksheets)

data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])

writer.save()

Question 14

Here is a helper function:

def append_df_to_excel(filename, df, sheet_name='Sheet1', startrow=None,
                       truncate_sheet=False, 
                       **to_excel_kwargs):
    """
    Append a DataFrame [df] to existing Excel file [filename]
    into [sheet_name] Sheet.
    If [filename] doesn't exist, then this function will create it.

    Parameters:
      filename : File path or existing ExcelWriter
                 (Example: '/path/to/file.xlsx')
      df : dataframe to save to workbook
      sheet_name : Name of sheet which will contain DataFrame.
                   (default: 'Sheet1')
      startrow : upper left cell row to dump data frame.
                 Per default (startrow=None) calculate the last row
                 in the existing DF and write to the next row...
      truncate_sheet : truncate (remove and recreate) [sheet_name]
                       before writing DataFrame to Excel file
      to_excel_kwargs : arguments which will be passed to `DataFrame.to_excel()`
                        [can be dictionary]

    Returns: None

    (c) [MaxU](https://stackoverflow.com/users/5741205/maxu?tab=profile)
    """
    from openpyxl import load_workbook

    # ignore [engine] parameter if it was passed
    if 'engine' in to_excel_kwargs:
        to_excel_kwargs.pop('engine')

    writer = pd.ExcelWriter(filename, engine='openpyxl')

    # Python 2.x: define [FileNotFoundError] exception if it doesn't exist 
    try:
        FileNotFoundError
    except NameError:
        FileNotFoundError = IOError


    try:
        # try to open an existing workbook
        writer.book = load_workbook(filename)
        
        # get the last row in the existing Excel sheet
        # if it was not specified explicitly
        if startrow is None and sheet_name in writer.book.sheetnames:
            startrow = writer.book[sheet_name].max_row

        # truncate sheet
        if truncate_sheet and sheet_name in writer.book.sheetnames:
            # index of [sheet_name] sheet
            idx = writer.book.sheetnames.index(sheet_name)
            # remove [sheet_name]
            writer.book.remove(writer.book.worksheets[idx])
            # create an empty sheet [sheet_name] using old index
            writer.book.create_sheet(sheet_name, idx)
        
        # copy existing sheets
        writer.sheets = {ws.title:ws for ws in writer.book.worksheets}
    except FileNotFoundError:
        # file does not exist yet, we will create it
        pass

    if startrow is None:
        startrow = 0

    # write out the new sheet
    df.to_excel(writer, sheet_name, startrow=startrow, **to_excel_kwargs)

    # save the workbook
    writer.save()

NOTE: for Pandas < 0.21.0, replace sheet_name with sheetname!

Usage examples:

append_df_to_excel('d:/temp/test.xlsx', df)

append_df_to_excel('d:/temp/test.xlsx', df, header=None, index=False)

append_df_to_excel('d:/temp/test.xlsx', df, sheet_name='Sheet2', index=False)

append_df_to_excel('d:/temp/test.xlsx', df, sheet_name='Sheet2', index=False, startrow=25)

Question 15

With openpyxlversion 2.4.0 and pandasversion 0.19.2, the process @ski came up with gets a bit simpler:

import pandas
from openpyxl import load_workbook

with pandas.ExcelWriter('Masterfile.xlsx', engine='openpyxl') as writer:
    writer.book = load_workbook('Masterfile.xlsx')
    data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])
#That's it!

Question 16

Starting in pandas 0.24 you can simplify this with the mode keyword argument of ExcelWriter:

import pandas as pd

with pd.ExcelWriter('the_file.xlsx', engine='openpyxl', mode='a') as writer: 
     data_filtered.to_excel(writer)

Question 17

Old question, but I am guessing some people still search for this – so…

I find this method nice because all worksheets are loaded into a dictionary of sheet name and dataframe pairs, created by pandas with the sheetname=None option. It is simple to add, delete or modify worksheets between reading the spreadsheet into the dict format and writing it back from the dict. For me the xlsxwriter works better than openpyxl for this particular task in terms of speed and format.

Note: future versions of pandas (0.21.0+) will change the “sheetname” parameter to “sheet_name”.

# read a single or multi-sheet excel file
# (returns dict of sheetname(s), dataframe(s))
ws_dict = pd.read_excel(excel_file_path,
                        sheetname=None)

# all worksheets are accessible as dataframes.

# easy to change a worksheet as a dataframe:
mod_df = ws_dict['existing_worksheet']

# do work on mod_df...then reassign
ws_dict['existing_worksheet'] = mod_df

# add a dataframe to the workbook as a new worksheet with
# ws name, df as dict key, value:
ws_dict['new_worksheet'] = some_other_dataframe

# when done, write dictionary back to excel...
# xlsxwriter honors datetime and date formats
# (only included as example)...
with pd.ExcelWriter(excel_file_path,
                    engine='xlsxwriter',
                    datetime_format='yyyy-mm-dd',
                    date_format='yyyy-mm-dd') as writer:

    for ws_name, df_sheet in ws_dict.items():
        df_sheet.to_excel(writer, sheet_name=ws_name)

For the example in the 2013 question:

ws_dict = pd.read_excel('Masterfile.xlsx',
                        sheetname=None)

ws_dict['Main'] = data_filtered[['Diff1', 'Diff2']]

with pd.ExcelWriter('Masterfile.xlsx',
                    engine='xlsxwriter') as writer:

    for ws_name, df_sheet in ws_dict.items():
        df_sheet.to_excel(writer, sheet_name=ws_name)

Question 18

I know this is an older thread, but this is the first item you find when searching, and the above solutions don’t work if you need to retain charts in a workbook that you already have created. In that case, xlwings is a better option – it allows you to write to the excel book and keeps the charts/chart data.

simple example:

import xlwings as xw
import pandas as pd

#create DF
months = ['2017-01','2017-02','2017-03','2017-04','2017-05','2017-06','2017-07','2017-08','2017-09','2017-10','2017-11','2017-12']
value1 = [x * 5+5 for x in range(len(months))]
df = pd.DataFrame(value1, index = months, columns = ['value1'])
df['value2'] = df['value1']+5
df['value3'] = df['value2']+5

#load workbook that has a chart in it
wb = xw.Book('C:\\data\\bookwithChart.xlsx')

ws = wb.sheets['chartData']

ws.range('A1').options(index=False).value = df

wb = xw.Book('C:\\data\\bookwithChart_updated.xlsx')

xw.apps[0].quit()

Question 19

There is a better solution in pandas 0.24:

with pd.ExcelWriter(path, mode='a') as writer:
    s.to_excel(writer, sheet_name='another sheet', index=False)

before:

after:

so upgrade your pandas now:

pip install --upgrade pandas

Question 20

def append_sheet_to_master(self, master_file_path, current_file_path, sheet_name):
    try:
        master_book = load_workbook(master_file_path)
        master_writer = pandas.ExcelWriter(master_file_path, engine='openpyxl')
        master_writer.book = master_book
        master_writer.sheets = dict((ws.title, ws) for ws in master_book.worksheets)
        current_frames = pandas.ExcelFile(current_file_path).parse(pandas.ExcelFile(current_file_path).sheet_names[0],
                                                               header=None,
                                                               index_col=None)
        current_frames.to_excel(master_writer, sheet_name, index=None, header=False)

        master_writer.save()
    except Exception as e:
        raise e

This works perfectly fine only thing is that formatting of the master file(file to which we add new sheet) is lost.

Question 21

writer = pd.ExcelWriter('prueba1.xlsx'engine='openpyxl',keep_date_col=True)

The “keep_date_col” hope help you

Question 22

book = load_workbook(xlsFilename)
writer = pd.ExcelWriter(self.xlsFilename)
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)
df.to_excel(writer, sheet_name=sheetName, index=False)
writer.save()

Question 23

Why python 2.7 doesn’t include Z character (Zulu or zero offset) at the end of UTC datetime object’s isoformat string unlike JavaScript?

>>> datetime.datetime.utcnow().isoformat()
'2013-10-29T09:14:03.895210'

Whereas in javascript

>>>  console.log(new Date().toISOString()); 
2013-10-29T09:38:41.341Z

Question 24

Python datetime objects don’t have time zone info by default, and without it, Python actually violates the ISO 8601 specification (if no time zone info is given, assumed to be local time). You can use the pytz package to get some default time zones, or directly subclass tzinfo yourself:

from datetime import datetime, tzinfo, timedelta
class simple_utc(tzinfo):
    def tzname(self,**kwargs):
        return "UTC"
    def utcoffset(self, dt):
        return timedelta(0)

Then you can manually add the time zone info to utcnow():

>>> datetime.utcnow().replace(tzinfo=simple_utc()).isoformat()
'2014-05-16T22:51:53.015001+00:00'

Note that this DOES conform to the ISO 8601 format, which allows for either Z or +00:00 as the suffix for UTC. Note that the latter actually conforms to the standard better, with how time zones are represented in general (UTC is a special case.)

Question 25

Option: `isoformat()`

Python’s datetime does not support the military timezone suffixes like ‘Z’ suffix for UTC. The following simple string replacement does the trick:

In [1]: import datetime

In [2]: d = datetime.datetime(2014, 12, 10, 12, 0, 0)

In [3]: str(d).replace('+00:00', 'Z')
Out[3]: '2014-12-10 12:00:00Z'

str(d) is essentially the same as d.isoformat(sep=' ')

See: Datetime, Python Standard Library

Option: `strftime()`

Or you could use strftime to achieve the same effect:

In [4]: d.strftime('%Y-%m-%d %H:%M:%SZ')
Out[4]: '2014-12-10 12:00:00Z'

Note: This option works only when you know the date specified is in UTC.

See: datetime.strftime()

Additional: Human Readable Timezone

Going further, you may be interested in displaying human readable timezone information, pytz with strftime %Z timezone flag:

In [5]: import pytz

In [6]: d = datetime.datetime(2014, 12, 10, 12, 0, 0, tzinfo=pytz.utc)

In [7]: d
Out[7]: datetime.datetime(2014, 12, 10, 12, 0, tzinfo=<UTC>)

In [8]: d.strftime('%Y-%m-%d %H:%M:%S %Z')
Out[8]: '2014-12-10 12:00:00 UTC'

Question 26

The following javascript and python scripts give identical outputs. I think it’s what you are looking for.

JavaScript

new Date().toISOString()

Python

from datetime import datetime

datetime.utcnow().isoformat()[:-3]+'Z'

The output they give is the utc (zelda) time formatted as an ISO string with a 3 millisecond significant digit and appended with a Z.

2019-01-19T23:20:25.459Z

Question 27

In Python >= 3.2 you can simply use this:

>>> from datetime import datetime, timezone
>>> datetime.now(timezone.utc).isoformat()
'2019-03-14T07:55:36.979511+00:00'

Question 28

Python datetimes are a little clunky. Use arrow.

> str(arrow.utcnow())
'2014-05-17T01:18:47.944126+00:00'

Arrow has essentially the same api as datetime, but with timezones and some extra niceties that should be in the main library.

A format compatible with Javascript can be achieved by:

arrow.utcnow().isoformat().replace("+00:00", "Z")
'2018-11-30T02:46:40.714281Z'

Javascript Date.parse will quietly drop microseconds from the timestamp.

Question 29

There are a lot of good answers on the post, but I wanted the format to come out exactly as it does with JavaScript. This is what I’m using and it works well.

In [1]: import datetime

In [1]: now = datetime.datetime.utcnow()

In [1]: now.strftime('%Y-%m-%dT%H:%M:%S') + now.strftime('.%f')[:4] + 'Z'
Out[3]: '2018-10-16T13:18:34.856Z'

Question 30

pip install python-dateutil

>>> a = "2019-06-27T02:14:49.443814497Z"
>>> dateutil.parser.parse(a)
datetime.datetime(2019, 6, 27, 2, 14, 49, 443814, tzinfo=tzutc())

Question 31

Pass a UTC timezone object to datetime.now() instead of using datetime.utcnow():

from datetime import datetime, timezone

datetime.now(timezone.utc)
>>> datetime.datetime(2020, 1, 8, 6, 6, 24, 260810, tzinfo=datetime.timezone.utc)

datetime.now(timezone.utc).isoformat()
>>> '2020-01-08T06:07:04.492045+00:00'

That looks good, so let’s see what Django and dateutil think:

from django.utils.timezone import is_aware

is_aware(datetime.now(timezone.utc))
>>> True

import dateutil.parser

is_aware(dateutil.parser.parse(datetime.now(timezone.utc).isoformat()))
>>> True

Okay, the Python datetime object and the ISO 8601 string are UTC “aware”. Now let’s look at what JavaScript thinks of the datetime string. Borrowing from this answer we get:

let date= '2020-01-08T06:07:04.492045+00:00';
const dateParsed = new Date(Date.parse(date))

document.write(dateParsed);
document.write("\n");
// Tue Jan 07 2020 22:07:04 GMT-0800 (Pacific Standard Time)

document.write(dateParsed.toISOString());
document.write("\n");
// 2020-01-08T06:07:04.492Z

document.write(dateParsed.toUTCString());
document.write("\n");
// Wed, 08 Jan 2020 06:07:04 GMT

Notes:

I approached this problem with a few goals:

generate a UTC “aware” datetime string in ISO 8601 format
use only Python Standard Library functions for datetime object and string creation
validate the datetime object and string with the Django timezone utility function and the dateutil parser
use JavaScript functions to validate that the ISO 8601 datetime string is UTC aware

Note that this approach does not include a Z suffix and does not use utcnow(). But it’s based on the recommendation in the Python documentation and it passes muster with both Django and JavaScript.

You may also want to read this blog post.

Question 32

By combining all answers above I came with following function :

from datetime import datetime, tzinfo, timedelta
class simple_utc(tzinfo):
    def tzname(self,**kwargs):
        return "UTC"
    def utcoffset(self, dt):
        return timedelta(0)


def getdata(yy, mm, dd, h, m, s) :
    d = datetime(yy, mm, dd, h, m, s)
    d = d.replace(tzinfo=simple_utc()).isoformat()
    d = str(d).replace('+00:00', 'Z')
    return d


print getdata(2018, 02, 03, 15, 0, 14)

Question 33

I use pendulum:

import pendulum


d = pendulum.now("UTC").to_iso8601_string()
print(d)

>>> 2019-10-30T00:11:21.818265Z

Question 34

>>> import arrow

>>> now = arrow.utcnow().format('YYYY-MM-DDTHH:mm:ss.SSS')
>>> now
'2018-11-28T21:34:59.235'
>>> zulu = "{}Z".format(now)
>>> zulu
'2018-11-28T21:34:59.235Z'

Or, to get it in one fell swoop:

>>> zulu = "{}Z".format(arrow.utcnow().format('YYYY-MM-DDTHH:mm:ss.SSS'))
>>> zulu
'2018-11-28T21:54:49.639Z'

Question 35

I’m really confused. I tried to encode but the error said can't decode....

>>> "你好".encode("utf8")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)

I know how to avoid the error with “u” prefix on the string. I’m just wondering why the error is “can’t decode” when encode was called. What is Python doing under the hood?

Question 36

"你好".encode('utf-8')

encode converts a unicode object to a string object. But here you have invoked it on a string object (because you don’t have the u). So python has to convert the string to a unicode object first. So it does the equivalent of

"你好".decode().encode('utf-8')

But the decode fails because the string isn’t valid ascii. That’s why you get a complaint about not being able to decode.

Question 37

Always encode from unicode to bytes.
In this direction, you get to choose the encoding.

>>> u"你好".encode("utf8")
'\xe4\xbd\xa0\xe5\xa5\xbd'
>>> print _
你好

The other way is to decode from bytes to unicode.
In this direction, you have to know what the encoding is.

>>> bytes = '\xe4\xbd\xa0\xe5\xa5\xbd'
>>> print bytes
你好
>>> bytes.decode('utf-8')
u'\u4f60\u597d'
>>> print _
你好

This point can’t be stressed enough. If you want to avoid playing unicode “whack-a-mole”, it’s important to understand what’s happening at the data level. Here it is explained another way:

A unicode object is decoded already, you never want to call decode on it.
A bytestring object is encoded already, you never want to call encode on it.

Now, on seeing .encode on a byte string, Python 2 first tries to implicitly convert it to text (a unicode object). Similarly, on seeing .decode on a unicode string, Python 2 implicitly tries to convert it to bytes (a str object).

These implicit conversions are why you can get UnicodeDecodeError when you’ve called encode. It’s because encoding usually accepts a parameter of type unicode; when receiving a str parameter, there’s an implicit decoding into an object of type unicode before re-encoding it with another encoding. This conversion chooses a default ‘ascii’ decoder^†, giving you the decoding error inside an encoder.

In fact, in Python 3 the methods str.decode and bytes.encode don’t even exist. Their removal was a [controversial] attempt to avoid this common confusion.

^† _{…or whatever coding sys.getdefaultencoding() mentions; usually this is ‘ascii’}

Question 38

You can try this

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

Or

You can also try following

Add following line at top of your .py file.

# -*- coding: utf-8 -*-

Question 39

If you’re using Python < 3, you’ll need to tell the interpreter that your string literal is Unicode by prefixing it with a u:

Python 2.7.2 (default, Jan 14 2012, 23:14:09) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> "你好".encode("utf8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
>>> u"你好".encode("utf8")
'\xe4\xbd\xa0\xe5\xa5\xbd'

Further reading: Unicode HOWTO.

Question 40

You use u"你好".encode('utf8') to encode an unicode string. But if you want to represent "你好", you should decode it. Just like:

"你好".decode("utf8")

You will get what you want. Maybe you should learn more about encode & decode.

Question 41

In case you’re dealing with Unicode, sometimes instead of encode('utf-8'), you can also try to ignore the special characters, e.g.

"你好".encode('ascii','ignore')

or as something.decode('unicode_escape').encode('ascii','ignore') as suggested here.

Not particularly useful in this example, but can work better in other scenarios when it’s not possible to convert some special characters.

Alternatively you can consider replacing particular character using replace().

Question 42

If you are starting the python interpreter from a shell on Linux or similar systems (BSD, not sure about Mac), you should also check the default encoding for the shell.

Call locale charmap from the shell (not the python interpreter) and you should see

[user@host dir] $ locale charmap
UTF-8
[user@host dir] $

If this is not the case, and you see something else, e.g.

[user@host dir] $ locale charmap
ANSI_X3.4-1968
[user@host dir] $

Python will (at least in some cases such as in mine) inherit the shell’s encoding and will not be able to print (some? all?) unicode characters. Python’s own default encoding that you see and control via sys.getdefaultencoding() and sys.setdefaultencoding() is in this case ignored.

If you find that you have this problem, you can fix that by

[user@host dir] $ export LC_CTYPE="en_EN.UTF-8"
[user@host dir] $ locale charmap
UTF-8
[user@host dir] $

(Or alternatively choose whichever keymap you want instead of en_EN.) You can also edit /etc/locale.conf (or whichever file governs the locale definition in your system) to correct this.

Question 43

I have a Macbook with OS X El Captain. I think that Python 2.7 comes preinstalled on it. However, I installed Python 3.5 too. When I started using Python 3, I read that if I want to install a package, I should type:

pip3 install some_package

Anyway, now when I use

pip install some_package

I get some_package installed for Python 3. I mean I can import it and use it without problems. Moreover, when I type just pip3 in Terminal, I got this message about the usage:

Usage:   
  pip <command> [options]

which is the same message I get when I type just pip.

Does it mean that in previos versions, things were different, and now pip and pip3 can be used interchangeably? If so, and for the sake of argument, how can I install packages for Python 2 instead of Python 3?

Question 44

Your pip is a soft link to the same executable file path with pip3. you can use the commands below to check where your pip and pip3 real paths are:

$ ls -l `which pip`
$ ls -l `which pip3`

You may also use the commands below to know more details:

$ pip show pip
$ pip3 show pip

When we install different versions of python, we may create such soft links to

set default pip to some version.
make different links for different versions.

It is the same situation with python, python2, python3

More information below if you’re interested in how it happens in different cases:

Question 45

If you had python 2.x and then installed python3, your pip will be pointing to pip3. you can verify that by typing pip --version which would be the same as pip3 --version.

On your system you have now pip, pip2 and pip3.

If you want you can change pip to point to pip2 instead of pip3.

Question 46

When you install python3, pip3 gets installed. And if you don’t have another python installation(like python2.7) then a link is created which points pip to pip3.

So pip is a link to to pip3 if there is no other version of python installed(other than python3). pip generally points to the first installation.

Question 47

This is a tricky subject. In the end, if you invoke pip it will invoke either pip2 or pip3, depending on how you set your system up.

Question 48

I think pip, pip2 and pip3 are not soft links to the same executable file path. Note these commands and results in my Linux terminal:

mrz@mrz-pc ~ $ ls -l `which pip`
-rwxr-xr-x 1 root root 292 Nov 10  2016 /usr/bin/pip
mrz@mrz-pc ~ $ ls -l `which pip2`
-rwxr-xr-x 1 root root 283 Nov 10  2016 /usr/bin/pip2
mrz@mrz-pc ~ $ ls -l `which pip3`
-rwxr-xr-x 1 root root 293 Nov 10  2016 /usr/bin/pip3
mrz@mrz-pc ~ $ pip -V
pip 9.0.1 from /home/mrz/.local/lib/python2.7/site-packages (python 2.7)
mrz@mrz-pc ~ $ pip2 -V
pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)
mrz@mrz-pc ~ $ pip3 -V
pip 9.0.1 from /home/mrz/.local/lib/python3.5/site-packages (python 3.5)

As you see they exist in different paths.

pip3 always operates on the Python3 environment only, as pip2 does with Python2. pip operates in whichever environment is appropriate to the context. For example, if you are in a Python3 venv, pip will operate on the Python3 environment.

Question 49

By illustration:

pip --version
  pip 19.0.3 from /usr/lib/python3.7/site-packages/pip (python 3.7)

pip3 --version
  pip 19.0.3 from /usr/lib/python3.7/site-packages/pip (python 3.7)

python --version
  Python 3.7.3

which python
  /usr/bin/python

ls -l '/usr/bin/python'
  lrwxrwxrwx 1 root root 7 Mar 26 14:43 /usr/bin/python -> python3

which python3
  /usr/bin/python3

ls -l /usr/bin/python3
  lrwxrwxrwx 1 root root 9 Mar 26 14:43 /usr/bin/python3 -> python3.7

ls -l /usr/bin/python3.7
  -rwxr-xr-x 2 root root 14120 Mar 26 14:43 /usr/bin/python3.7

Thus, my in my default system python (Python 3.7.3), pip is pip3.

Question 50

If you installed Python 2.7, I think you could use pip2 and pip2.7 to install packages specifically for Python 2, like

pip2 install some_pacakge

or

pip2.7 install some_package

And you may use pip3 or pip3.5 to install pacakges specifically for Python 3.

Question 51

On my Windows instance – and I do not fully understand my environment – using pip3 to install the kaggle-cli package worked – whereas pip did not. I was working in a conda environment and the environments appear to be different.

(fastai) C:\Users\redact\Downloads\fast.ai\deeplearning1\nbs>pip –version

pip 9.0.1 from C:\ProgramData\Anaconda3\envs\fastai\lib\site-packages (python 3.6)

(fastai) C:\Users\redact\Downloads\fast.ai\deeplearning1\nbs>pip3 –version

pip 9.0.1 from c:\users\redact\appdata\local\programs\python\python36\lib\site-packages (python 3.6)

Question 52

Given an activated Python 3.6 virtualenv in somepath/venv, the following aliases resolved the various issues on a macOS Sierra where pip insisted on pointing to Apple’s 2.7 Python.

alias pip='python somepath/venv/lib/python3.6/site-packages/pip/__main__.py'

This didn’t work so well when I had to do sudo pip as the root user doesn’t know anything about my alias or the virtualenv, so I had to add an extra alias to handle this as well. It’s a hack, but it works, and I know what it does:

alias sudopip='sudo somepath/venv/bin/python somepath/venv/lib/python3.6/site-packages/pip/__main__.py'

background:

pip3 did not exist to start (command not found) with and which pip would return /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip, the Apple Python.

Python 3.6 was installed via macports.

After activation of the 3.6 virtualenv I wanted to work with, which python would return somepath/venv/bin/python

Somehow pip install would do the right thing and hit my virtualenv, but pip list would rattle off Python 2.7 packages.

For Python, this is batting way beneath my expectations in terms of beginner-friendliness.

Question 53

I have a function (foo) which calls another function (bar). If invoking bar() raises an HttpError, I want to handle it specially if the status code is 404, otherwise re-raise.

I am trying to write some unit tests around this foo function, mocking out the call to bar(). Unfortunately, I am unable to get the mocked call to bar() to raise an Exception which is caught by my except block.

Here is my code which illustrates my problem:

import unittest
import mock
from apiclient.errors import HttpError


class FooTests(unittest.TestCase):
    @mock.patch('my_tests.bar')
    def test_foo_shouldReturnResultOfBar_whenBarSucceeds(self, barMock):
        barMock.return_value = True
        result = foo()
        self.assertTrue(result)  # passes

    @mock.patch('my_tests.bar')
    def test_foo_shouldReturnNone_whenBarRaiseHttpError404(self, barMock):
        barMock.side_effect = HttpError(mock.Mock(return_value={'status': 404}), 'not found')
        result = foo()
        self.assertIsNone(result)  # fails, test raises HttpError

    @mock.patch('my_tests.bar')
    def test_foo_shouldRaiseHttpError_whenBarRaiseHttpErrorNot404(self, barMock):
        barMock.side_effect = HttpError(mock.Mock(return_value={'status': 500}), 'error')
        with self.assertRaises(HttpError):  # passes
            foo()

def foo():
    try:
        result = bar()
        return result
    except HttpError as error:
        if error.resp.status == 404:
            print '404 - %s' % error.message
            return None
        raise

def bar():
    raise NotImplementedError()

I followed the Mock docs which say that you should set the side_effect of a Mock instance to an Exception class to have the mocked function raise the error.

I also looked at some other related StackOverflow Q&As, and it looks like I am doing the same thing they are doing to cause and Exception to be raised by their mock.

Why is setting the side_effect of barMock not causing the expected Exception to be raised? If I am doing something weird, how should I go about testing logic in my except block?

Question 54

Your mock is raising the exception just fine, but the error.resp.status value is missing. Rather than use return_value, just tell Mock that status is an attribute:

barMock.side_effect = HttpError(mock.Mock(status=404), 'not found')

Additional keyword arguments to Mock() are set as attributes on the resulting object.

I put your foo and bar definitions in a my_tests module, added in the HttpError class so I could use it too, and your test then can be ran to success:

>>> from my_tests import foo, HttpError
>>> import mock
>>> with mock.patch('my_tests.bar') as barMock:
...     barMock.side_effect = HttpError(mock.Mock(status=404), 'not found')
...     result = my_test.foo()
... 
404 - 
>>> result is None
True

You can even see the print '404 - %s' % error.message line run, but I think you wanted to use error.content there instead; that’s the attribute HttpError() sets from the second argument, at any rate.

Question 55

What’s the most pythonic way to mesh two strings together?

For example:

Input:

u = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
l = 'abcdefghijklmnopqrstuvwxyz'

Output:

'AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz'

Question 56

For me, the most pythonic* way is the following which pretty much does the same thing but uses the + operator for concatenating the individual characters in each string:

res = "".join(i + j for i, j in zip(u, l))
print(res)
# 'AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz'

It is also faster than using two join() calls:

In [5]: l1 = 'A' * 1000000; l2 = 'a' * 1000000

In [6]: %timeit "".join("".join(item) for item in zip(l1, l2))
1 loops, best of 3: 442 ms per loop

In [7]: %timeit "".join(i + j for i, j in zip(l1, l2))
1 loops, best of 3: 360 ms per loop

Faster approaches exist, but they often obfuscate the code.

Note: If the two input strings are not the same length then the longer one will be truncated as zip stops iterating at the end of the shorter string. In this case instead of zip one should use zip_longest (izip_longest in Python 2) from the itertools module to ensure that both strings are fully exhausted.

_{*To take a quote from the Zen of Python: Readability counts.

Pythonic = readability for me; i + j is just visually parsed more easily, at least for my eyes.}

Question 57

Faster Alternative

Another way:

res = [''] * len(u) * 2
res[::2] = u
res[1::2] = l
print(''.join(res))

Output:

'AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz'

Speed

Looks like it is faster:

%%timeit
res = [''] * len(u) * 2
res[::2] = u
res[1::2] = l
''.join(res)

100000 loops, best of 3: 4.75 µs per loop

than the fastest solution so far:

%timeit "".join(list(chain.from_iterable(zip(u, l))))

100000 loops, best of 3: 6.52 µs per loop

Also for the larger strings:

l1 = 'A' * 1000000; l2 = 'a' * 1000000

%timeit "".join(list(chain.from_iterable(zip(l1, l2))))
1 loops, best of 3: 151 ms per loop


%%timeit
res = [''] * len(l1) * 2
res[::2] = l1
res[1::2] = l2
''.join(res)

10 loops, best of 3: 92 ms per loop

Python 3.5.1.

Variation for strings with different lengths

u = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
l = 'abcdefghijkl'

Shorter one determines length (`zip()` equivalent)

min_len = min(len(u), len(l))
res = [''] * min_len * 2 
res[::2] = u[:min_len]
res[1::2] = l[:min_len]
print(''.join(res))

Output:

AaBbCcDdEeFfGgHhIiJjKkLl

Longer one determines length (`itertools.zip_longest(fillvalue='')` equivalent)

min_len = min(len(u), len(l))
res = [''] * min_len * 2 
res[::2] = u[:min_len]
res[1::2] = l[:min_len]
res += u[min_len:] + l[min_len:]
print(''.join(res))

Output:

AaBbCcDdEeFfGgHhIiJjKkLlMNOPQRSTUVWXYZ

Question 58

With join() and zip().

>>> ''.join(''.join(item) for item in zip(u,l))
'AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz'

Question 59

On Python 2, by far the faster way to do things, at ~3x the speed of list slicing for small strings and ~30x for long ones, is

res = bytearray(len(u) * 2)
res[::2] = u
res[1::2] = l
str(res)

This wouldn’t work on Python 3, though. You could implement something like

res = bytearray(len(u) * 2)
res[::2] = u.encode("ascii")
res[1::2] = l.encode("ascii")
res.decode("ascii")

but by then you’ve already lost the gains over list slicing for small strings (it’s still 20x the speed for long strings) and this doesn’t even work for non-ASCII characters yet.

FWIW, if you are doing this on massive strings and need every cycle, and for some reason have to use Python strings… here’s how to do it:

res = bytearray(len(u) * 4 * 2)

u_utf32 = u.encode("utf_32_be")
res[0::8] = u_utf32[0::4]
res[1::8] = u_utf32[1::4]
res[2::8] = u_utf32[2::4]
res[3::8] = u_utf32[3::4]

l_utf32 = l.encode("utf_32_be")
res[4::8] = l_utf32[0::4]
res[5::8] = l_utf32[1::4]
res[6::8] = l_utf32[2::4]
res[7::8] = l_utf32[3::4]

res.decode("utf_32_be")

Special-casing the common case of smaller types will help too. FWIW, this is only 3x the speed of list slicing for long strings and a factor of 4 to 5 slower for small strings.

Either way I prefer the join solutions, but since timings were mentioned elsewhere I thought I might as well join in.

Question 60

If you want the fastest way, you can combine itertools with operator.add:

In [36]: from operator import add

In [37]: from itertools import  starmap, izip

In [38]: timeit "".join([i + j for i, j in uzip(l1, l2)])
1 loops, best of 3: 142 ms per loop

In [39]: timeit "".join(starmap(add, izip(l1,l2)))
1 loops, best of 3: 117 ms per loop

In [40]: timeit "".join(["".join(item) for item in zip(l1, l2)])
1 loops, best of 3: 196 ms per loop

In [41]:  "".join(starmap(add, izip(l1,l2))) ==  "".join([i + j   for i, j in izip(l1, l2)]) ==  "".join(["".join(item) for item in izip(l1, l2)])
Out[42]: True

But combining izip and chain.from_iterable is faster again

In [2]: from itertools import  chain, izip

In [3]: timeit "".join(chain.from_iterable(izip(l1, l2)))
10 loops, best of 3: 98.7 ms per loop

There is also a substantial difference between chain(* and chain.from_iterable(....

In [5]: timeit "".join(chain(*izip(l1, l2)))
1 loops, best of 3: 212 ms per loop

There is no such thing as a generator with join, passing one is always going to be slower as python will first build a list using the content because it does two passes over the data, one to figure out the size needed and one to actually do the join which would not be possible using a generator:

join.h:

 /* Here is the general case.  Do a pre-pass to figure out the total
  * amount of space we'll need (sz), and see whether all arguments are
  * bytes-like.
   */

Also if you have different length strings and you don’t want to lose data you can use izip_longest :

In [22]: from itertools import izip_longest    
In [23]: a,b = "hlo","elworld"

In [24]:  "".join(chain.from_iterable(izip_longest(a, b,fillvalue="")))
Out[24]: 'helloworld'

For python 3 it is called zip_longest

But for python2, veedrac’s suggestion is by far the fastest:

In [18]: %%timeit
res = bytearray(len(u) * 2)
res[::2] = u
res[1::2] = l
str(res)
   ....: 
100 loops, best of 3: 2.68 ms per loop

Question 61

You could also do this using map and operator.add:

from operator import add

u = 'AAAAA'
l = 'aaaaa'

s = "".join(map(add, u, l))

Output:

'AaAaAaAaAa'

What map does is it takes every element from the first iterable u and the first elements from the second iterable l and applies the function supplied as the first argument add. Then join just joins them.

Question 62

Jim’s answer is great, but here’s my favorite option, if you don’t mind a couple of imports:

from functools import reduce
from operator import add

reduce(add, map(add, u, l))

Question 63

A lot of these suggestions assume the strings are of equal length. Maybe that covers all reasonable use cases, but at least to me it seems that you might want to accomodate strings of differing lengths too. Or am I the only one thinking the mesh should work a bit like this:

u = "foobar"
l = "baz"
mesh(u,l) = "fboaozbar"

One way to do this would be the following:

def mesh(a,b):
    minlen = min(len(a),len(b))
    return "".join(["".join(x+y for x,y in zip(a,b)),a[minlen:],b[minlen:]])

Question 64

I like using two fors, the variable names can give a hint/reminder to what is going on:

"".join(char for pair in zip(u,l) for char in pair)

问题：在iPython Notebook中进行调试的正确方法是什么？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

问题：如何在不覆盖数据的情况下（使用熊猫）写入现有的excel文件？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

问题：Python UTC日期时间对象的ISO格式不包含Z（Zulu或零偏移）

回答 0

回答 1

选项： isoformat()

选项： strftime()

附加：人类可读的时区

Option: isoformat()

Option: strftime()

Additional: Human Readable Timezone

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

问题：Python-‘ASCII’编解码器无法解码字节

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：pip或pip3为Python 3安装软件包？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

背景：

background:

问题：模拟函数引发异常以测试except块

回答 0

问题：插入两个字符串的最pythonic方式

回答 0

回答 1

更快的选择

速度

不同长度字符串的变化

较短的一个确定长度（zip()等效）

更长的长度决定长度（itertools.zip_longest(fillvalue='')等效）

Faster Alternative

Speed

Variation for strings with different lengths

Shorter one determines length (zip() equivalent)

Longer one determines length (itertools.zip_longest(fillvalue='') equivalent)

回答 2

回答 3

回答 4

回答 5

回答 6

选项： `isoformat()`

选项： `strftime()`

Option: `isoformat()`

Option: `strftime()`

较短的一个确定长度（`zip()`等效）

更长的长度决定长度（`itertools.zip_longest(fillvalue='')`等效）

Shorter one determines length (`zip()` equivalent)

Longer one determines length (`itertools.zip_longest(fillvalue='')` equivalent)