It seems they canceled in Python 3 all the easy way to quickly load a script by removing execfile()

Is there an obvious alternative I’m missing?

According to the documentation, instead of

execfile("./filename") 

Use

exec(open("./filename").read())

See:

• What’s New In Python 3.0

You are just supposed to read the file and exec the code yourself. 2to3 current replaces

execfile("somefile.py", global_vars, local_vars)

as

with open("somefile.py") as f:
    code = compile(f.read(), "somefile.py", 'exec')
    exec(code, global_vars, local_vars)

(The compile call isn’t strictly needed, but it associates the filename with the code object making debugging a little easier.)

See:

• http://docs.python.org/release/2.7.3/library/functions.html#execfile
• http://docs.python.org/release/3.2.3/library/functions.html#compile
• http://docs.python.org/release/3.2.3/library/functions.html#exec

While exec(open("filename").read()) is often given as an alternative to execfile("filename"), it misses important details that execfile supported.

The following function for Python3.x is as close as I could get to having the same behavior as executing a file directly. That matches running python /path/to/somefile.py.

def execfile(filepath, globals=None, locals=None):
    if globals is None:
        globals = {}
    globals.update({
        "__file__": filepath,
        "__name__": "__main__",
    })
    with open(filepath, 'rb') as file:
        exec(compile(file.read(), filepath, 'exec'), globals, locals)

# execute the file
execfile("/path/to/somefile.py")

Notes:

• Uses binary reading to avoid encoding issues
• Guaranteed to close the file (Python3.x warns about this)
• Defines __main__, some scripts depend on this to check if they are loading as a module or not for eg. if __name__ == "__main__"
• Setting __file__ is nicer for exception messages and some scripts use __file__ to get the paths of other files relative to them.

• Takes optional globals & locals arguments, modifying them in-place as execfile does – so you can access any variables defined by reading back the variables after running.

• Unlike Python2’s execfile this does not modify the current namespace by default. For that you have to explicitly pass in globals() & locals().

As suggested on the python-dev mailinglist recently, the runpy module might be a viable alternative. Quoting from that message:

https://docs.python.org/3/library/runpy.html#runpy.run_path

import runpy
file_globals = runpy.run_path("file.py")

There are subtle differences to execfile:

• run_path always creates a new namespace. It executes the code as a module, so there is no difference between globals and locals (which is why there is only a init_globals argument). The globals are returned.

  execfile executed in the current namespace or the given namespace. The semantics of locals and globals, if given, were similar to locals and globals inside a class definition.

• run_path can not only execute files, but also eggs and directories (refer to its documentation for details).

This one is better, since it takes the globals and locals from the caller:

import sys
def execfile(filename, globals=None, locals=None):
    if globals is None:
        globals = sys._getframe(1).f_globals
    if locals is None:
        locals = sys._getframe(1).f_locals
    with open(filename, "r") as fh:
        exec(fh.read()+"\n", globals, locals)

You could write your own function:

def xfile(afile, globalz=None, localz=None):
    with open(afile, "r") as fh:
        exec(fh.read(), globalz, localz)

If you really needed to…

If the script you want to load is in the same directory than the one you run, maybe “import” will do the job ?

If you need to dynamically import code the built-in function __ import__ and the module imp are worth looking at.

>>> import sys
>>> sys.path = ['/path/to/script'] + sys.path
>>> __import__('test')
<module 'test' from '/path/to/script/test.pyc'>
>>> __import__('test').run()
'Hello world!'

test.py:

def run():
        return "Hello world!"

If you’re using Python 3.1 or later, you should also take a look at importlib.

Here’s what I had (file is already assigned to the path to the file with the source code in both examples):

execfile(file)

Here’s what I replaced it with:

exec(compile(open(file).read(), file, 'exec'))

My favorite part: the second version works just fine in both Python 2 and 3, meaning it’s not necessary to add in version dependent logic.

Note that the above pattern will fail if you’re using PEP-263 encoding declarations that aren’t ascii or utf-8. You need to find the encoding of the data, and encode it correctly before handing it to exec().

class python3Execfile(object):
    def _get_file_encoding(self, filename):
        with open(filename, 'rb') as fp:
            try:
                return tokenize.detect_encoding(fp.readline)[0]
            except SyntaxError:
                return "utf-8"

    def my_execfile(filename):
        globals['__file__'] = filename
        with open(filename, 'r', encoding=self._get_file_encoding(filename)) as fp:
            contents = fp.read()
        if not contents.endswith("\n"):
            # http://bugs.python.org/issue10204
            contents += "\n"
        exec(contents, globals, globals)

Also, while not a pure Python solution, if you’re using IPython (as you probably should anyway), you can do:

%run /path/to/filename.py

Which is equally easy.

I’m just a newbie here so maybe it’s pure luck if I found this :

After trying to run a script from the interpreter prompt >>> with the command

    execfile('filename.py')

for which I got a “NameError: name ‘execfile’ is not defined” I tried a very basic

    import filename

it worked well :-)

I hope this can be helpful and thank you all for the great hints, examples and all those masterly commented pieces of code that are a great inspiration for newcomers !

I use Ubuntu 16.014 LTS x64. Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609] on linux

To me, the cleanest approach is to use importlib and import the file as a module by path, like so:

from importlib import util

def load_file(name, path):
    spec = util.spec_from_file_location(name, path)
    module = util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

Usage example

Let’s have a file foo.py:

print('i got imported')
def hello():
    print('hello from foo')

Now just import and use it like a normal module:

>>> foo = load_file('foo', './foo.py')
i got imported
>>> foo.hello()
hello from foo

I prefer this technique over direct approaches like exec(open(...)) because it does not clutter your namespaces or unnecessarily messes with $PATH.

I would like to make a deep copy of a dict in python. Unfortunately the .deepcopy() method doesn’t exist for the dict. How do I do that?

>>> my_dict = {'a': [1, 2, 3], 'b': [4, 5, 6]}
>>> my_copy = my_dict.deepcopy()
Traceback (most recent calll last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'deepcopy'
>>> my_copy = my_dict.copy()
>>> my_dict['a'][2] = 7
>>> my_copy['a'][2]
7

The last line should be 3.

I would like that modifications in my_dict don’t impact the snapshot my_copy.

How do I do that? The solution should be compatible with Python 3.x.

How about:

import copy
d = { ... }
d2 = copy.deepcopy(d)

Python 2 or 3:

Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import copy
>>> my_dict = {'a': [1, 2, 3], 'b': [4, 5, 6]}
>>> my_copy = copy.deepcopy(my_dict)
>>> my_dict['a'][2] = 7
>>> my_copy['a'][2]
3
>>>

dict.copy() is a shallow copy function for dictionary
id is built-in function that gives you the address of variable

First you need to understand “why is this particular problem is happening?”

In [1]: my_dict = {'a': [1, 2, 3], 'b': [4, 5, 6]}

In [2]: my_copy = my_dict.copy()

In [3]: id(my_dict)
Out[3]: 140190444167808

In [4]: id(my_copy)
Out[4]: 140190444170328

In [5]: id(my_copy['a'])
Out[5]: 140190444024104

In [6]: id(my_dict['a'])
Out[6]: 140190444024104

The address of the list present in both the dicts for key ‘a’ is pointing to same location.
Therefore when you change value of the list in my_dict, the list in my_copy changes as well.

Solution for data structure mentioned in the question:

In [7]: my_copy = {key: value[:] for key, value in my_dict.items()}

In [8]: id(my_copy['a'])
Out[8]: 140190444024176

Or you can use deepcopy as mentioned above.

Python 3.x

from copy import deepcopy

my_dict = {'one': 1, 'two': 2}
new_dict_deepcopy = deepcopy(my_dict)

Without deepcopy, I am unable to remove the hostname dictionary from within my domain dictionary.

Without deepcopy I get the following error:

"RuntimeError: dictionary changed size during iteration"

…when I try to remove the desired element from my dictionary inside of another dictionary.

import socket
import xml.etree.ElementTree as ET
from copy import deepcopy

domain is a dictionary object

def remove_hostname(domain, hostname):
    domain_copy = deepcopy(domain)
    for domains, hosts in domain_copy.items():
        for host, port in hosts.items():
           if host == hostname:
                del domain[domains][host]
    return domain

Example output: [orginal]domains = {‘localdomain’: {‘localhost’: {‘all’: ‘4000’}}}

[new]domains = {‘localdomain’: {} }}

So what’s going on here is I am iterating over a copy of a dictionary rather than iterating over the dictionary itself. With this method, you are able to remove elements as needed.

I like and learned a lot from Lasse V. Karlsen. I modified it into the following example, which highlights pretty well the difference between shallow dictionary copies and deep copies:

    import copy

    my_dict = {'a': [1, 2, 3], 'b': [4, 5, 6]}
    my_copy = copy.copy(my_dict)
    my_deepcopy = copy.deepcopy(my_dict)

Now if you change

    my_dict['a'][2] = 7

and do

    print("my_copy a[2]: ",my_copy['a'][2],",whereas my_deepcopy a[2]: ", my_deepcopy['a'][2])

you get

    >> my_copy a[2]:  7 ,whereas my_deepcopy a[2]:  3

A simpler (in my view) solution is to create a new dictionary and update it with the contents of the old one:

my_dict={'a':1}

my_copy = {}

my_copy.update( my_dict )

my_dict['a']=2

my_dict['a']
Out[34]: 2

my_copy['a']
Out[35]: 1

The problem with this approach is it may not be ‘deep enough’. i.e. is not recursively deep. good enough for simple objects but not for nested dictionaries. Here is an example where it may not be deep enough:

my_dict1={'b':2}

my_dict2={'c':3}

my_dict3={ 'b': my_dict1, 'c':my_dict2 }

my_copy = {}

my_copy.update( my_dict3 )

my_dict1['b']='z'

my_copy
Out[42]: {'b': {'b': 'z'}, 'c': {'c': 3}}

By using Deepcopy() I can eliminate the semi-shallow behavior, but I think one must decide which approach is right for your application. In most cases you may not care, but should be aware of the possible pitfalls… final example:

import copy

my_copy2 = copy.deepcopy( my_dict3 )

my_dict1['b']='99'

my_copy2
Out[46]: {'b': {'b': 'z'}, 'c': {'c': 3}}

I am creating a program that will download a .jar (java) file from a web server, by reading the URL that is specified in the .jad file of the same game/application. I’m using Python 3.2.1

I’ve managed to extract the URL of the JAR file from the JAD file (every JAD file contains the URL to the JAR file), but as you may imagine, the extracted value is type() string.

Here’s the relevant function:

def downloadFile(URL=None):
    import httplib2
    h = httplib2.Http(".cache")
    resp, content = h.request(URL, "GET")
    return content

downloadFile(URL_from_file)

However I always get an error saying that the type in the function above has to be bytes, and not string. I’ve tried using the URL.encode(‘utf-8′), and also bytes(URL,encoding=’utf-8’), but I’d always get the same or similar error.

So basically my question is how to download a file from a server when the URL is stored in a string type?

If you want to obtain the contents of a web page into a variable, just read the response of urllib.request.urlopen:

import urllib.request
...
url = 'http://example.com/'
response = urllib.request.urlopen(url)
data = response.read()      # a `bytes` object
text = data.decode('utf-8') # a `str`; this step can't be used if data is binary

The easiest way to download and save a file is to use the urllib.request.urlretrieve function:

import urllib.request
...
# Download the file from `url` and save it locally under `file_name`:
urllib.request.urlretrieve(url, file_name)

import urllib.request
...
# Download the file from `url`, save it in a temporary directory and get the
# path to it (e.g. '/tmp/tmpb48zma.txt') in the `file_name` variable:
file_name, headers = urllib.request.urlretrieve(url)

But keep in mind that urlretrieve is considered legacy and might become deprecated (not sure why, though).

So the most correct way to do this would be to use the urllib.request.urlopen function to return a file-like object that represents an HTTP response and copy it to a real file using shutil.copyfileobj.

import urllib.request
import shutil
...
# Download the file from `url` and save it locally under `file_name`:
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

If this seems too complicated, you may want to go simpler and store the whole download in a bytes object and then write it to a file. But this works well only for small files.

import urllib.request
...
# Download the file from `url` and save it locally under `file_name`:
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
    data = response.read() # a `bytes` object
    out_file.write(data)

It is possible to extract .gz (and maybe other formats) compressed data on the fly, but such an operation probably requires the HTTP server to support random access to the file.

import urllib.request
import gzip
...
# Read the first 64 bytes of the file inside the .gz archive located at `url`
url = 'http://example.com/something.gz'
with urllib.request.urlopen(url) as response:
    with gzip.GzipFile(fileobj=response) as uncompressed:
        file_header = uncompressed.read(64) # a `bytes` object
        # Or do anything shown above using `uncompressed` instead of `response`.

I use requests package whenever I want something related to HTTP requests because its API is very easy to start with:

first, install requests

$ pip install requests

then the code:

from requests import get  # to make GET request


def download(url, file_name):
    # open in binary mode
    with open(file_name, "wb") as file:
        # get request
        response = get(url)
        # write to file
        file.write(response.content)

I hope I understood the question right, which is: how to download a file from a server when the URL is stored in a string type?

I download files and save it locally using the below code:

import requests

url = 'https://www.python.org/static/img/python-logo.png'
fileName = 'D:\Python\dwnldPythonLogo.png'
req = requests.get(url)
file = open(fileName, 'wb')
for chunk in req.iter_content(100000):
    file.write(chunk)
file.close()

Here we can use urllib’s Legacy interface in Python3:

The following functions and classes are ported from the Python 2 module urllib (as opposed to urllib2). They might become deprecated at some point in the future.

Example (2 lines code):

import urllib.request

url = 'https://www.python.org/static/img/python-logo.png'
urllib.request.urlretrieve(url, "logo.png")

You can use wget which is popular downloading shell tool for that. https://pypi.python.org/pypi/wget This will be the simplest method since it does not need to open up the destination file. Here is an example.

import wget
url = 'https://i1.wp.com/python3.codes/wp-content/uploads/2015/06/Python3-powered.png?fit=650%2C350'  
wget.download(url, '/Users/scott/Downloads/cat4.jpg') 

Yes, definietly requests is great package to use in something related to HTTP requests. but we need to be careful with the encoding type of the incoming data as well below is an example which explains the difference

from requests import get

# case when the response is byte array
url = 'some_image_url'

response = get(url)
with open('output', 'wb') as file:
    file.write(response.content)


# case when the response is text
# Here unlikely if the reponse content is of type **iso-8859-1** we will have to override the response encoding
url = 'some_page_url'

response = get(url)
# override encoding by real educated guess as provided by chardet
r.encoding = r.apparent_encoding

with open('output', 'w', encoding='utf-8') as file:
    file.write(response.content)

Motivation

Sometimes, we are want to get the picture but not need to download it to real files,

i.e., download the data and keep it on memory.

For example, If I use the machine learning method, train a model that can recognize an image with the number (bar code).

When I spider some websites and that have those images so I can use the model to recognize it,

and I don’t want to save those pictures on my disk drive,

then you can try the below method to help you keep download data on memory.

Points

import requests
from io import BytesIO
response = requests.get(url)
with BytesIO as io_obj:
    for chunk in response.iter_content(chunk_size=4096):
        io_obj.write(chunk)

basically, is like to @Ranvijay Kumar

An Example

import requests
from typing import NewType, TypeVar
from io import StringIO, BytesIO
import matplotlib.pyplot as plt
import imageio

URL = NewType('URL', str)
T_IO = TypeVar('T_IO', StringIO, BytesIO)


def download_and_keep_on_memory(url: URL, headers=None, timeout=None, **option) -> T_IO:
    chunk_size = option.get('chunk_size', 4096)  # default 4KB
    max_size = 1024 ** 2 * option.get('max_size', -1)  # MB, default will ignore.
    response = requests.get(url, headers=headers, timeout=timeout)
    if response.status_code != 200:
        raise requests.ConnectionError(f'{response.status_code}')

    instance_io = StringIO if isinstance(next(response.iter_content(chunk_size=1)), str) else BytesIO
    io_obj = instance_io()
    cur_size = 0
    for chunk in response.iter_content(chunk_size=chunk_size):
        cur_size += chunk_size
        if 0 < max_size < cur_size:
            break
        io_obj.write(chunk)
    io_obj.seek(0)
    """ save it to real file.
    with open('temp.png', mode='wb') as out_f:
        out_f.write(io_obj.read())
    """
    return io_obj


def main():
    headers = {
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
        'Accept-Encoding': 'gzip, deflate',
        'Accept-Language': 'zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7',
        'Cache-Control': 'max-age=0',
        'Connection': 'keep-alive',
        'Host': 'statics.591.com.tw',
        'Upgrade-Insecure-Requests': '1',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'
    }
    io_img = download_and_keep_on_memory(URL('http://statics.591.com.tw/tools/showPhone.php?info_data=rLsGZe4U%2FbphHOimi2PT%2FhxTPqI&type=rLEFMu4XrrpgEw'),
                                         headers,  # You may need this. Otherwise, some websites will send the 404 error to you.
                                         max_size=4)  # max loading < 4MB
    with io_img:
        plt.rc('axes.spines', top=False, bottom=False, left=False, right=False)
        plt.rc(('xtick', 'ytick'), color=(1, 1, 1, 0))  # same of plt.axis('off')
        plt.imshow(imageio.imread(io_img, as_gray=False, pilmode="RGB"))
        plt.show()


if __name__ == '__main__':
    main()

from urllib import request

def get(url):
    with request.urlopen(url) as r:
        return r.read()


def download(url, file=None):
    if not file:
        file = url.split('/')[-1]
    with open(file, 'wb') as f:
        f.write(get(url))