I’m trying to write a simple Python script that will copy a index.tpl to index.html in all of the subdirectories (with a few exceptions).

I’m getting bogged down by trying to get the list of subdirectories.

I did some speed testing on various functions to return the full path to all current subdirectories.

tl;dr: Always use scandir:

list_subfolders_with_paths = [f.path for f in os.scandir(path) if f.is_dir()]

Bonus: With scandir you can also simply only get folder names by using f.name instead of f.path.

This (as well as all other functions below) will not use natural sorting. This means results will be sorted like this: 1, 10, 2. To get natural sorting (1, 2, 10), please have a look at https://stackoverflow.com/a/48030307/2441026

Results: scandir is: 3x faster than walk, 32x faster than listdir (with filter), 35x faster than Pathlib and 36x faster than listdir and 37x (!) faster than glob.

Scandir:           0.977
Walk:              3.011
Listdir (filter): 31.288
Pathlib:          34.075
Listdir:          35.501
Glob:             36.277

Tested with W7x64, Python 3.8.1. Folder with 440 subfolders.
In case you wonder if listdir could be speed up by not doing os.path.join() twice, yes, but the difference is basically nonexistent.

Code:

import os
import pathlib
import timeit
import glob

path = r"<example_path>"



def a():
    list_subfolders_with_paths = [f.path for f in os.scandir(path) if f.is_dir()]
    # print(len(list_subfolders_with_paths))


def b():
    list_subfolders_with_paths = [os.path.join(path, f) for f in os.listdir(path) if os.path.isdir(os.path.join(path, f))]
    # print(len(list_subfolders_with_paths))


def c():
    list_subfolders_with_paths = []
    for root, dirs, files in os.walk(path):
        for dir in dirs:
            list_subfolders_with_paths.append( os.path.join(root, dir) )
        break
    # print(len(list_subfolders_with_paths))


def d():
    list_subfolders_with_paths = glob.glob(path + '/*/')
    # print(len(list_subfolders_with_paths))


def e():
    list_subfolders_with_paths = list(filter(os.path.isdir, [os.path.join(path, f) for f in os.listdir(path)]))
    # print(len(list(list_subfolders_with_paths)))


def f():
    p = pathlib.Path(path)
    list_subfolders_with_paths = [x for x in p.iterdir() if x.is_dir()]
    # print(len(list_subfolders_with_paths))



print(f"Scandir:          {timeit.timeit(a, number=1000):.3f}")
print(f"Listdir:          {timeit.timeit(b, number=1000):.3f}")
print(f"Walk:             {timeit.timeit(c, number=1000):.3f}")
print(f"Glob:             {timeit.timeit(d, number=1000):.3f}")
print(f"Listdir (filter): {timeit.timeit(e, number=1000):.3f}")
print(f"Pathlib:          {timeit.timeit(f, number=1000):.3f}")

import os
def get_immediate_subdirectories(a_dir):
    return [name for name in os.listdir(a_dir)
            if os.path.isdir(os.path.join(a_dir, name))]

Why has no one mentioned glob? glob lets you use Unix-style pathname expansion, and is my go to function for almost everything that needs to find more than one path name. It makes it very easy:

from glob import glob
paths = glob('*/')

Note that glob will return the directory with the final slash (as unix would) while most path based solutions will omit the final slash.

Check “Getting a list of all subdirectories in the current directory“.

Here’s a Python 3 version:

import os

dir_list = next(os.walk('.'))[1]

print(dir_list)

import os, os.path

To get (full-path) immediate sub-directories in a directory:

def SubDirPath (d):
    return filter(os.path.isdir, [os.path.join(d,f) for f in os.listdir(d)])

To get the latest (newest) sub-directory:

def LatestDirectory (d):
    return max(SubDirPath(d), key=os.path.getmtime)

os.walk is your friend in this situation.

Straight from the documentation:

walk() generates the file names in a directory tree, by walking the tree either top down or bottom up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).

This method nicely does it all in one go.

from glob import glob
subd = [s.rstrip("/") for s in glob(parent_dir+"*/")]

Using Twisted’s FilePath module:

from twisted.python.filepath import FilePath

def subdirs(pathObj):
    for subpath in pathObj.walk():
        if subpath.isdir():
            yield subpath

if __name__ == '__main__':
    for subdir in subdirs(FilePath(".")):
        print "Subdirectory:", subdir

Since some commenters have asked what the advantages of using Twisted’s libraries for this is, I’ll go a bit beyond the original question here.

There’s some improved documentation in a branch that explains the advantages of FilePath; you might want to read that.

More specifically in this example: unlike the standard library version, this function can be implemented with no imports. The “subdirs” function is totally generic, in that it operates on nothing but its argument. In order to copy and move the files using the standard library, you need to depend on the “open” builtin, “listdir“, perhaps “isdir” or “os.walk” or “shutil.copy“. Maybe “os.path.join” too. Not to mention the fact that you need a string passed an argument to identify the actual file. Let’s take a look at the full implementation which will copy each directory’s “index.tpl” to “index.html”:

def copyTemplates(topdir):
    for subdir in subdirs(topdir):
        tpl = subdir.child("index.tpl")
        if tpl.exists():
            tpl.copyTo(subdir.child("index.html"))

The “subdirs” function above can work on any FilePath-like object. Which means, among other things, ZipPath objects. Unfortunately ZipPath is read-only right now, but it could be extended to support writing.

You can also pass your own objects for testing purposes. In order to test the os.path-using APIs suggested here, you have to monkey with imported names and implicit dependencies and generally perform black magic to get your tests to work. With FilePath, you do something like this:

class MyFakePath:
    def child(self, name):
        "Return an appropriate child object"

    def walk(self):
        "Return an iterable of MyFakePath objects"

    def exists(self):
        "Return true or false, as appropriate to the test"

    def isdir(self):
        "Return true or false, as appropriate to the test"
...
subdirs(MyFakePath(...))

I just wrote some code to move vmware virtual machines around, and ended up using os.path and shutil to accomplish file copying between sub-directories.

def copy_client_files (file_src, file_dst):
    for file in os.listdir(file_src):
            print "Copying file: %s" % file
            shutil.copy(os.path.join(file_src, file), os.path.join(file_dst, file))

It’s not terribly elegant, but it does work.

Here’s one way:

import os
import shutil

def copy_over(path, from_name, to_name):
  for path, dirname, fnames in os.walk(path):
    for fname in fnames:
      if fname == from_name:
        shutil.copy(os.path.join(path, from_name), os.path.join(path, to_name))


copy_over('.', 'index.tpl', 'index.html')

I have to mention the path.py library, which I use very often.

Fetching the immediate subdirectories become as simple as that:

my_dir.dirs()

The full working example is:

from path import Path

my_directory = Path("path/to/my/directory")

subdirs = my_directory.dirs()

NB: my_directory still can be manipulated as a string, since Path is a subclass of string, but providing a bunch of useful methods for manipulating paths

def get_folders_in_directories_recursively(directory, index=0):
    folder_list = list()
    parent_directory = directory

    for path, subdirs, _ in os.walk(directory):
        if not index:
            for sdirs in subdirs:
                folder_path = "{}/{}".format(path, sdirs)
                folder_list.append(folder_path)
        elif path[len(parent_directory):].count('/') + 1 == index:
            for sdirs in subdirs:
                folder_path = "{}/{}".format(path, sdirs)
                folder_list.append(folder_path)

    return folder_list

The following function can be called as:

get_folders_in_directories_recursively(directory, index=1) -> gives the list of folders in first level

get_folders_in_directories_recursively(directory) -> gives all the sub folders

import glob
import os

def child_dirs(path):
     cd = os.getcwd()        # save the current working directory
     os.chdir(path)          # change directory 
     dirs = glob.glob("*/")  # get all the subdirectories
     os.chdir(cd)            # change directory to the script original location
     return dirs

The child_dirs function takes a path a directory and returns a list of the immediate subdirectories in it.

dir
 |
  -- dir_1
  -- dir_2

child_dirs('dir') -> ['dir_1', 'dir_2']

import pathlib


def list_dir(dir):
    path = pathlib.Path(dir)
    dir = []
    try:
        for item in path.iterdir():
            if item.is_dir():
                dir.append(item)
        return dir
    except FileNotFoundError:
        print('Invalid directory')

One liner using pathlib:

list_subfolders_with_paths = [p for p in pathlib.Path(path).iterdir() if p.is_dir()]