是否可以使用pip从私有GitHub存储库安装软件包？

I am trying to install a Python package from a private GitHub repository. For a public repository, I can issue the following command which works fine:

pip install git+git://github.com/django/django.git

However, if I try this for a private repository:

pip install git+git://github.com/echweb/echweb-utils.git

I get the following output:

Downloading/unpacking git+git://github.com/echweb/echweb-utils.git
Cloning Git repository git://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build
Complete output from command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build:
fatal: The remote end hung up unexpectedly

Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build...

----------------------------------------
Command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build failed with error code 128

I guess this is because I am trying to access a private repository without providing any authentication. I therefore tried to use Git + ssh hoping that pip would use my SSH public key to authenticate:

pip install git+ssh://github.com/echweb/echweb-utils.git

This gives the following output:

Downloading/unpacking git+ssh://github.com/echweb/echweb-utils.git
Cloning Git repository ssh://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build
Complete output from command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build:
Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build...

Permission denied (publickey).

fatal: The remote end hung up unexpectedly

----------------------------------------
Command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build failed with error code 128

Is what I am trying to achieve even possible? If so, how can I do it?

You can use the git+ssh URI scheme, but you must set a username:

pip install git+ssh://git@github.com/echweb/echweb-utils.git

Do you see the git@ part into the URI?

PS: Also read about deploy keys.

PPS: In my installation, the “git+ssh” URI scheme works only with “editable” requirements:

pip install -e URI#egg=EggName

Remember: Change the : character that git remote -v prints to a / character before using the remote’s address in the pip command:

$ git remote -v
origin  git@github.com:echweb/echweb-utils.git (fetch)
                      ^ change this to a '/' character

If you forget, you will get this error:

ssh: Could not resolve hostname github.com:echweb:
         nodename nor servname provided, or not known

As an additional technique, if you have the private repository cloned locally, you can do:

pip install git+file://c:/repo/directory

More modernly, you can just do this (and the -e will mean you don’t have to commit changes before they’re reflected):

pip install -e C:\repo\directory

You can do it directly with the HTTPS URL like this:

pip install git+https://github.com/username/repo.git

This also works just appending that line in the requirements.txt in a Django project, for instance.

It also works with Bitbucket:

pip install git+ssh://git@bitbucket.org/username/projectname.git

Pip will use your SSH keys in this case.

The syntax for the requirements file is given here:

https://pip.pypa.io/en/latest/reference/pip_install.html#requirements-file-format

So for example, use:

-e git+http://github.com/rwillmer/django-behave#egg=django-behave

if you want the source to stick around after installation.

Or just

git+http://github.com/rwillmer/django-behave#egg=django-behave

if you just want it to be installed.

I found it much easier to use tokens than SSH keys. I couldn’t find much good documentation on this, so I came across this solution mainly through trial and error. Further, installing from pip and setuptools have some subtle differences; but this way should work for both.

GitHub don’t (currently, as of August 2016) offer an easy way to get the zip / tarball of private repositories. So you need to point setuptools to tell setuptools that you’re pointing to a Git repository:

from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

• For private repositories, you need to authenticate with GitHub; the simplest way I found is to create an OAuth token, drop that into your environment, and then include it with the URL
• You need to include some version number (here is 0) at the end of the link, even if there’s isn’t any package on PyPI. This has to be a actual number, not a word.
• You need to preface with git+ to tell setuptools it’s to clone the repository, rather than pointing at a zip / tarball
• version can be a branch, a tag, or a commit hash
• You need to supply --process-dependency-links if installing from pip

I figured out a way to automagically ‘pip install’ a GitLab private repository that requires no password prompt. This approach uses GitLab “Deploy Keys” and an SSH configuration file, so you can deploy using keys other than your personal SSH keys (in my case, for use by a ‘bot). Perhaps someone kind soul can verify using GitHub.

Create a New SSH key:

ssh-keygen -t rsa -C "GitLab_Robot_Deploy_Key"

The file should show up as ~/.ssh/GitLab_Robot_Deploy_Key and ~/.ssh/GitLab_Robot_Deploy_Key.pub.

Copy and paste the contents of the ~/.ssh/GitLab_Robot_Deploy_Key.pub file into the GitLab “Deploy Keys” dialog.

Test the New Deploy Key

The following command tells SSH to use your new deploy key to set up the connection. On success, you should get the message: “Welcome to GitLab, UserName!”

ssh -T -i ~/.ssh/GitLab_Robot_Deploy_Key git@gitlab.mycorp.com

Create the SSH Configuration File

Next, use an editor to create a ~/.ssh/config file. Add the following contents. The ‘Host’ value can be anything you want (just remember it, because you’ll be using it later). The HostName is the URL to your GitLab instance. The IdentifyFile is path to the SSH key file you created in the first step.

Host GitLab
  HostName gitlab.mycorp.com
  IdentityFile ~/.ssh/GitLab_Robot_Deploy_Key

Point SSH to the Configuration file

oxyum gave us the recipe for using pip with SSH:

pip install git+ssh://git@gitlab.mycorp.com/my_name/my_repo.git

We just need to modify it a bit to make SSH use our new Deploy Key. We do that by pointing SSH to the Host entry in the SSH configuration file. Just replace the ‘gitlab.mycorp.com’ in the command to the host name we used in the SSH configuration file:

pip install git+ssh://git@GitLab/my_name/my_repo.git

The package should now install without any password prompt.

Reference A
Reference B

When I was installing from GitHub I was able to use:

pip install git+ssh://git@github.com/<username>/<projectname>.git#egg=<eggname>

But, since I had to run pip as sudo, the SSH keys were not working with GitHub any more, and “git clone” failed on “Permission denied (publickey)”. Using git+https allowed me to run the command as sudo, and have GitHub ask me for my user/password.

sudo pip install git+https://github.com/<username>/<projectname>.git#egg=<eggname>

You can also install a private repository dependency via git+https://github.com/… URL by providing login credentials (login and password, or deploy token) for curl with the .netrc file:

echo "machine github.com login ei-grad password mypasswordshouldbehere" > ~/.netrc
pip install "git+https://github.com/ei-grad/my_private_repo.git#egg=my_private_repo"

If you want to install dependencies from a requirements file within a CI server or alike, you can do this:

git config --global credential.helper 'cache'
echo "protocol=https
host=example.com
username=${GIT_USER}
password=${GIT_PASS}
" | git credential approve
pip install -r requirements.txt

In my case, I used GIT_USER=gitlab-ci-token and GIT_PASS=${CI_JOB_TOKEN}.

This method has a clear advantage. You have a single requirements file which contains all of your dependencies.

If you don’t want to use SSH, you could add the username and password in the HTTPS URL.

The code below assumes that you have a file called “pass” in the working directory that contains your password.

export PASS=$(cat pass)
pip install git+https://<username>:$PASS@github.com/echweb/echweb-utils.git

oxyum’s solution is OK for this answer. I just want to point out that you need to be careful if you are installing using sudo as the keys must be stored for root too (for example, /root/.ssh).

Then you can type

sudo pip install git+ssh://git@github.com/echweb/echweb-utils.git

If you have your own library/package on GitHub, GitLab, etc., you have to add a tag to commit with a concrete version of the library, for example, v2.0, and then you can install your package:

pip install git+ssh://link/name/repo.git@v2.0

This works for me. Other solutions haven’t worked for me.

Here’s a quick method that worked for me. Simply fork the repo and install it from your own GitHub account with

pip install git+https://github.com/yourName/repoName

Just copy the remote from the original git clone command (or from git remote -v). You will get something like this:

• Bitbucket: git+ssh://git@bitbucket.org:your_account/my_pro.git

• GitHub: git+ssh://git@github.com:your_account/my_pro.git

Next, you need to replace : with / next to the domain name.

So install using:

pip install git+ssh://git@bitbucket.org/your_account/my_pro.git

You may try

pip install git+git@gitlab.mycorp.com/my_name/my_repo.git

without ssh:.... That works for me.