I have a project hosted on GitHub. For this I have written my README using the Markdown syntax in order to have it nicely formatted on GitHub.
As my project is in Python I also plan to upload it to PyPi. The syntax used for READMEs on PyPi is reStructuredText.
I would like to avoid having to handle two READMEs containing roughly the same content; so I searched for a markdown to RST (or the other way around) translator, but couldn’t find any.
The other solution I see is to perform a markdown/HTML and then a HTML/RST translation. I found some ressources for this here and here so I guess it should be possible.
Would you have any idea that could fit better with what I want to do?
I would recommend Pandoc, the “swiss-army knife for converting files from one markup format into another” (check out the diagram of supported conversions at the bottom of the page, it is quite impressive). Pandoc allows markdown to reStructuredText translation directly. There is also an online editor here which lets you try it out, so you could simply use the online editor to convert your README files.
As @Chris suggested, you can use Pandoc to convert Markdown to RST. This can be simply automated using pypandoc module and some magic in setup.py:
from setuptools import setup
try:
from pypandoc import convert
read_md = lambda f: convert(f, 'rst')
except ImportError:
print("warning: pypandoc module not found, could not convert Markdown to RST")
read_md = lambda f: open(f, 'r').read()
setup(
# name, version, ...
long_description=read_md('README.md'),
install_requires=[]
)
This will automatically convert README.md to RST for the long description using on PyPi. When pypandoc is not available, then it just reads README.md without the conversion – to not force others to install pypandoc when they wanna just build the module, not upload to PyPi.
So you can write in Markdown as usual and don’t care about RST mess anymore. ;)
The PyPI Warehouse now supports rendering Markdown as well! You just need to update your package configuration and add the long_description_content_type='text/markdown' to it. e.g.:
setup(
name='an_example_package',
# other arguments omitted
long_description=long_description,
long_description_content_type='text/markdown'
)
Therefore, there is no need to keep the README in two formats any longer.
You can find more information about it in the documentation.
Old answer:
The Markup library used by GitHub supports reStructuredText. This means you can write a README.rst file.
They even support syntax specific color highlighting using the code and code-block directives (Example)
In setup.py, set long_description to a Markdown string, add long_description_content_type="text/markdown" and make sure you’re using recent tooling (setuptools 38.6.0+, twine 1.11+).
For my requirements I didn’t want to install Pandoc in my computer. I used docverter. Docverter is a document conversion server with an HTTP interface using Pandoc for this.
import requests
r = requests.post(url='http://c.docverter.com/convert',
data={'to':'rst','from':'markdown'},
files={'input_files[]':open('README.md','rb')})
if r.ok:
print r.content
You might also be interested in the fact that it is possible to write in a common subset so that your document comes out the same way when rendered as markdown or rendered as reStructuredText: https://gist.github.com/dupuy/1855764 ☺
回答 6
我遇到了这个问题，并使用以下两个bash脚本解决了这个问题。
请注意，我已将LaTeX捆绑到Markdown中。
#!/usr/bin/env bashif[ $# -lt 1 ]; then
echo "$0 file.md"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"if["$extension"="md"]; then
rst=".rst"
pandoc $1 -o $filename$rst
fi
将其转换为html也很有用。md2html：
#!/usr/bin/env bashif[ $# -lt 1 ]; then
echo "$0 file.md <style.css>"
exit;
fi
filename=$(basename "$1")
extension="${filename##*.}"
filename="${filename%.*}"if["$extension"="md"]; then
html=".html"if[-z $2 ]; then
# if no css
pandoc -s -S --mathjax --highlight-style pygments $1 -o $filename$html
else
pandoc -s -S --mathjax --highlight-style pygments -c $2 $1 -o $filename$html
fi
fi
#!/usr/bin/env python'''
Recursively and destructively creates a .rst file for all Markdown
files in the target directory and below.
Created to deal with PyPa without changing anything in setup based on
the idea that getting proper Markdown support later is worth waiting
for rather than forcing a pandoc dependency in sample packages and such.
Vote for
(https://bitbucket.org/pypa/pypi/issue/148/support-markdown-for-readmes)
'''import sys, os, re
markdown_sufs =('.md','.markdown','.mkd')
markdown_regx ='\.(md|markdown|mkd)$'
target ='.'if len(sys.argv)>=2: target = sys.argv[1]
md_files =[]for root, dirnames, filenames in os.walk(target):for name in filenames:if name.endswith(markdown_sufs):
md_files.append(os.path.join(root, name))for md in md_files:
bare = re.sub(markdown_regx,'',md)
cmd='pandoc --from=markdown --to=rst "{}" -o "{}.rst"'print(cmd.format(md,bare))
os.system(cmd.format(md,bare))
Using the pandoc tool suggested by others I created a md2rst utility to create the rst files. Even though this solution means you have both an md and an rst it seemed to be the least invasive and would allow for whatever future markdown support is added. I prefer it over altering setup.py and maybe you would as well:
#!/usr/bin/env python
'''
Recursively and destructively creates a .rst file for all Markdown
files in the target directory and below.
Created to deal with PyPa without changing anything in setup based on
the idea that getting proper Markdown support later is worth waiting
for rather than forcing a pandoc dependency in sample packages and such.
Vote for
(https://bitbucket.org/pypa/pypi/issue/148/support-markdown-for-readmes)
'''
import sys, os, re
markdown_sufs = ('.md','.markdown','.mkd')
markdown_regx = '\.(md|markdown|mkd)$'
target = '.'
if len(sys.argv) >= 2: target = sys.argv[1]
md_files = []
for root, dirnames, filenames in os.walk(target):
for name in filenames:
if name.endswith(markdown_sufs):
md_files.append(os.path.join(root, name))
for md in md_files:
bare = re.sub(markdown_regx,'',md)
cmd='pandoc --from=markdown --to=rst "{}" -o "{}.rst"'
print(cmd.format(md,bare))
os.system(cmd.format(md,bare))
It’s still calling pip under the covers, but you can now unify your conda and pip package specifications in a single environment.yml file.
If you wanted to update your root environment with this file, you would need to save this to a file (for example, environment.yml), then run the command: conda env update -f environment.yml.
It’s more likely that you would want to create a new environment:
conda env create -f environment.yml (changed as supposed in the comments)
conda doesn’t support this directly because it installs from binaries, whereas git install would be from source. conda build does support recipes that are built from git. On the other hand, if all you want to do is keep up-to-date with the latest and greatest of a package, using pip inside of Anaconda is just fine, or alternately, use setup.py develop against a git clone.
I’m playing with both learning python and trying to get github issues into a readable form. Using the advice on How can I convert JSON to CSV? I came up with this:
import json
import csv
f=open('issues.json')
data = json.load(f)
f.close()
f=open("issues.csv","wb+")
csv_file=csv.writer(f)
csv_file.writerow(["gravatar_id","position","number","votes","created_at","comments","body","title","updated_at","html_url","user","labels","state"])
for item in data:
csv_file.writerow([item["gravatar_id"], item["position"], item["number"], item["votes"], item["created_at"], item["comments"], item["body"], item["title"], item["updated_at"], item["html_url"], item["user"], item["labels"], item["state"]])
Where “issues.json” is the json file containing my github issues. When I try to run that, I get
File "foo.py", line 14, in <module>
csv_file.writerow([item["gravatar_id"], item["position"], item["number"], item["votes"], item["created_at"], item["comments"], item["body"], item["title"], item["updated_at"], item["html_url"], item["user"], item["labels"], item["state"]])
TypeError: string indices must be integers
What am I missing here? Which are the “string indices”? I’m sure that once I get this working I’ll have more issues, but for now , I’d just love for this to work!
When I tweak the for statement to simply
for item in data:
print item
what I get is … “issues” — so I’m doing something more basic wrong. Here’s a bit of my json:
item is most likely a string in your code; the string indices are the ones in the square brackets, e.g., gravatar_id. So I’d first check your data variable to see what you received there; I guess that data is a list of strings (or at least a list containing at least one string) while it should be a list of dictionaries.
回答 1
该变量item是一个字符串。索引如下所示：
>>> mystring ='helloworld'>>>print mystring[0]'h'
上面的示例使用0字符串的索引来引用第一个字符。
字符串不能具有字符串索引（就像字典一样）。所以这行不通：
>>> mystring ='helloworld'>>>print mystring['stringindex']TypeError: string indices must be integers
tl;dr: use a colon: instead of a comma in between the two indices a and b in str[a:b]
When working with strings and slice notation (a common sequence operation), it can happen that a TypeError is raised, pointing out that the indices must be integers, even if they obviously are.
Example
>>> my_string = "hello world"
>>> my_string[0,5]
TypeError: string indices must be integers
We obviously passed two integers for the indices to the slice notation, right? So what is the problem here?
This error can be very frustrating – especially at the beginning of learning Python – because the error message is a little bit misleading.
Explanation
We implicitly passed a tuple of two integers (0 and 5) to the slice notation when we called my_string[0,5] because 0,5 (even without the parentheses) evaluates to the same tuple as (0,5) would do.
A comma , is actually enough for Python to evaluate something as a tuple:
A clearer and more helpful error message could have been something like:
TypeError: string indices must be integers (not tuple)
A good error message shows the user directly what they did wrong and it would have been more obvious how to solve the problem.
[So the next time when you find yourself responsible for writing an error description message, think of this example and add the reason or other useful information to error message to let you and maybe other people understand what went wrong.]
Lessons learned
slice notation uses colons : to separate its indices (and step range, e.g. str[from:to:step])
tuples are defined by commas , (e.g. t = 1,)
add some information to error messages for users to understand what went wrong
Cheers and happy programming
winklerrr
[I know this question was already answered and this wasn’t exactly the question the thread starter asked, but I came here because of the above problem which leads to the same error message. At least it took me quite some time to find that little typo.
So I hope that this will help someone else who stumbled upon the same error and saves them some time finding that tiny mistake.]
This can happen if a comma is missing. I ran into it when I had a list of two-tuples, each of which consisted of a string in the first position, and a list in the second. I erroneously omitted the comma after the first component of a tuple in one case, and the interpreter thought I was trying to index the first component.
Downloading/unpacking git+git://github.com/echweb/echweb-utils.git
Cloning Git repository git://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build
Complete output from command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build:
fatal: The remote end hung up unexpectedly
Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build...
----------------------------------------
Command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build failed with error code 128
I guess this is because I am trying to access a private repository without providing any authentication. I therefore tried to use Git + ssh hoping that pip would use my SSH public key to authenticate:
from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(# ...
install_requires='package',
dependency_links =['git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'.format(github_token=github_token, package=package, version=master)]
I found it much easier to use tokens than SSH keys. I couldn’t find much good documentation on this, so I came across this solution mainly through trial and error. Further, installing from pip and setuptools have some subtle differences; but this way should work for both.
GitHub don’t (currently, as of August 2016) offer an easy way to get the zip / tarball of private repositories. So you need to point setuptools to tell setuptools that you’re pointing to a Git repository:
from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
For private repositories, you need to authenticate with GitHub; the simplest way I found is to create an OAuth token, drop that into your environment, and then include it with the URL
You need to include some version number (here is 0) at the end of the link, even if there’s isn’t any package on PyPI. This has to be a actual number, not a word.
You need to preface with git+ to tell setuptools it’s to clone the repository, rather than pointing at a zip / tarball
version can be a branch, a tag, or a commit hash
You need to supply --process-dependency-links if installing from pip
I figured out a way to automagically ‘pip install’ a GitLab private repository that requires no password prompt. This approach uses GitLab “Deploy Keys” and an SSH configuration file, so you can deploy using keys other than your personal SSH keys (in my case, for use by a ‘bot). Perhaps someone kind soul can verify using GitHub.
Create a New SSH key:
ssh-keygen -t rsa -C "GitLab_Robot_Deploy_Key"
The file should show up as ~/.ssh/GitLab_Robot_Deploy_Key and ~/.ssh/GitLab_Robot_Deploy_Key.pub.
Copy and paste the contents of the ~/.ssh/GitLab_Robot_Deploy_Key.pub file into the GitLab “Deploy Keys” dialog.
Test the New Deploy Key
The following command tells SSH to use your new deploy key to set up the connection. On success, you should get the message: “Welcome to GitLab, UserName!”
Next, use an editor to create a ~/.ssh/config file. Add the following contents. The ‘Host’ value can be anything you want (just remember it, because you’ll be using it later). The HostName is the URL to your GitLab instance. The IdentifyFile is path to the SSH key file you created in the first step.
We just need to modify it a bit to make SSH use our new Deploy Key. We do that by pointing SSH to the Host entry in the SSH configuration file. Just replace the ‘gitlab.mycorp.com’ in the command to the host name we used in the SSH configuration file:
But, since I had to run pip as sudo, the SSH keys were not working with GitHub any more, and “git clone” failed on “Permission denied (publickey)”. Using git+https allowed me to run the command as sudo, and have GitHub ask me for my user/password.
You can also install a private repository dependency via git+https://github.com/… URL by providing login credentials (login and password, or deploy token) for curl with the .netrc file:
oxyum’s solution is OK for this answer. I just want to point out that you need to be careful if you are installing using sudo as the keys must be stored for root too (for example, /root/.ssh).
If you have your own library/package on GitHub, GitLab, etc., you have to add a tag to commit with a concrete version of the library, for example, v2.0, and then you can install your package:
pip install git+ssh://link/name/repo.git@v2.0
This works for me. Other solutions haven’t worked for me.
Downloading/unpacking elasticutils==0.7.dev (from-r requirements.txt (line 20))Couldnot find a version that satisfies the requirement elasticutils==0.7.dev (from-r requirements.txt (line 20))(from versions:)No distributions matching the version for elasticutils==0.7.dev (from-r requirements.txt (line 20))
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I’ve looked at other tickets like this but that didn’t solve my problem. If I put something like
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here’s my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you’d probably want to publish your module in it’s own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
I’m finding that it’s kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04’s package manager) to actually install the thing I tell it to install. I’m posting this answer to save anyone’s time who runs into this problem.
By “failed” I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
configure Configure gitsome.
create-comment Create a comment on the given issue.
create-issue Create an issue.
create-repo Create a repo.
emails List all the user's registered emails.
emojis List all GitHub supported emojis.
feed List all activity for the given user or repo.
followers List all followers and the total follower count.
following List all followed users and the total followed count.
gitignore-template Output the gitignore template for the given language.
gitignore-templates Output all supported gitignore templates.
issue Output detailed information about the given issue.
issues List all issues matching the filter.
license Output the license template for the given license.
licenses Output all supported license templates.
me List information about the logged in user.
notifications List all notifications.
octo Output an Easter egg or the given message from Octocat.
pull-request Output detailed information about the given pull request.
pull-requests List all pull requests.
rate-limit Output the rate limit. Not available for Enterprise.
repo Output detailed information about the given filter.
repos List all repos matching the given filter.
search-issues Search for all issues matching the given query.
search-repos Search for all repos matching the given query.
starred Output starred repos.
trending List trending repos for the given language.
user List information about the given user.
view View the given index in the terminal or a browser.
Copyright 2016 Donne Martin
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
logger.add("out.log", backtrace=True, diagnose=True) # Caution, may leak sensitive data in prod
def func(a, b):
return a / b
def nested(c):
try:
func(5, c)
except ZeroDivisionError:
logger.exception("What?!")
nested(0)
