标签归档：html-parsing

知识问答

使用Python解析HTML

问题：使用Python解析HTML

我正在寻找适用于Python的HTML Parser模块，该模块可以帮助我以Python列表/字典/对象的形式获取标签。

如果我有以下格式的文件：

<html>
<head>Heading</head>
<body attr1='val1'>
    <div class='container'>
        <div id='class'>Something here</div>
        <div>Something else</div>
    </div>
</body>
</html>

那么它应该给我一种通过HTML标签的名称或ID访问嵌套标签的方法，这样我基本上可以要求它为我div提供class='container'包含在body标签中或类似标签的标签中的内容/文本。

如果您使用了Firefox的“检查元素”功能（查看HTML），您就会知道它以一种很好的嵌套方式（如树）为您提供了所有标签。

我更喜欢一个内置模块，但是可能要求太多。

我在Stack Overflow上遇到了很多问题，在互联网上也有一些博客，其中大多数都建议使用BeautifulSoup或lxml或HTMLParser，但是其中很少有详细介绍功能，最后只是参数哪个更快/更有效。

点击查看英文原文

I’m looking for an HTML Parser module for Python that can help me get the tags in the form of Python lists/dictionaries/objects.

If I have a document of the form:

<html>
<head>Heading</head>
<body attr1='val1'>
    <div class='container'>
        <div id='class'>Something here</div>
        <div>Something else</div>
    </div>
</body>
</html>

then it should give me a way to access the nested tags via the name or id of the HTML tag so that I can basically ask it to get me the content/text in the div tag with class='container' contained within the body tag, or something similar.

If you’ve used Firefox’s “Inspect element” feature (view HTML) you would know that it gives you all the tags in a nice nested manner like a tree.

I’d prefer a built-in module but that might be asking a little too much.

I went through a lot of questions on Stack Overflow and a few blogs on the internet and most of them suggest BeautifulSoup or lxml or HTMLParser but few of these detail the functionality and simply end as a debate over which one is faster/more efficent.

回答 0

这样我就可以要求它为我获取div标签中的内容/文本，其中body =包含class =’container’或类似内容。

try: 
    from BeautifulSoup import BeautifulSoup
except ImportError:
    from bs4 import BeautifulSoup
html = #the HTML code you've written above
parsed_html = BeautifulSoup(html)
print(parsed_html.body.find('div', attrs={'class':'container'}).text)

我猜您不需要性能描述-只需阅读BeautifulSoup的工作原理即可。查看其官方文档。

点击查看英文原文

So that I can ask it to get me the content/text in the div tag with class=’container’ contained within the body tag, Or something similar.

try: 
    from BeautifulSoup import BeautifulSoup
except ImportError:
    from bs4 import BeautifulSoup
html = #the HTML code you've written above
parsed_html = BeautifulSoup(html)
print(parsed_html.body.find('div', attrs={'class':'container'}).text)

You don’t need performance descriptions I guess – just read how BeautifulSoup works. Look at its official documentation.

回答 1

我猜你在找什么pyquery：

pyquery：类似jQuery的python库。

您想要的示例可能像：

from pyquery import PyQuery    
html = # Your HTML CODE
pq = PyQuery(html)
tag = pq('div#id') # or     tag = pq('div.class')
print tag.text()

并且它使用与Firefox或Chrome的inspect元素相同的选择器。例如：

元素选择器为“ div＃mw-head.noprint”

被检查的元素选择器是“ div＃mw-head.noprint”。因此，在pyquery中，您只需要传递此选择器：

pq('div#mw-head.noprint')
点击查看英文原文

I guess what you’re looking for is pyquery:

pyquery: a jquery-like library for python.

An example of what you want may be like:

from pyquery import PyQuery    
html = # Your HTML CODE
pq = PyQuery(html)
tag = pq('div#id') # or     tag = pq('div.class')
print tag.text()

And it uses the same selectors as Firefox’s or Chrome’s inspect element. For example:

the element selector is 'div#mw-head.noprint'

The inspected element selector is ‘div#mw-head.noprint’. So in pyquery, you just need to pass this selector:

pq('div#mw-head.noprint')

回答 2

在这里，您可以了解有关Python中不同HTML解析器及其性能的更多信息。即使文章有些陈旧，它仍然可以为您提供很好的概述。

Python HTML解析器性能

即使它不是内置的，我也建议使用BeautifulSoup。只是因为它很容易处理这些任务。例如：

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen('http://www.google.com/')
soup = BeautifulSoup(page)

x = soup.body.find('div', attrs={'class' : 'container'}).text
点击查看英文原文

Here you can read more about different HTML parsers in Python and their performance. Even though the article is a bit dated it still gives you a good overview.

Python HTML parser performance

I’d recommend BeautifulSoup even though it isn’t built in. Just because it’s so easy to work with for those kinds of tasks. Eg:

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen('http://www.google.com/')
soup = BeautifulSoup(page)

x = soup.body.find('div', attrs={'class' : 'container'}).text

回答 3

与其他解析器库相比，lxml速度非常快：

而且，cssselect它也非常容易用于抓取HTML页面：

from lxml.html import parse
doc = parse('http://www.google.com').getroot()
for div in doc.cssselect('a'):
    print '%s: %s' % (div.text_content(), div.get('href'))

lxml.html文档

点击查看英文原文

Compared to the other parser libraries lxml is extremely fast:

And with cssselect it’s quite easy to use for scraping HTML pages too:

from lxml.html import parse
doc = parse('http://www.google.com').getroot()
for div in doc.cssselect('a'):
    print '%s: %s' % (div.text_content(), div.get('href'))

lxml.html Documentation

回答 4

我建议使用lxml解析HTML。请参阅“解析HTML”（在lxml网站上）。

以我的经验，Beautiful Soup将一些复杂的HTML弄乱了。我相信这是因为Beautiful Soup不是解析器，而是非常好的字符串分析器。

点击查看英文原文

I recommend lxml for parsing HTML. See “Parsing HTML” (on the lxml site).

In my experience Beautiful Soup messes up on some complex HTML. I believe that is because Beautiful Soup is not a parser, rather a very good string analyzer.

回答 5

我建议使用justext库：

https://github.com/miso-belica/jusText

用法： Python2：

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print paragraph.text

Python3： 

import requests
import justext

response = requests.get("http://bbc.com/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print (paragraph.text)
点击查看英文原文

I recommend using justext library:

https://github.com/miso-belica/jusText

Usage: Python2:

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print paragraph.text

Python3: 

import requests
import justext

response = requests.get("http://bbc.com/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
    print (paragraph.text)

回答 6

我会使用EHP

https://github.com/iogf/ehp

这里是：

from ehp import *

doc = '''<html>
<head>Heading</head>
<body attr1='val1'>
    <div class='container'>
        <div id='class'>Something here</div>
        <div>Something else</div>
    </div>
</body>
</html>
'''

html = Html()
dom = html.feed(doc)
for ind in dom.find('div', ('class', 'container')):
    print ind.text()

输出：

Something here
Something else
点击查看英文原文

I would use EHP

https://github.com/iogf/ehp

Here it is:

from ehp import *

doc = '''<html>
<head>Heading</head>
<body attr1='val1'>
    <div class='container'>
        <div id='class'>Something here</div>
        <div>Something else</div>
    </div>
</body>
</html>
'''

html = Html()
dom = html.feed(doc)
for ind in dom.find('div', ('class', 'container')):
    print ind.text()

Output:

Something here
Something else