How can I get the HTML source in a variable using the Selenium module with Python?

I wanted to do something like this:

from selenium import webdriver

browser = webdriver.Firefox()
browser.get("http://example.com")
if "whatever" in html_source:
    # Do something
else:
    # Do something else

How can I do this? I don’t know how to access the HTML source.

You need to access the page_source property:

from selenium import webdriver

browser = webdriver.Firefox()
browser.get("http://example.com")

html_source = browser.page_source
if "whatever" in html_source:
    # do something
else:
    # do something else

With Selenium2Library you can use get_source()

import Selenium2Library
s = Selenium2Library.Selenium2Library()
s.open_browser("localhost:7080", "firefox")
source = s.get_source()

driver.page_source will help you get the page source code. You can check if the text is present in the page source or not.

from selenium import webdriver
driver = webdriver.Firefox()
driver.get("some url")
if "your text here" in driver.page_source:
    print('Found it!')
else:
    print('Did not find it.')

If you want to store the page source in a variable, add below line after driver.get:

var_pgsource=driver.page_source

and change the if condition to:

if "your text here" in var_pgsource:

By using the page source you will get the whole HTML code.
So first decide the block of code or tag in which you require to retrieve the data or to click the element..

options = driver.find_elements_by_name_("XXX")
for option in options:
    if option.text == "XXXXXX":
        print(option.text)
        option.click()

You can find the elements by name, XPath, id, link and CSS path.

To answer your question about getting the URL to use for urllib, just execute this JavaScript code:

url = browser.execute_script("return window.location;")

You can simply use the WebDriver object, and access to the page source code via its @property field page_source…

Try this code snippet :-)

from selenium import webdriver
driver = webdriver.Firefox('path/to/executable')
driver.get('https://some-domain.com')
source = driver.page_source
if 'stuff' in source:
    print('found...')
else:
    print('not in source...')

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
html_source_code = driver.execute_script("return document.body.innerHTML;")
html_soup: BeautifulSoup = BeautifulSoup(html_source_code, 'html.parser')

Now you can apply BeautifulSoup function to extract data…

I’d recommend getting the source with urllib and, if you’re going to parse, use something like Beautiful Soup.

import urllib

url = urllib.urlopen("http://example.com") # Open the URL.
content = url.readlines() # Read the source and save it to a variable.

I’m trying to get the current url after a series of navigations in Selenium. I know there’s a command called getLocation for ruby, but I can’t find the syntax for Python.

Use current_url element for Python 2:

print browser.current_url

For Python 3 and later versions of selenium:

print(driver.current_url)

According to this documentation (a place full of goodies:)):

driver.current_url

or, see official documentation: https://seleniumhq.github.io/docs/site/en/webdriver/browser_manipulation/#get-current-url

Selenium2Library has get_location():

import Selenium2Library
s = Selenium2Library.Selenium2Library()
url = s.get_location()

Another way to do it would be to inspect the url bar in chrome to find the id of the element, have your WebDriver click that element, and then send the keys you use to copy and paste using the keys common function from selenium, and then printing it out or storing it as a variable, etc.

I’ve got some tests where I’m checking that the proper error message appears when text in certain fields are invalid. One check for validity is that a certain textarea element is not empty.

If this textarea already has text in it, how can I tell selenium to clear the field?

something like:

driver.get_element_by_id('foo').clear_field()

driver.find_element_by_id('foo').clear()

You can use

 webElement.clear();

If this element is a text entry element, this will clear the value.

Note that the events fired by this event may not be as you’d expect. In particular, we don’t fire any keyboard or mouse events. If you want to ensure keyboard events are fired, consider using something like sendKeys(CharSequence). E.g.:

 webElement.sendKeys(Keys.BACK_SPACE); //do repeatedly, e.g. in while loop

or:

 webElement.sendKeys(Keys.CONTROL + "a");
 webElement.sendKeys(Keys.DELETE);

I ran into a field where .clear() did not work. Using a combination of the first two answers worked for this field.

from selenium.webdriver.common.keys import Keys

#...your code (I was using python 3)

driver.find_element_by_id('foo').send_keys(Keys.CONTROL + "a");
driver.find_element_by_id('foo').send_keys(Keys.DELETE);

In the most recent Selenium version, use:

driver.find_element_by_id('foo').clear()

for java

driver.findelement(By.id('foo').clear();

or

webElement.clear();

If this element is a text entry element, this will clear the value.

It is general syntax

driver.find_element_by_id('Locator value').clear();
driver.find_element_by_name('Locator value').clear();

With a simple call of clear() it appears in the DOM that the corresponding input/textarea component still has its old value, so any following changes on that component (e.g. filling the component with a new value) will not be processed in time.

If you take a look in the selenium source code you’ll find that the clear()-method is documented with the following comment:

/** If this element is a text entry element, this will clear the value. Has no effect on other elements. Text entry elements are INPUT and TEXTAREA elements. Note that the events fired by this event may not be as you’d expect. In particular, we don’t fire any keyboard or mouse events. If you want to ensure keyboard events are fired, consider using something like {@link #sendKeys(CharSequence…)} with the backspace key. To ensure you get a change event, consider following with a call to {@link #sendKeys(CharSequence…)} with the tab key. */

So using this helpful hint to clear an input/textarea (component that already has a value) AND assign a new value to it, you’ll get some code like the following:

public void waitAndClearFollowedByKeys(By by, CharSequence keys) {
    LOG.debug("clearing element");
    wait(by, true).clear();
    sendKeys(by, Keys.BACK_SPACE.toString() + keys);
}

public void sendKeys(By by, CharSequence keysToSend) {
    WebElement webElement = wait(by, true);
    LOG.info("sending keys '{}' to {}", escapeProperly(keysToSend), by);
    webElement.sendKeys(keysToSend);
    LOG.info("keys sent");
}

private String escapeProperly(CharSequence keysToSend) {
    String result = "" + keysToSend;
    result = result.replace(Keys.TAB, "\\t");
    result = result.replace(Keys.ENTER, "\\n");
    result = result.replace(Keys.RETURN, "\\r");

    return result;
}

Sorry for this code being Java and not Python. Also, I had to skip out an additional “waitUntilPageIsReady()-method that would make this post way too long.

Hope this helps you on your journey with Selenium!

In my experience, this turned out to be the most efficient

driver.find_element_by_css_selector('foo').send_keys(u'\ue009' + u'\ue003')

We are sending Ctrl + Backspace to delete all characters from the input, you can also replace backspace with delete.

EDIT: removed Keys dependency

driver.find_element_by_xpath("path").send_keys(Keys.CONTROL + u'\ue003') worked great with FireFox

• u’\ue003′ is a BACK_SPACE for those like me – never remembering it)

I’m trying to test a complicated JavaScript interface with Selenium (using the Python interface, and across multiple browsers). I have a number of buttons of the form:

<div>My Button</div>

I’d like to be able to search for buttons based on “My Button” (or non-case-sensitive, partial matches such as “my button” or “button”).

I’m finding this amazingly difficult, to the extent to which I feel like I’m missing something obvious. The best thing I have so far is:

driver.find_elements_by_xpath('//div[contains(text(), "' + text + '")]')

This is case-sensitive, however. The other thing I’ve tried is iterating through all the divs on the page, and checking the element.text property. However, every time you get a situation of the form:

<div class="outer"><div class="inner">My Button</div></div>

div.outer also has “My Button” as the text. To fix that, I’ve tried looking to see if div.outer is the parent of div.inner, but I couldn’t figure out how to do that (element.get_element_by_xpath(‘..’) returns an element’s parent, but it tests not equal to div.outer).

Also, iterating through all the elements on the page seems to be really slow, at least using the Chrome webdriver.

Ideas?

I asked (and answered) a more specific version here: How to get text of an element in Selenium WebDriver, without including child element text?

Try the following:

driver.find_elements_by_xpath("//*[contains(text(), 'My Button')]")

You could try an XPath expression like:

'//div[contains(text(), "{0}") and @class="inner"]'.format(text)

You can also use it with Page Object Pattern, e.g:

Try this code:

@FindBy(xpath = "//*[contains(text(), 'Best Choice')]")
WebElement buttonBestChoice;

//* will be looking for any HTML tag. Where if some text is common for Button and div tag and if //* is categories it will not work as expected. If you need to select any specific then You can get it by declaring HTML Element tag. Like:

driver.find_element_by_xpath("//div[contains(text(),'Add User')]")
driver.find_element_by_xpath("//button[contains(text(),'Add User')]")

Interestingly virtually all answers revolve around xpath’s function contains(), neglecting the fact it is case sensitive – contrary to OP’s ask.
If you need case insensitivity, that is achievable in xpath 1.0 (the version contemporary browsers support), though it’s not pretty – by using the translate() function. It substitutes a source character to its desired form, by using a translation table.

Constructing a table of all upper case characters will effectively transform the node’s text to its lower() form – allowing case-insensitive matching (here’s just the prerogative):

[
  contains(
    translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'),
    'my button'
  )
]
# will match a source text like "mY bUTTon"

The full python call:

driver.find_elements_by_xpath("//*[contains(translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZЙ', 'abcdefghijklmnopqrstuvwxyzй'), 'my button')]")

Naturally this approach has its drawbacks – as given, it’ll work only for latin text; if you want to cover unicode characters – you’ll have to add them to the translation table. I’ve done that in the sample above – the last character is the Cyrillic symbol "Й".

And if we lived in a world where browsers supported xpath 2.0 and up (🤞, but not happening any time soon ☹️), we could having used the functions lower-case() (yet, not fully locale-aware), and matches (for regex searches, with case-insensitive ('i') flag).

In the HTML which you have provided:

<div>My Button</div>

The text My Button is the innerHTML and have no whitespaces around it so you can easily use text() as follows:

my_element = driver.find_element_by_xpath("//div[text()='My Button']")

Note: text() selects all text node children of the context node

Text with leading/trailing spaces

Incase the relevant text containing whitespaces either in the beginning:

<div>   My Button</div>

or at the end:

<div>My Button   </div>

or at both the ends:

<div> My Button </div>  

In these cases you have 2 options:

• You can use contains() function which determines whether the first argument string contains the second argument string and returns boolean true or false as follows:

  my_element = driver.find_element_by_xpath("//div[contains(., 'My Button')]")
  

• You can use normalize-space() function which strips leading and trailing white-space from a string, replaces sequences of whitespace characters by a single space, and returns the resulting string as follows:

  driver.find_element_by_xpath("//div[normalize-space()='My Button']]")
  

xpath for variable Text

Incase the text is a variable you can use:

foo= "foo_bar"
my_element = driver.find_element_by_xpath("//div[.='" + foo + "']")

wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[contains(text(), 'YourTextHere')]")));
assertNotNull(driver.findElement(By.xpath("//*[contains(text(), 'YourTextHere')]")));
String yourButtonName = driver.findElement(By.xpath("//*[contains(text(), 'YourTextHere')]")).getAttribute("innerText");
assertTrue(yourButtonName.equalsIgnoreCase("YourTextHere"));

Similar problem: Find <button>Advanced...</button>

Maybe this will give you some ideas (please transfer the concept from Java to Python):

wait.until(ExpectedConditions.elementToBeClickable(//
    driver.findElements(By.tagName("button")).stream().filter(i -> i.getText().equals("Advanced...")).findFirst().get())).click();

Use driver.find_elements_by_xpath and matches regex matching function for the case insensitive search of the element by its text.

driver.find_elements_by_xpath("//*[matches(.,'My Button', 'i')]")

Try this. It’s very easy:

driver.getPageSource().contains("text to search");

This really worked for me in Selenium WebDriver.