Here on StackOverflow, I’ve seen users reporting that they cannot click an element via selenium WebDriver “click” command and can work around it with a JavaScript click by executing a script.
Example in Python:
element = driver.find_element_by_id("myid")
driver.execute_script("arguments[0].click();", element)
Example in WebDriverJS/Protractor:
var elm = $("#myid");
browser.executeScript("arguments[0].click();", elm.getWebElement());
The Question:
Why is clicking “via JavaScript” works when a regular WebDriver click does not? When exactly is this happening and what is the downside of this workaround (if any)?
I personally used this workaround without fully understanding why I have to do it and what problems it can lead to.
Contrarily to what the currently accepted answer suggests, there’s nothing specific to PhantomJS when it comes to the difference between having WebDriver do a click and doing it in JavaScript.
The Difference
The essential difference between the two methods is common to all browsers and can be explained pretty simply:
WebDriver: When WebDriver does the click, it attempts as best as it can to simulate what happens when a real user uses the browser. Suppose you have an element A which is a button that says “Click me” and an element B which is a div element which is transparent but has its dimensions and zIndex set so that it completely covers A. Then you tell WebDriver to click A. WebDriver will simulate the click so that B receives the click first. Why? Because B covers A, and if a user were to try to click on A, then B would get the event first. Whether or not A would eventually get the click event depends on how B handles the event. At any rate, the behavior with WebDriver in this case is the same as when a real user tries to click on A.
JavaScript: Now, suppose you use JavaScript to do A.click(). This method of clicking does not reproduce what really happens when the user tries to click A. JavaScript sends the click event directly to A, and B will not get any event.
Why a JavaScript Click Works When a WebDriver Click Does Not?
As I mentioned above WebDriver will try to simulate as best it can what happens when a real user is using a browser. The fact of the matter is that the DOM can contain elements that a user cannot interact with, and WebDriver won’t allow you to click on these element. Besides the overlapping case I mentioned, this also entails that invisible elements cannot be clicked. A common case I see in Stack Overflow questions is someone who is trying to interact with a GUI element that already exists in the DOM but becomes visible only when some other element has been manipulated. This sometimes happens with dropdown menus: you have to first click on the button the brings up the dropdown before a menu item can be selected. If someone tries to click the menu item before the menu is visible, WebDriver will balk and say that the element cannot be manipulated. If the person then tries to do it with JavaScript, it will work because the event is delivered directly to the element, irrespective of visibility.
When Should You Use JavaScript for Clicking?
If you are using Selenium for testing an application, my answer to this question is “almost never”. By and large, your Selenium test should reproduce what a user would do with the browser. Taking the example of the drop down menu: a test should click on the button that brings up the drop down first, and then click on the menu item. If there is a problem with the GUI because the button is invisible, or the button fails to show the menu items, or something similar, then your test will fail and you’ll have detected the bug. If you use JavaScript to click around, you won’t be able to detect these bugs through automated testing.
I say “almost never” because there may be exceptions where it makes sense to use JavaScript. They should be very rare, though.
If you are using Selenium for scraping sites, then it is not as critical to attempt to reproduce user behavior. So using JavaScript to bypass the GUI is less of an issue.
The click executed by the driver tries to simulate the behavior of a real user as close as possible while the JavaScript HTMLElement.click() performs the default action for the click event, even if the element is not interactable.
The differences are:
The driver ensures that the element is visible by scrolling it into the view and checks that the element is interactable.
The driver will raise an error:
when the element on top at the coordinates of the click is not the targeted element or a descendant
when the element doesn’t have a positive size or if it is fully transparent
when the element is a disabled input or button (attribute/property disabled is true)
when the element has the mouse pointer disabled (CSS pointer-events is none)
A JavaScript HTMLElement.click() will always perform the default action or will at best silently fail if the element is a disabled.
The driver is expected to bring the element into focus if it is focusable.
A JavaScript HTMLElement.click() won’t.
The driver is expected to emit all the events (mousemove, mousedown, mouseup, click, …) just like like a real user.
A JavaScript HTMLElement.click() emits only the click event.
The page might rely on these extra events and might behave differently if they are not emitted.
These are the events emitted by the driver for a click with Chrome:
Note that some of the drivers are still generating untrusted events. This is the case with PhantomJS as of version 2.1.
The event emitted by a JavaScript .click()doesn’t have the coordinates of the click.
The properties clientX, clientY, screenX, screenY, layerX, layerY are set to 0. The page might rely on them and might behave differently.
It may be ok to use a JavaScript .click() to scrap some data, but it is not in a testing context. It defeats the purpose of the test since it doesn’t simulate the behavior of a user. So, if the click from the driver fails, then a real user will most likely also fail to perform the same click in the same conditions.
What makes the driver fail to click an element when we expect it to succeed?
The targeted element is not yet visible/interactable due to a delay or a transition effect.
Add a delay matching the duration of the animation/transition :
browser.sleep(250);
The targeted element ends-up covered by a floating element once scrolled into the view:
The driver automatically scrolls the element into the view to make it visible. If the page contains a floating/sticky element (menu, ads, footer, notification, cookie policy..), the element may end-up covered and will no longer be visible/interactable.
NOTE: let’s call ‘click’ is end-user click. ‘js click’ is click via JS
Why is clicking “via JavaScript” works when a regular WebDriver click does not?
There are 2 cases for this to happen:
I. If you are using PhamtomJS
Then this is the most common known behavior of PhantomJS . Some elements are sometimes not clickable, for example <div>. This is because PhantomJS was original made for simulating the engine of browsers (like initial HTML + CSS -> computing CSS -> rendering). But it does not mean to be interacted with as an end user’s way (viewing, clicking, dragging). Therefore PhamtomJS is only partially supported with end-users interaction.
WHY DOES JS CLICK WORK? As for either click, they are all mean click. It is like a gun with 1 barrel and 2 triggers. One from the viewport, one from JS. Since PhamtomJS great in simulating browser’s engine, a JS click should work perfectly.
II. The event handler of “click” got to bind in the bad period of time.
For example, we got a <div>
-> We do some calculation
-> then we bind event of click to the <div>.
-> Plus with some bad coding of angular (e.g. not handling scope’s cycle properly)
We may end up with the same result. Click won’t work, because WebdriverJS trying to click on the element when it has no click event handler.
WHY DOES JS CLICK WORK? Js click is like injecting js directly into the browser. Possible with 2 ways,
Fist is through devtools console (yes, WebdriverJS does communicate with devtools’ console).
Second is inject a <script> tag directly into HTML.
For each browser, the behavior will be different. But regardless, these methods are more complicating than clicking on the button. Click is using what already there (end-users click), js click is going through backdoor.
And for js click will appear to be an asynchronous task. This is related a with a kinda complex topic of ‘browser asynchronous task and CPU task scheduling‘ (read it a while back can’t find the article again). For short this will mostly result as js click will need to wait for a cycle of task scheduling of CPU and it will be ran a bit slower after the binding of the click event.
(You could know this case when you found the element sometimes clickable, sometimes not.
)
When exactly is this happening and what is the downside of this
workaround (if any)?
=> As mention above, both mean for one purpose, but about using which entrance:
Click: is using what providing by default of browser.
JS click: is going through backdoor.
=> For performance, it is hard to say because it relies on browsers. But generally:
Click: doesn’t mean faster but only signed higher position in schedule list of CPU execution task.
JS click: doesn’t mean slower but only it signed into the last position of schedule list of CPU task.
=> Downsides:
Click: doesn’t seem to have any downside except you are using PhamtomJS.
JS click: very bad for health. You may accidentally click on something that doesn’t there on the view. When you use this, make sure the element is there and available to view and click as the point of view of end-user.
P.S. if you are looking for a solution.
Using PhantomJS? I will suggest using Chrome headless instead. Yes, you can set up Chrome headless on Ubuntu. Thing runs just like Chrome but it only does not have a view and less buggy like PhantomJS.
Not using PhamtomJS but still having problems? I will suggest using ExpectedCondition of Protractor with browser.wait() (check this for more information)
(I want to make it short, but ended up badly. Anything related with theory is complicated to explain…)
My app makes a call to an API that returns a dictionary. I want to pass information from this dict to JavaScript in the view. I am using the Google Maps API in the JS, specifically, so I’d like to pass it a list of tuples with the long/lat information. I know that render_template will pass these variables to the view so they can be used in HTML, but how could I pass them to JavaScript in the template?
Think of it as a two-stage process: First, Jinja (the template engine Flask uses) generates your text output. This gets sent to the user who executes the JavaScript he sees. If you want your Flask variable to be available in JavaScript as an array, you have to generate an array definition in your output:
The ideal way to go about getting pretty much any Python object into a JavaScript object is to use JSON. JSON is great as a format for transfer between systems, but sometimes we forget that it stands for JavaScript Object Notation. This means that injecting JSON into the template is the same as injecting JavaScript code that describes the object.
Flask provides a Jinja filter for this: tojson dumps the structure to a JSON string and marks it safe so that Jinja does not autoescape it.
Using a data attribute on an HTML element avoids having to use inline scripting, which in turn means you can use stricter CSP rules for increased security.
Then access it in a static JavaScript file like so:
// Raw JavaScript
var geocode = JSON.parse(document.getElementById("mydiv").dataset.geocode);
// jQuery
var geocode = JSON.parse($("#mydiv").data("geocode"));
Just another alternative solution for those who want to pass variables to a script which is sourced using flask, I only managed to get this working by defining the variables outside and then calling the script as follows:
<script>
var myfileuri = "/static/my_csv.csv"
var mytableid = 'mytable';
</script>
<script type="text/javascript" src="/static/test123.js"></script>
If I input jinja variables in test123.js it doesn’t work and you will get an error.
{% for i in range(maintenance_next|length): %}
<labelid="maintenance_next_{{i}}"name="{{maintenance_next[i]}}"style="display: none;"></label><labelid="maintenance_block_time_{{i}}"name="{{maintenance_block_time[i]}}"style="display: none;"></label>
{% endfor%}
<script>var total_len ={{ total_len }};for(var i =0; i < total_len; i++){var tm1 = document.getElementById("maintenance_next_"+ i).getAttribute("name");var tm2 = document.getElementById("maintenance_block_time_"+ i).getAttribute("name");//Do what you need to do with tm1 and tm2.
console.log(tm1);
console.log(tm2);}</script>
Well, I have a tricky method for this job. The idea is as follow-
Make some invisible HTML tags like <label>, <p>, <input> etc. in HTML body and make a pattern in tag id, for example, use list index in tag id and list value at tag class name.
Here I have two lists maintenance_next[] and maintenance_block_time[] of the same length. I want to pass these two list’s data to javascript using the flask. So I take some invisible label tag and set its tag name is a pattern of list index and set its class name as value at index.
{% for i in range(maintenance_next|length): %}
<label id="maintenance_next_{{i}}" name="{{maintenance_next[i]}}" style="display: none;"></label>
<label id="maintenance_block_time_{{i}}" name="{{maintenance_block_time[i]}}" style="display: none;"></label>
{% endfor%}
After this, I retrieve the data in javascript using some simple javascript operation.
<script>
var total_len = {{ total_len }};
for (var i = 0; i < total_len; i++) {
var tm1 = document.getElementById("maintenance_next_" + i).getAttribute("name");
var tm2 = document.getElementById("maintenance_block_time_" + i).getAttribute("name");
//Do what you need to do with tm1 and tm2.
console.log(tm1);
console.log(tm2);
}
</script>
回答 8
一些js文件来自网络或库,它们不是您自己编写的。他们获得的代码如下所示:
var queryString = document.location.search.substring(1);varparams=PDFViewerApplication.parseQueryString(queryString);var file ='file'inparams?params.file : DEFAULT_URL;
Some js files come from the web or library, they are not written by yourself.
The code they get variable like this:
var queryString = document.location.search.substring(1);
var params = PDFViewerApplication.parseQueryString(queryString);
var file = 'file' in params ? params.file : DEFAULT_URL;
This method makes js files unchanged(keep independence), and pass variable correctly!
There were no real associative arrays in Javascript until 2015 (release of ECMAScript 6). Since then you can use the Map object as Robocat states. Look up the details in MDN. Example:
let map = new Map();
map.set('key', {'value1', 'value2'});
let values = map.get('key');
Without support for ES6 you can try using objects:
var x = new Object();
x["Key"] = "Value";
However with objects it is not possible to use typical array properties or methods like array.length. At least it is possible to access the “object-array” in a for-in-loop.
var dict =newMap();
dict.set("foo","bar");//returns "bar"
dict.get("foo");
与javascript的普通对象不同,它允许任何对象作为键:
var foo ={};var bar ={};var dict =newMap();
dict.set(foo,"Foo");
dict.set(bar,"Bar");//returns "Bar"
dict.get(bar);//returns "Foo"
dict.get(foo);//returns undefined, as {} !== foo and {} !== bar
dict.get({});
I realize this is an old question, but it pops up in Google when you search for ‘javascript dictionaries’, so I’d like to add to the above answers that in ECMAScript 6, the official Map object has been introduced, which is a dictionary implementation:
var dict = new Map();
dict.set("foo", "bar");
//returns "bar"
dict.get("foo");
Unlike javascript’s normal objects, it allows any object as a key:
var foo = {};
var bar = {};
var dict = new Map();
dict.set(foo, "Foo");
dict.set(bar, "Bar");
//returns "Bar"
dict.get(bar);
//returns "Foo"
dict.get(foo);
//returns undefined, as {} !== foo and {} !== bar
dict.get({});
回答 3
在JS中创建了一个简单的字典:
functionJSdict(){this.Keys=[];this.Values=[];}// Check if dictionary extensions aren't implemented yet.// Returns value of a keyif(!JSdict.prototype.getVal){JSdict.prototype.getVal =function(key){if(key ==null){return"Key cannot be null";}for(var i =0; i <this.Keys.length; i++){if(this.Keys[i]== key){returnthis.Values[i];}}return"Key not found!";}}// Check if dictionary extensions aren't implemented yet.// Updates value of a keyif(!JSdict.prototype.update){JSdict.prototype.update =function(key, val){if(key ==null|| val ==null){return"Key or Value cannot be null";}// Verify dict integrity before each operationif(keysLength != valsLength){return"Dictionary inconsistent. Keys length don't match values!";}var keysLength =this.Keys.length;var valsLength =this.Values.length;var flag =false;for(var i =0; i < keysLength; i++){if(this.Keys[i]== key){this.Values[i]= val;
flag =true;break;}}if(!flag){return"Key does not exist";}}}// Check if dictionary extensions aren't implemented yet.// Adds a unique key value pairif(!JSdict.prototype.add){JSdict.prototype.add=function(key, val){// Allow only strings or numbers as keysif(typeof(key)=="number"||typeof(key)=="string"){if(key ==null|| val ==null){return"Key or Value cannot be null";}if(keysLength != valsLength){return"Dictionary inconsistent. Keys length don't match values!";}var keysLength =this.Keys.length;var valsLength =this.Values.length;for(var i =0; i < keysLength; i++){if(this.Keys[i]== key){return"Duplicate keys not allowed!";}}this.Keys.push(key);this.Values.push(val);}else{return"Only number or string can be key!";}}}// Check if dictionary extensions aren't implemented yet.// Removes a key value pairif(!JSdict.prototype.remove){JSdict.prototype.remove=function(key){if(key ==null){return"Key cannot be null";}if(keysLength != valsLength){return"Dictionary inconsistent. Keys length don't match values!";}var keysLength =this.Keys.length;var valsLength =this.Values.length;var flag =false;for(var i =0; i < keysLength; i++){if(this.Keys[i]== key){this.Keys.shift(key);this.Values.shift(this.Values[i]);
flag =true;break;}}if(!flag){return"Key does not exist";}}}
上面的实现现在可以用于模拟字典,如下所示:
var dict =newJSdict();
dict.add(1,"one")
dict.add(1,"one more")"Duplicate keys not allowed!"
dict.getVal(1)"one"
dict.update(1,"onne")
dict.getVal(1)"onne"
dict.remove(1)
dict.getVal(1)"Key not found!"
function JSdict() {
this.Keys = [];
this.Values = [];
}
// Check if dictionary extensions aren't implemented yet.
// Returns value of a key
if (!JSdict.prototype.getVal) {
JSdict.prototype.getVal = function (key) {
if (key == null) {
return "Key cannot be null";
}
for (var i = 0; i < this.Keys.length; i++) {
if (this.Keys[i] == key) {
return this.Values[i];
}
}
return "Key not found!";
}
}
// Check if dictionary extensions aren't implemented yet.
// Updates value of a key
if (!JSdict.prototype.update) {
JSdict.prototype.update = function (key, val) {
if (key == null || val == null) {
return "Key or Value cannot be null";
}
// Verify dict integrity before each operation
if (keysLength != valsLength) {
return "Dictionary inconsistent. Keys length don't match values!";
}
var keysLength = this.Keys.length;
var valsLength = this.Values.length;
var flag = false;
for (var i = 0; i < keysLength; i++) {
if (this.Keys[i] == key) {
this.Values[i] = val;
flag = true;
break;
}
}
if (!flag) {
return "Key does not exist";
}
}
}
// Check if dictionary extensions aren't implemented yet.
// Adds a unique key value pair
if (!JSdict.prototype.add) {
JSdict.prototype.add = function (key, val) {
// Allow only strings or numbers as keys
if (typeof (key) == "number" || typeof (key) == "string") {
if (key == null || val == null) {
return "Key or Value cannot be null";
}
if (keysLength != valsLength) {
return "Dictionary inconsistent. Keys length don't match values!";
}
var keysLength = this.Keys.length;
var valsLength = this.Values.length;
for (var i = 0; i < keysLength; i++) {
if (this.Keys[i] == key) {
return "Duplicate keys not allowed!";
}
}
this.Keys.push(key);
this.Values.push(val);
}
else {
return "Only number or string can be key!";
}
}
}
// Check if dictionary extensions aren't implemented yet.
// Removes a key value pair
if (!JSdict.prototype.remove) {
JSdict.prototype.remove = function (key) {
if (key == null) {
return "Key cannot be null";
}
if (keysLength != valsLength) {
return "Dictionary inconsistent. Keys length don't match values!";
}
var keysLength = this.Keys.length;
var valsLength = this.Values.length;
var flag = false;
for (var i = 0; i < keysLength; i++) {
if (this.Keys[i] == key) {
this.Keys.shift(key);
this.Values.shift(this.Values[i]);
flag = true;
break;
}
}
if (!flag) {
return "Key does not exist";
}
}
}
The above implementation can now be used to simulate a dictionary as:
var dict = new JSdict();
dict.add(1, "one")
dict.add(1, "one more")
"Duplicate keys not allowed!"
dict.getVal(1)
"one"
dict.update(1, "onne")
dict.getVal(1)
"onne"
dict.remove(1)
dict.getVal(1)
"Key not found!"
This is just a basic simulation.
It can be further optimized by implementing a better running time algorithm to work in atleast O(nlogn) time complexity or even less. Like merge/quick sort on arrays and then some B-search for lookups.
I Didn’t give a try or searched about mapping a hash function in JS.
Also, Key and Value for the JSdict obj can be turned into private variables to be sneaky.
Hope this helps!
EDIT >>
After implementing the above, I personally used the JS objects as associative arrays that are available out-of-the-box.
However, I would like to make a special mention about two methods that actually proved helpful to make it a convenient hashtable experience.
Use JavaScript objects. You can access their properties like keys in a dictionary. This is the foundation of JSON. The syntax is similar to Python dictionaries. See: JSON.org
//Constructorvar dict =newDict(overwrite:Boolean);//If overwrite, allows over-writing of duplicate keys,//otherwise, will not add duplicate keys to dictionary.
dict.put(key,value);//Add a pair
dict.get(key);//Get value from key
dict.remove(key);//Remove pair by key
dict.clearAll(value);//Remove all pairs with this value
dict.iterate(function(key,value){//Send all pairs as arguments to this function:
console.log(key+' is key for '+value);});
dict.get(key);//Get value from key
If you didn’t know, the AS3 dictionary allows you to use any object as the key, as opposed to just strings. They come in very handy once you’ve found a use for them.
It’s not as fast as a native object would be, but I’ve not found any significant problems with it in that respect.
API:
//Constructor
var dict = new Dict(overwrite:Boolean);
//If overwrite, allows over-writing of duplicate keys,
//otherwise, will not add duplicate keys to dictionary.
dict.put(key, value);//Add a pair
dict.get(key);//Get value from key
dict.remove(key);//Remove pair by key
dict.clearAll(value);//Remove all pairs with this value
dict.iterate(function(key, value){//Send all pairs as arguments to this function:
console.log(key+' is key for '+value);
});
dict.get(key);//Get value from key
2018: this answer keeps getting upvotes, so here’s an update. The code below is obsolete, but luckily ES6 standardized generators and the yield keyword, and they are universally supported across platforms. An example of the lazy range() using yield can be found here.
In addition to what’s already said, Javascript 1.7+ provides support for iterators and generators which can be used to create a lazy, memory-efficient version of range, simlar to xrange in Python2:
function range(low, high) {
return {
__iterator__: function() {
return {
next: function() {
if (low > high)
throw StopIteration;
return low++;
}
}
}
}
}
for (var i in range(3, 5))
console.log(i); // 3,4,5
A port of the range function from Python 2 is provided by the underscore.js and lodash utility libraries (along with many other useful tools). Examples copied from the underscore docs:
The following is a natural adaption of Python’s range() function to JavaScript:
// Generate range from start (inclusive) to stop (exclusive):
function* range(start, stop, step = 1) {
if (stop === undefined) [start, stop] = [0, start];
if (step > 0) while (start < stop) yield start, start += step;
else if (step < 0) while (start > stop) yield start, start += step;
else throw new RangeError('range() step argument invalid');
}
// Examples:
console.log([...range(3)]); // [0, 1, 2]
console.log([...range(0, 3)]); // [0, 1, 2]
console.log([...range(0, 3, -1)]);// []
console.log([...range(0, 0)]); // []
console.log([...range(-3)]); // []
console.log([...range(-3, 0)]); // [-3, -2, -1]
It supports any argument which can be compared to 0 and stop and can be incremented by step. It behaves identical to the Python version when used with numbers not exceeding Number.MAX_SAFE_INTEGER.
In contrast to @Tadeck’s, @Volv’s and @janka102’s answer which return [], undefined or enter an infinite loop when step evaluates to 0 or NaN, this generator function throws an exception similar to Python’s behavior.
回答 10
使用ES6默认参数进一步完善。
let range = function*(start = 0, stop, step = 1) {
let cur = (stop === undefined) ? 0 : start;
let max = (stop === undefined) ? start : stop;
for (let i = cur; step < 0 ? i > max : i < max; i += step)
yield i
}
let range = function*(start = 0, stop, step = 1) {
let cur = (stop === undefined) ? 0 : start;
let max = (stop === undefined) ? start : stop;
for (let i = cur; step < 0 ? i > max : i < max; i += step)
yield i
}
import {range} from 'pythonic';
// ...const results = range(5).map(wouldBeInvokedFiveTimes);
// `results` is now an array containing elements from// 5 calls to wouldBeInvokedFiveTimes
pythonic mimics the Python range behaviour best it can using JS’ generators (yield), supporting both the range(stop) and range(start, stop, step) use cases. In addition, pythonic‘s range function returns an Iterator object similar to Python that supports map and filter, so one could do fancy one-liners like:
import {range} from 'pythonic';
// ...
const results = range(5).map(wouldBeInvokedFiveTimes);
// `results` is now an array containing elements from
// 5 calls to wouldBeInvokedFiveTimes
Is there a function in JavaScript similar to Python’s range()?
All of the solutions here are referring to Python 2’s range (probably because of the code example you gave). However in Python 3, the range() method returns an iterator. JavaScript also has iterators and they’re more space efficient than generating the whole array and storing it in memory.
So the more accurate representation of Python 3’s range(n) function is Array(n).keys().
For example:
for (let i of Array(n).keys()) {
console.log(i) // 0, 1, 2, 3, ..., n
}
One more example (which has already been covered in the other answers). Converting the iterator to an array (ES6):
Still no built-in function that is equivalent to range(), but with the most recent version – ES2015 – you can build your own implementation. Here’s a limited version of it. Limited because it doesn’t take into account the step parameter. Just min, max.
const range = (min = null, max = null) =>
Array.from({length:max ? max - min : min}, (v,k) => max ? k + min : k)
This is accomplished by the Array.from method able to build an array from any object that has a length property. So passing in a simple object with just the length property will create an ArrayIterator that will yield length number of objects.
回答 16
这是我的首选方式。它允许您像在Python中一样指定一个或两个输入。
function range(start, end) {
return Array.from(Array(end||start).keys()).slice(!!end*start)
}
functionrange(start, stop, step = 1)
{
// This will make the function behave as range(stop)if(arguments.length === 1)
{
return [...Array(arguments[0]).keys()]
}
// Adjusts step to go towards the stop valueif((start > stop && !(step < 0)) ||
(start < stop && !(step > 0)))
{
step *= -1
}
let returnArray = []
// Checks if i is in the interval between start and stop no matter if stop// is lower than start or vice-versafor(let i = start; (i-start)*(i-stop) <= 0; i += step)
{
returnArray.push(i)
}
return returnArray
}
Is there a function in JavaScript similar to Python’s range()?
As answered before: no, there’s not. But you can make your own.
I believe this is an interesting approach for ES6. It works very similar to Python 2.7 range(), but it’s much more dynamic.
function range(start, stop, step = 1)
{
// This will make the function behave as range(stop)
if(arguments.length === 1)
{
return [...Array(arguments[0]).keys()]
}
// Adjusts step to go towards the stop value
if((start > stop && !(step < 0)) ||
(start < stop && !(step > 0)))
{
step *= -1
}
let returnArray = []
// Checks if i is in the interval between start and stop no matter if stop
// is lower than start or vice-versa
for(let i = start; (i-start)*(i-stop) <= 0; i += step)
{
returnArray.push(i)
}
return returnArray
}
This function can behave in three different ways (just like Python’s range()):
Note that instead of iterating over the array with the in operator (like python), you have to use of. Thus the i variable assumes the value, and not the index, of the array’s element.
for(let i of range(5))
{
// do something with i...
}
I’m using Selenium2 for some automated tests of my website, and I’d like to be able to get the return value of some Javascript code. If I have a foobar() Javascript function in my webpage and I want to call that and get the return value into my Python code, what can I call to do that?
You can return values even if you don’t have your snipped of code written as a function like in the below example code, by just adding return var; at the end where var is the variable you want to return.
result = driver.execute_script('''
cells = document.querySelectorAll('a');
URLs = [];
[].forEach.call(cells, function (el) {
URLs.push(el.href)
});
return URLs
''')
result will contain the array that is in URLs this case.
I have recently been learning Python and am dipping my hand into building a web-scraper. It’s nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.
Most of the issues are solvable and I’m having a good little mess around. However I’m hitting a massive hurdle over one issue. If a site loads a table of horses and lists current betting prices this information is not in any source file. The clue is that this data is live sometimes, with the numbers being updated obviously from some remote server. The HTML on my PC simply has a hole where their servers are pushing through all the interesting data that I need.
Now my experience with dynamic web content is low, so this thing is something I’m having trouble getting my head around.
I think Java or Javascript is a key, this pops up often.
The scraper is simply a odds comparison engine. Some sites have APIs but I need this for those that don’t. I’m using the scrapy library with Python 2.7
I do apologize if this question is too open-ended. In short, my question is: how can scrapy be used to scrape this dynamic data so that I can use it? So that I can scrape this betting odds data in real-time?
Webkit based browsers (like Google Chrome or Safari) has built-in developer tools. In Chrome you can open it Menu->Tools->Developer Tools. The Network tab allows you to see all information about every request and response:
In the bottom of the picture you can see that I’ve filtered request down to XHR – these are requests made by javascript code.
Tip: log is cleared every time you load a page, at the bottom of the picture, the black dot button will preserve log.
After analyzing requests and responses you can simulate these requests from your web-crawler and extract valuable data. In many cases it will be easier to get your data than parsing HTML, because that data does not contain presentation logic and is formatted to be accessed by javascript code.
Firefox has similar extension, it is called firebug. Some will argue that firebug is even more powerful but I like the simplicity of webkit.
Here is a simple example of scrapy with an AJAX request. Let see the site rubin-kazan.ru.
All messages are loaded with an AJAX request. My goal is to fetch these messages with all their attributes (author, date, …):
When I analyze the source code of the page I can’t see all these messages because the web page uses AJAX technology. But I can with Firebug from Mozilla Firefox (or an equivalent tool in other browsers) to analyze the HTTP request that generate the messages on the web page:
It doesn’t reload the whole page but only the parts of the page that contain messages. For this purpose I click an arbitrary number of page on the bottom:
And I observe the HTTP request that is responsible for message body:
After finish, I analyze the headers of the request (I must quote that this URL I’ll extract from source page from var section, see the code below):
And the form data content of the request (the HTTP method is “Post”):
And the content of response, which is a JSON file:
Which presents all the information I’m looking for.
From now, I must implement all this knowledge in scrapy. Let’s define the spider for this purpose:
from scrapy.contrib.spiders importCrawlSpider,Rulefrom scrapy.contrib.linkextractors.sgml importSgmlLinkExtractorfrom scrapy.selector importHtmlXPathSelectorfrom scrapy.http importRequestfrom selenium import selenium
classSeleniumSpider(CrawlSpider):
name ="SeleniumSpider"
start_urls =["http://www.domain.com"]
rules =(Rule(SgmlLinkExtractor(allow=('\.html',)), callback='parse_page',follow=True),)def __init__(self):CrawlSpider.__init__(self)self.verificationErrors =[]self.selenium = selenium("localhost",4444,"*chrome","http://www.domain.com")self.selenium.start()def __del__(self):self.selenium.stop()printself.verificationErrors
CrawlSpider.__del__(self)def parse_page(self, response):
item =Item()
hxs =HtmlXPathSelector(response)#Do some XPath selection with Scrapy
hxs.select('//div').extract()
sel =self.selenium
sel.open(response.url)#Wait for javscript to load in Selenium
time.sleep(2.5)#Do some crawling of javascript created content with Selenium
sel.get_text("//div")yield item
# Snippet imported from snippets.scrapy.org (which no longer works)# author: wynbennett# date : Jun 21, 2011
Many times when crawling we run into problems where content that is rendered on the page is generated with Javascript and therefore scrapy is unable to crawl for it (eg. ajax requests, jQuery craziness).
However, if you use Scrapy along with the web testing framework Selenium then we are able to crawl anything displayed in a normal web browser.
Some things to note:
You must have the Python version of Selenium RC installed for this to work, and you must have set up Selenium properly. Also this is just a template crawler. You could get much crazier and more advanced with things but I just wanted to show the basic idea. As the code stands now you will be doing two requests for any given url. One request is made by Scrapy and the other is made by Selenium. I am sure there are ways around this so that you could possibly just make Selenium do the one and only request but I did not bother to implement that and by doing two requests you get to crawl the page with Scrapy too.
This is quite powerful because now you have the entire rendered DOM available for you to crawl and you can still use all the nice crawling features in Scrapy. This will make for slower crawling of course but depending on how much you need the rendered DOM it might be worth the wait.
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
from selenium import selenium
class SeleniumSpider(CrawlSpider):
name = "SeleniumSpider"
start_urls = ["http://www.domain.com"]
rules = (
Rule(SgmlLinkExtractor(allow=('\.html', )), callback='parse_page',follow=True),
)
def __init__(self):
CrawlSpider.__init__(self)
self.verificationErrors = []
self.selenium = selenium("localhost", 4444, "*chrome", "http://www.domain.com")
self.selenium.start()
def __del__(self):
self.selenium.stop()
print self.verificationErrors
CrawlSpider.__del__(self)
def parse_page(self, response):
item = Item()
hxs = HtmlXPathSelector(response)
#Do some XPath selection with Scrapy
hxs.select('//div').extract()
sel = self.selenium
sel.open(response.url)
#Wait for javscript to load in Selenium
time.sleep(2.5)
#Do some crawling of javascript created content with Selenium
sel.get_text("//div")
yield item
# Snippet imported from snippets.scrapy.org (which no longer works)
# author: wynbennett
# date : Jun 21, 2011
classSpider(CrawlSpider):# define unique name of spider
name ="spider"
start_urls =["https://www.url.de"]def parse(self, response):# initialize items
item =CrawlerItem()# store data as items
item["js_enabled"]= response.body.decode("utf-8")
Another solution would be to implement a download handler or download handler middleware. (see scrapy docs for more information on downloader middleware) The following is an example class using selenium with headless phantomjs webdriver:
1) Define class within the middlewares.py script.
from selenium import webdriver
from scrapy.http import HtmlResponse
class JsDownload(object):
@check_spider_middleware
def process_request(self, request, spider):
driver = webdriver.PhantomJS(executable_path='D:\phantomjs.exe')
driver.get(request.url)
return HtmlResponse(request.url, encoding='utf-8', body=driver.page_source.encode('utf-8'))
2) Add JsDownload() class to variable DOWNLOADER_MIDDLEWARE within settings.py:
3) Integrate the HTMLResponse within your_spider.py. Decoding the response body will get you the desired output.
class Spider(CrawlSpider):
# define unique name of spider
name = "spider"
start_urls = ["https://www.url.de"]
def parse(self, response):
# initialize items
item = CrawlerItem()
# store data as items
item["js_enabled"] = response.body.decode("utf-8")
Optional Addon:
I wanted the ability to tell different spiders which middleware to use so I implemented this wrapper:
Advantage:
The main advantage to implementing it this way rather than in the spider is that you only end up making one request. In A T’s solution for example: The download handler processes the request and then hands off the response to the spider. The spider then makes a brand new request in it’s parse_page function — That’s two requests for the same content.
I was using a custom downloader middleware, but wasn’t very happy with it, as I didn’t manage to make the cache work with it.
A better approach was to implement a custom download handler.
There is a working example here. It looks like this:
# encoding: utf-8
from __future__ import unicode_literals
from scrapy import signals
from scrapy.signalmanager import SignalManager
from scrapy.responsetypes import responsetypes
from scrapy.xlib.pydispatch import dispatcher
from selenium import webdriver
from six.moves import queue
from twisted.internet import defer, threads
from twisted.python.failure import Failure
class PhantomJSDownloadHandler(object):
def __init__(self, settings):
self.options = settings.get('PHANTOMJS_OPTIONS', {})
max_run = settings.get('PHANTOMJS_MAXRUN', 10)
self.sem = defer.DeferredSemaphore(max_run)
self.queue = queue.LifoQueue(max_run)
SignalManager(dispatcher.Any).connect(self._close, signal=signals.spider_closed)
def download_request(self, request, spider):
"""use semaphore to guard a phantomjs pool"""
return self.sem.run(self._wait_request, request, spider)
def _wait_request(self, request, spider):
try:
driver = self.queue.get_nowait()
except queue.Empty:
driver = webdriver.PhantomJS(**self.options)
driver.get(request.url)
# ghostdriver won't response when switch window until page is loaded
dfd = threads.deferToThread(lambda: driver.switch_to.window(driver.current_window_handle))
dfd.addCallback(self._response, driver, spider)
return dfd
def _response(self, _, driver, spider):
body = driver.execute_script("return document.documentElement.innerHTML")
if body.startswith("<head></head>"): # cannot access response header in Selenium
body = driver.execute_script("return document.documentElement.textContent")
url = driver.current_url
respcls = responsetypes.from_args(url=url, body=body[:100].encode('utf8'))
resp = respcls(url=url, body=body, encoding="utf-8")
response_failed = getattr(spider, "response_failed", None)
if response_failed and callable(response_failed) and response_failed(resp, driver):
driver.close()
return defer.fail(Failure())
else:
self.queue.put(driver)
return defer.succeed(resp)
def _close(self):
while not self.queue.empty():
driver = self.queue.get_nowait()
driver.close()
Suppose your scraper is called “scraper”. If you put the mentioned code inside a file called handlers.py on the root of the “scraper” folder, then you could add to your settings.py:
yes, Scrapy can scrap dynamic websites, website that are rendered through javaScript.
There are Two approaches to scrapy these kind of websites.
First,
you can use splash to render Javascript code and then parse the rendered HTML.
you can find the doc and project here Scrapy splash, git
Second,
As everyone is stating, by monitoring the network calls, yes, you can find the api call that fetch the data and mock that call in your scrapy spider might help you to get desired data.
I handle the ajax request by using Selenium and the Firefox web driver. It is not that fast if you need the crawler as a daemon, but much better than any manual solution. I wrote a short tutorial here for reference
When I render a page using the Django template renderer, I can pass in a dictionary variable containing various values to manipulate them in the page using {{ myVar }}.
Is there a way to access the same variable in Javascript (perhaps using the DOM, I don’t know how Django makes the variables accessible)? I want to be able to lookup details using an AJAX lookup based on the values contained in the variables passed in.
CAUTION Check ticket #17419 for discussion on adding similar tag into Django core and possible XSS vulnerabilities introduced by using this template tag with user generated data. Comment from amacneil discusses most of the concerns raised in the ticket.
I think the most flexible and handy way of doing this is to define a template filter for variables you want to use in JS code. This allows you to ensure, that your data is properly escaped and you can use it with complex data structures, such as dict and list. That’s why I write this answer despite there is an accepted answer with a lot of upvotes.
Here is an example of template filter:
// myapp/templatetags/js.py
from django.utils.safestring import mark_safe
from django.template import Library
import json
register = Library()
@register.filter(is_safe=True)
def js(obj):
return mark_safe(json.dumps(obj))
This template filters converts variable to JSON string. You can use it like so:
then when I want to use a variable in the javascript files, I create a DJdata dictionary and I add it to the context by a json : context['DJdata'] = json.dumps(DJdata)
For a dictionary, you’re best of encoding to JSON first. You can use simplejson.dumps() or if you want to convert from a data model in App Engine, you could use encode() from the GQLEncoder library.
回答 6
我面临着类似的问题,S.Lott建议的答案为我工作。
<scripttype="text/javascript">var a ="{{someDjangoVariable}}"</script>
I was facing simillar issue and answer suggested by S.Lott worked for me.
<script type="text/javascript">
var a = "{{someDjangoVariable}}"
</script>
However I would like to point out major implementation limitation here.
If you are planning to put your javascript code in different file and include that file in your template. This won’t work.
This works only when you main template and javascript code is in same file.
Probably django team can address this limitation.
I’ve been struggling with this too. On the surface it seems that the above solutions should work. However, the django architecture requires that each html file has its own rendered variables (that is, {{contact}} is rendered to contact.html, while {{posts}} goes to e.g. index.html and so on). On the other hand, <script> tags appear after the {%endblock%} in base.html from which contact.html and index.html inherit. This basically means that any solution including
<script type="text/javascript">
var myVar = "{{ myVar }}"
</script>
is bound to fail, because the variable and the script cannot co-exist in the same file.
The simple solution I eventually came up with, and worked for me, was to simply wrap the variable with a tag with id and later refer to it in the js file, like so:
// index.html
<div id="myvar">{{ myVar }}</div>
and then:
// somecode.js
var someVar = document.getElementById("myvar").innerHTML;
and just include <script src="static/js/somecode.js"></script> in base.html as usual.
Of course this is only about getting the content. Regarding security, just follow the other answers.
For a JavaScript object stored in a Django field as text, which needs to again become a JavaScript object dynamically inserted into on-page script, you need to use both escapejs and JSON.parse():
var CropOpts = JSON.parse("{{ profile.last_crop_coords|escapejs }}");
Django’s escapejs handles the quoting properly, and JSON.parse() converts the string back into a JS object.
Note, that if you want to pass a variable to an external .js script then you need to precede your script tag with another script tag that declares a global variable.
直通JS客户端:如果您希望在Azure Bot服务中使用Direct Line频道,并且没有使用Webchat客户端,则可以在您的自定义应用程序中使用Direct Line JS客户端。[Readme]
直接线路语音信道:我们正在将Bot框架和Microsoft的语音服务结合在一起,以提供一个通道,支持从客户端到BOT应用程序的双向流式语音和文本。若要注册,请向您的Azure Bot服务添加“Direct Line Speech”频道
为您的Bot-Direct Line App服务扩展提供更好的隔离:Direct Line App Service Extension可以作为VNET的一部分进行部署,使IT管理员能够更好地控制会话流量,并由于减少了跳数而改善了会话延迟。单击此处开始使用Direct Line App Service Extension。VNET允许您在Azure中创建自己的私有空间,并且对您的云网络至关重要,因为它提供隔离、分段和其他主要优势
I am pretty new to this whole MV* client-side framework frenzy. It doesn’t have to be AngularJS, but I picked it because it feels more natural to me than either Knockout, Ember or Backbone. Anyway what is the workflow like? Do people start with developing a client-side application in AngularJS and then hooking up the back-end to it?
Or the other way around by first building the back-end in Django, Flask, Rails and then attaching an AngularJS app to it? Is there a “right” way of doing it, or is it just a personal preference in the end?
I am also not sure whether to structure my project according to the Flask or AngularJS? community practices.
For example, Flask’s minitwit app is structured like so:
minitwit
|-- minitwit.py
|-- static
|-- css, js, images, etc...
`-- templates
|-- html files and base layout
AngularJS tutorial app is structured like this:
angular-phonecat
|-- app
`-- css
`-- img
`-- js
`-- lib
`-- partials
`-- index.html
|-- scripts
`-- node.js server and test server files
I could picture a Flask app by itself, and it’s fairly easy to see AngularJS app like ToDo List by itself but when it comes to using both of these technologies I don’t understand how they work together. It almost seems like I don’t need a server-side web-framework when you already have AngularJS, a simple Python web server will suffice. In the AngularJS to-do app for example they use MongoLab to talk to the database using Restful API. There was no need having a web framework on the back-end.
Maybe I am just awfully confused, and AngularJS is nothing more than a fancy jQuery library so I should use just like I would use jQuery in my Flask projects (assuming I change the AngularJS template syntax to something that doesn’t conflict with Jinja2). I hope my questions make some sense. I mainly work on the back-end and this client-side framework is an unknown territory for me.
And as btford mentioned, if you are doing an Angular app, you’ll want to focus on using Angular client-side templates and stay away from server-side templates. Using render_template(‘index.html’) will cause Flask to interpret your angular templates as jinja templates, so they won’t render correctly. Instead, you’ll want to do the following:
At this point, you haven’t yet constructed your RESTful API, so you can have your js controllers return predefined sample data (only a temporary setup). When you’re ready, implement the RESTful API and hook it up to your angular app with angular-resource.js.
EDIT: I put together an app template that, though a little more complex that what I’ve described above, illustrates how one could build an app with AngularJS + Flask, complete with communication between AngularJS and a simple Flask API. Here it is if you want to check it out: https://github.com/rxl/angular-flask
You are right that you probably don’t need a full server-side framework with AngularJS. It’s typically better to serve static HTML/CSS/JavaScript files, and provide a RESTful API for the back end for the client to consume. One thing that you should probably avoid is mixing server-side templates with AngularJS client-side templates.
If you want to use Flask to serve your files (might be overkill, but you can use it nonetheless) you would copy the contents of “app” from “angular-phonecat” into the “static” folder of “minitwit.”
AngularJS is more targeted at AJAX-like applications, whereas flask gives you the ability to do both the older-style web apps as well as create RESTful APIs. There are advantages and disadvantages to each approach, so it really depends what you want to do. If you give me some insights, I might be able to make further recommendations.
回答 2
John Lindquist的这个Jetbrains PyCharm官方视频(angular.js和jetbrains大师)是一个不错的起点,因为它显示了烧瓶中Web服务,数据库和angular.js的相互作用。
This official Jetbrains PyCharm video by John Lindquist (angular.js and jetbrains guru) is a nice starting point as it shows the interplay of webservice, database and angular.js within flask.
He builds a pinterest clone with flask, sqlalchemy, flask-restless and angular.js in less than 25 minutes.
edit: The new Angular2 style guide suggests a similar, if not the same structure in much more detail.
The answer below target large scale projects.
I have spend quite some time thinking and experimenting with several approaches so I can combine some server side framework (Flask with App Engine in my case) for back-end functionality along with a client side framework like Angular. Both answers are very good, but I would like to suggest a slightly different approach which (in my mind at least) scales in a more human way.
When you are implementing a TODO example, things are quite straight forward. When you start adding functionality though and small nice details for user experience improvement, its not difficult to get lost in chaos of styles, javascript etc..
My application started to grow quite big, so I had to take a step back and rethink. Initially an approach like suggested above would work, by having all the styles together and all JavaScript together, but its not modular and not easily maintainable.
What if we organized the client code per feature and not per file type:
If we build it like this, we can wrap every directory of ours in an angular module. And we have split our files in a nice way that we don’t have to go through irrelevant code when we are working with a specific feature.
A task runner like Grunt properly configured, will be able to find and concatenate and compile your files without much hassle.
Files related to flask goes under the server folder and files related to angularjs goes under the client folder. This way, it will be easier to change the backend or front-end. For example, you might want to switch from Flask to Django or AngularJS to ReactJS in the future.
I think it is important to determine at what end you want to do most of your data processing – front end or back end.
If it’s front end, then go with the angular workflow, which means your flask app will function as more of an api where an extension like flask-restful will come end.
But if like me, you are doing most work on the backend, then go with the flask structure and only plug angular ( or in my case vue.js) to build the front end (when neccessary)