We Keep Coding

Error whilst trying to use the .difference() function in Python Jupyter

Context
I am currently going through a course on webscraping. Upon getting to the module on scraping javascript, a function set_1.difference(set_2) was used to distinguish the old variables from the newly created variables. But when I did it, it brought up this error:
AttributeError: 'list' object has no attribute 'difference'
I searched online and stumbled on this website. But running the example on their own website brought up an error
Problem
Any reason why this is not working? I want to print the newly generated javascript links. Below is the code I am trying to run:
from requests_html import AsyncHTMLSession
session = AsyncHTMLSession()
r = await session.get('https://www.ons.gov.uk/economy/economicoutputandproductivity/output/datasets/economicactivityfasterindicatorsuk')
r.status_code
divs = r.html.find('div')
downloads = r.html.find('a')
urls = r.html.absolute_links
# Now need to render the javascript. Downloads chromium the first time we use it,
# It is a browser that has no GUI
await r.html.arender()
new_divs = r.html.find('div')
new_downloads = r.html.find('a')
new_urls = r.html.absolute_links
# Get only the newly created html
new_downloads.difference(downloads)

Don't know what the "r" object is, so can't verify your code but difference is a method of sets, not lists.
https://docs.python.org/3/library/stdtypes.html#frozenset.difference
This should do the trick: set(new_downloads).difference(downloads)

HERE-MAPS Altitude Information of Route in javascript

I am trying to get the gradient information of the calculated route from here maps.
According to the documentation I need to add:
returnElevation = True
to cause "all shapes inside routing response to consist of 3 values instead of 2"
this was done here:
jsonAttributes: 1,
linkAttributes: 'sh,sl,ds',
routeAttributes: 'waypoints,shape,boundingBox',
legAttributes: 'boundingBox,shape',
maneuverAttributes: 'po,sh,rn,sp,rs,sa,sq',
returnElevation=true,
representation: 'turnByTurn',
later I receive the data with this:
var geo = links.shape[0].split(",");
latitudes.push(parseFloat(geo[0]));
longitudes.push(parseFloat(geo[1]));
altitude.push(parseFloat(geo[2]));
For some reason, this approach does not work, any Idea what I might be doing wrong?

Try replacing returnElevation=true with returnElevation:true, i think returnElevation=true will work directly in REST API request however javascript needs it with a ":"

RangeError: index out of range when connect to cassandra databse in node js

I'm working on NodeJS project and I've tried to select data from table stored in cassandra database using cassandra-driver package,
I start the client connection using this line:
const cassandra = require('cassandra-driver');
const cassandraClient = new cassandra.Client({ contactPoints: ['192.168.0.253'], keyspace: 'test' });
sometimes when I call the url, the engine fetchs the result successfully, by the way sometimes raises this error:
{ [RangeError: index out of range]
coordinator: '192.168.0.253:9042',
query: 'SELECT * FROM table where hidden=false ALLOW FILTERING' }
this error usually raise when the cassandra table updated recently!
I don't know if this important, but there is another service connect to cassandra database and insert new data, is this related to problem? and how would I solve this error?
if the two services are the reason of this error, Are there any way to make cassandra table not blocked for (reading and writing) when there is active service write to cassandra table, mean no problem with ghost reading?
stack error:
{ [RangeError: index out of range]
coordinator: '192.168.0.253:9042',
query: 'SELECT * FROM table where hidden=false ALLOW FILTERING' }
RangeError: index out of range
at checkOffset (buffer.js:663:11)
at Buffer.readInt32BE (buffer.js:828:5)
at Function.Long.fromBuffer (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/types/index.js:466:25)
at Encoder.decodeLong (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/encoder.js:133:17)
at Encoder.decodeTimestamp (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/encoder.js:142:26)
at Encoder.decode (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/encoder.js:1202:18)
at Parser.parseRows (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/streams.js:377:36)
at Parser.parseResult (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/streams.js:335:10)
at Parser.parseBody (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/streams.js:185:19)
at Parser._transform (/home/l.alassadi/alarm-socket/node_modules/cassandra-driver/lib/streams.js:137:10)

TLDR fixed in cassandra-driver v3.6.0
RangeError is caused by a problem in cassandra-driver, specifically in how it decodes the data from db server. It appears in the following conditions:
The table contains a column with a map type
The map's value type is not nullable
The selected row has the said column
The value of the column (the map) has a key without a value (not quite sure how it's possible but it is)
Here are all nullable types:
text
ascii
varchar
custom
blob
Long story short, the driver does not handle the case when a field of a not-nullable type (like int) inside a map is serialized as an empty buffer. In this case the driver is unconditionally trying to read from the buffer the amount of bytes equivalent to the type's size and fails with the said error.
Having this investigated, I've submitted a fix to cassandra-driver. It has been accepted and released in 3.6.0.

I received the same error while doing a fetch from table. Since the error is at Buffer.readInt32BE, guessed that it's while parsing fields which could be of numeric type.
Looking at the stacktrace, went to the function Function.Long.fromBuffer and printed the bytes argument that gets passed. Now, looking at my query which had two fields of Timestamp type which can even be expressed as Integer type.
My query goes here select item_id, created_date, modified_date from mytable. Modified the library method to print the values which it encounters in encoder.js:
this.decodeLong = function (bytes) {
console.log("---->" + Long.fromBuffer(bytes));
return Long.fromBuffer(bytes);
};
Found that the first three values getting printed while executing my service method. Since it was the fourth item getting failed and my query had just two numeric types, concluded that the exception was during parsing of modified_date filed of second row.
Tried updating the cassandra-driver, the issue persisted. Deleted the row which was causing the issue and got this resolved. So, I guess, updating the field again to a different value should solve this. Looks like an inconsistency with the cassandra data.

Uploading/Downloading Byte Arrays with AngularJS and ASP.NET Web API

I have spent several days researching and working on a solution for uploading/downloading byte[]’s. I am close, but have one remaining issue that appears to be in my AngularJS code block.
There is a similar question on SO, but it has no responses. See https://stackoverflow.com/questions/23849665/web-api-accept-and-post-byte-array
Here is some background information to set the context before I state my problem.
I am attempting to create a general purpose client/server interface to upload and download byte[]’s, which are used as part of a proprietary server database.
I am using TypeScript, AngularJS, JavaScript, and Bootstrap CSS on the client to create a single page app (SPA).
I am using ASP.NET Web API/C# on the server.
The SPA is being developed to replace an existing product that was developed in Silverlight so it is constrained to existing system requirements. The SPA also needs to target a broad range of devices (mobile to desktop) and major OSs.
With the help of several online resources (listed below), I have gotten most of my code working. I am using an asynchronous multimedia formatter for byte[]’s from the Byte Rot link below.
http://byterot.blogspot.com/2012/04/aspnet-web-api-series-part-5.html
Returning binary file from controller in ASP.NET Web API
I am using a jpeg converted to a Uint8Array as my test case on the client.
The actual system byte arrays will contain mixed content compacted into predefined data packets. However, I need to be able to handle any valid byte array so an image is a valid test case.
The data is transmitted to the server correctly using the client and server code shown below AND the Byte Rot Formatter (NOT shown but available on their website).
I have verified that the jpeg is received properly on the server as a byte[] along with the string parameter metadata.
I have used Fiddler to verify that the correct response is sent back to the client.
The size is correct
The image is viewable in Fiddler.
My problem is that the server response in the Angular client code shown below is not correct.
By incorrect, I mean the wrong size (~10K versus ~27.5K) and it is not recognized as a valid value for the UintArray constructor. Visual Studio shows JFIF when I place the cursor over the returned “response” shown in the client code below, but there is no other visible indicator of the content.
/********************** Server Code ************************/
Added missing item to code after [FromBody]byte[]
public class ItemUploadController : ApiController{
[AcceptVerbs("Post")]
public HttpResponseMessage Upload(string var1, string var2, [FromBody]byte[] item){
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new MemoryStream(item);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
return result;
}
}
/***************** Example Client Code ********************/
The only thing that I have omitted from the code are the actual variable parameters.
$http({
url: 'api/ItemUpload/Upload',
method: 'POST',
headers: { 'Content-Type': 'application/octet-stream' },// Added per Byte Rot blog...
params: {
// Other params here, including string metadata about uploads
var1: var1,
var2: var2
},
data: new Uint8Array(item),
// arrybuffer must be lowecase. Once changed, it fixed my problem.
responseType: 'arraybuffer',// Added per http://www.html5rocks.com/en/tutorials/file/xhr2/
transformRequest: [],
})
.success((response, status) => {
if (status === 200) {
// The response variable length is about 10K, whereas the correct Fiddler size is ~27.5K.
// The error that I receive here is that the constructor argument in invalid.
// My guess is that I am doing something incorrectly with the AngularJS code, but I
// have implemented everything that I have read about. Any thoughts???
var unsigned8Int = new Uint8Array(response);
// For the test case, I want to convert the byte array back to a base64 encoded string
// before verifying with the original source that was used to generate the byte[] upload.
var b64Encoded = btoa(String.fromCharCode.apply(null, unsigned8Int));
callback(b64Encoded);
}
})
.error((data, status) => {
console.log('[ERROR] Status Code:' + status);
});
/****************************************************************/
Any help or suggestions would be greatly appreciated.
Thanks...
Edited to include more diagnostic data
First, I used the angular.isArray function to determine that the response value is NOT an array, which I think it should be.
Second, I used the following code to interrogate the response, which appears to be an invisible string. The leading characters do not seem to correspond to any valid sequence in the image byte array code.
var buffer = new ArrayBuffer(response.length);
var data = new Uint8Array(buffer);
var len = data.length, i;
for (i = 0; i < len; i++) {
data[i] = response[i].charCodeAt(0);
}
Experiment Results
I ran an experiment by creating byte array values from 0 - 255 on the server, which I downloaded. The AngularJS client received the first 128 bytes correctly (i.e., 0,1,...,126,127), but the remaining values were 65535 in Internet Explorer 11, and 65533 in Chrome and Firefox. Fiddler shows that 256 values were sent over the network, but there are only 217 characters received in the AngularJS client code. If I only use 0-127 as the server values, everything seems to work. I have no idea what can cause this, but the client response seems more in line with signed bytes, which I do not think is possible.
Fiddler Hex data from the server shows 256 bytes with the values ranging from 00,01,...,EF,FF, which is correct. As I mentioned earlier, I can return an image and view it properly in Fiddler, so the Web API server interface works for both POST and GET.
I am trying vanilla XMLHttpRequest to see I can get that working outside of the AngularJS environment.
XMLHttpRequest Testing Update
I have been able to confirm that vanilla XMLHttpRequest works with the server for the GET and is able to return the correct byte codes and the test image.
The good news is that I can hack around AngularJS to get my system working, but the bad news is that I do not like doing this. I would prefer to stay with Angular for all my client-side server communication.
I am going to open up a separate issue on Stack Overflow that only deals with the GET byte[] issues that I am have with AngularJS. If I can get a resolution, I will update this issue with the solution for historical purposes to help others.
Update
Eric Eslinger on Google Groups sent me a small code segment highlighting that responseType should be "arraybuffer", all lower case. I updated the code block above to show the lowercase value and added a note.
Thanks...

I finally received a response from Eric Eslinger on Google Group. He pointed out that he uses
$http.get('http://example.com/bindata.jpg', {responseType: 'arraybuffer'}).
He mentioned that the camelcase was probably significant, which it is. Changed one character and the entire flow is working now.
All credit goes to Eric Eslinger.

bokeh plot empty when rendering Flask template

I am trying to embed a bokeh plot in a webpage served by a simple Flask app, using the embed.autoload_server function that I picked up from looking over the bokeh embed examples on github. Everything seems to be working as expected on the python side of things, but the page renders without any data (even though the data is within the JS plot object). I do see the 5 bokeh plot manipulation buttons but I do not see the actual plot. After turning on the JS console I see that the i variable is being returned as undefined in the following statement (line 23512, bokeh.js):
i = this.get('dimension');
As a result, ranges[i] is also undefined, which is the error I'm getting in the console.
I can navigate the browser to the actual plot json and I see all the data as expected there, which is why I turned to the JS console to troubleshoot.
Any ideas would be very appreciated, my JS is pretty rusty at the moment. Is there a relationship between the attributes of the python "plot" objects and the JS "plot" objects? It seems like this is just an issue of my front end object missing the "dimension" attribute.
In response to the question, here is the code, it is pretty much lifted directly from the candlestick example code, but that was from a pull from several weeks ago, so it very well could be dated. I pulled again since and didn't revisit this code since there were no issues creating the plot data.
def candlestick():
store = pd.HDFStore('../data/dt_metastock.h5')
keys = [key for key in store.keys() if 'daily' in key]
df = store[keys[0]][:800]
#df['date'] = pd.to_datetime(df['date'])
mids = (df.open + df.close)/2
spans = abs(df.close-df.open)
inc = df.close > df.open
dec = df.open > df.close
w = 12*60*60*1000 # half day in ms
output_server("candlestick")
figure(tools="pan,wheel_zoom,box_zoom,reset,previewsave",
plot_width=1000, name="candlestick")
hold()
segment(df.idx, df.high, df.idx, df.low, color='black')
w = .5
rect(df.idx[inc].values, mids[inc], w, spans[inc], fill_color="#D5E1DD", line_color="black")
rect(df.idx[dec].values, mids[dec], w, spans[dec], fill_color="#F2583E", line_color="black")
curplot().title = keys[0]
xaxis().major_label_orientation = pi/4
grid().grid_line_alpha=0.3
tag = embed.autoload_server(curplot(), cursession())
return tag

Can you post the code of your plot? Recently, we have merged a new layout system and it seems to me that you are probably using and old way to set up the axes in your plot...