We Keep Coding

How to ensure webpage elements load correctly after selenium execute_script call before scraping

I have a project where I am attempting to scrape data from a database-type webpage, but find its load times are very unpredictable. The button clicks call JavaScript functions that change the buttons in the frame. I have developed a script that navigates through the scriptural book, then chapters, then verses, until it gets to the "bottom" level of the tree and returns information about which talks used that particular verse as a citation. The goal is to collect all the data about each verse cited, by whom, when, etc.
My current focus is to just get all the data downloaded, but that is proving to be tricky. I have tried to use browser.implicitly_wait to get the behaviour I am wanting, and have also tried WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "citationindex")) and that hasn't seemed to work either.
The current setup is shown below. It works as follows:
Selenium opens the base url in Chrome and calls the recursive function
dig_into_citations is called on the browser which recursively starts search through the links in the 'citationindex' or 'citationindex2' element.
The get_citationindex function is called first to determine which one the data is in, as it appears to switch back and forth, depending on what level of the tree you are at. If anyone knows why this is happening, that would be hugely helpful). This 'waits' for the citationindex element to load, which is determines by checking when link elements appear inside one of the two possible elements. Hacky but this seems to work most of the time.
Then the logic in dig_into_citations checks to see if we are at the base level yet, which is where the talks appear. If so, I'll collect the data somehow (not worried about the details now) but am currently just printing out for debugging purposes.
If not yet at the base level, then we are looking at either books, chapters, or verses and want to dig into each of those. I do this by running the javascript function within the onclick attribute of each of the button links (which has the form getFilter('1', '101') where the number of arguments depends on the level). I am passing the script through as well to store as the id for each talk.
The problem I am having is that call to execute_script basically just updates the current webpage with new buttons, and it's not clear how long that takes each time, so I'm putting a 1 second sleep in before each iteration. I find anything slower causes it to choke.
Like I say, I have tried the "wait-until" functionality in Selenium but it hasn't seemed to function as expected. I suspect that this is again because running the script in each case causes the elements inside the 'citationindex' (or 'citationindex2') function to update but the element itself is already there and already "loaded" so maybe that's why the wait-until stuff isn't working.
Any insights into how this process could be improved and made robust would be greatly appreciated. If I am going about this in the wrong way, it would also help to know if there is a simpler approach to get at all the data I'm looking for rather than this rickety recursion through button click hell. Thanks in advance.
from bs4 import BeautifulSoup
from selenium import webdriver
import time
def get_citationindex(browser):
"""
Gets the name of the citationindex element that contains the links of interest.
"""
CITATION_ID_NAMES = ['citationindex', 'citationindex2']
# Gets the HTML associated with both citation index elements and attempts to find links inside
def get_soup_list(browser):
citation_id_elems = [browser.find_element_by_id(x).get_attribute('innerHTML') for x in CITATION_ID_NAMES]
citation_id_soups = [BeautifulSoup(x, features="lxml") for x in citation_id_elems]
soup_list = [soup for soup in citation_id_soups if soup.find_all('a')]
return soup_list
# This runs until the page has actually loaded and the links are found inside the proper citationindex element
soup_list = []
while not soup_list:
soup_list = get_soup_list(browser) # ! this hack is not great
# the BeautifulSoup object of the citation index element is returned (i.e. for 'citationindex' or 'citationindex2')
citationindex_soup = soup_list[0]
return citationindex_soup
def dig_into_citations(browser, script=''):
"""
Recursive function that digs into the citations (through book, chapter, and verse) until it find the talk references.
It operates by calling the javascript functions on the page.
"""
soup = get_citationindex(browser)
# the desired links are the only ones with 'div' tags
links = [x for x in soup.find_all('a') if x.find('div')]
talks = [x for x in links if "getTalk" in x.get('onclick')]
header = soup.find(class_='volumetitle')
if talks:
for k in talks:
ref = k.find(class_='reference').text
title = k.find(class_='talktitle').text
print(script, header.text, ref, title)
else:
for k in links:
script = k.get('onclick')
browser.execute_script(script)
time.sleep(1) # ! this hack is also not great
dig_into_citations(browser, script)
# Run a test case for one particular book
browser = webdriver.Chrome()
browser.implicitly_wait(20)
browser.get('https://scriptures.byu.edu/#::fNYNY7267e401074')
dig_into_citations(browser)
Note I am running this with Python 3.8 with selenium=3.141.0 and beautifulsoup4=4.9.3 using Selenium driver for Chrome 87

How to one-way process html data for the view in CKEditor 5?

In the CKEditor view for authors I need to change links to files so that the session ID of the author gets attached. However in the actual content for normal users the specific user ID is added automatically. Therefore the authors ID must not be saved in the content the author edits with CKEditor, it just has to be there in the view while he edits so that he can see an image for example. On save the 'clean' link without any IDs need to be saved.
In CKEditor 5 there seem to be more possibilities to achieve such a one-way data filtering for example with
Conversions
the Editing Engine generally
the HtmlDataProcessor specifically
However I couldn't find a good example respectively an easy and clean approach to achieve this. (My tries turned out to become quite complicated and didn't work properly...) I'd guess this is a quite common use case so maybe I'm overlooking something. Is there a good solution to this?
Update 1: Example links would be:
"clean link" how it has to be saved but will never work:https://example.com/some-image.png
modified link for specific users in content (and how it has to be modified in ckeditor view for authors as well): https://example.com/some-image.png?sessionId=currentUsersSessionId
Update 2:
While I was working further with CKEditor I came across more things like this which simply are very unpleasant from a developers point of view. And it seems this is by design, since quote from a Contributor 'fredck':
[...] we want to bring the editor out of the "HTML Editor" thing, making it the perfect soluting for "quality content writing".
Implicitly this means, if you are a developer and you have advanced users with advanced use cases (which may be likely the case if you are on Stackoverflow) you are not the target audience and shouldn't use CKEditor in the first place.
You can read more about this for example in the discussion here (also it is about another feature): https://github.com/ckeditor/ckeditor5/issues/592

To modify downloaded links you can write a custom downcast converter, which modifies obtained href.
Here is a working sample which adds the current timestamp to URLs:
https://codepen.io/msamsel/pen/zVMvZN?editors=1010
editor.conversion.for( 'dataDowncast' ).add( dispatcher => {
dispatcher.on(
'attribute:linkHref',
( evt, data, conversionApi ) => {
if ( !conversionApi.consumable.test( data.item, 'attribute:linkHref' ) ) {
return;
}
if ( data.attributeNewValue ) {
data.attributeNewValue += `#time=${ ( new Date() ).getTime() }`;
}
},
{ priority: 'high' }
);
} );
Few words how it works.
There is created listener which reacts on attribute:linkHref changes (it's fired only when data are obtained anyway because it's dataDowncast). Listeners fires with 'high' priority to change URL before the actual Link plugin will create an output. First is checked if the given model element is not consumed, but without consuming it, because we want to preserve native behavior which will process this same element again. The attribute value is extended with a timestamp, what finish this listener. After that, the native behaviour is fired, which has 'normal' priority.
A similar approach was used to implement custom link attributes. More about dispatcher and conversion process might be found here:
https://ckeditor.com/docs/ckeditor5/latest/framework/guides/architecture/editing-engine.html#conversion
https://ckeditor.com/docs/ckeditor5/latest/api/module_engine_conversion_downcastdispatcher-DowncastDispatcher.html

How to efficiently have many readonly Monaco Diff Views on one page?

My understanding is that Monaco is optimized for editing, and for having showing one file at a time, with a fixed size editor that has its own scroll bar.
Instead I am trying to build one page with the diffs of multiple files below each other
allowing showing/hiding each file, up to ~100 files
hiding portions of the file that have not changed (and allowing to show them as context if desired)
not have one scrollbar per file, but one for the entire page
the files are usually view-only, but editing should be supported for one file at a time
I realize this is quite the departure from what Monaco was built for, but in the end it seems as if the same viewport and virtual rendering tricks would apply, so maybe it is somehow possible?
I tried creating one Monaco instance per file, but that starts getting really sluggish around 30 instances.
One pretty ugly workaround might be to have a single Monaco instance, concat all the files, and then work with ViewZones, custom line number providers and code folding providers to achieve the impression of multiple files. Is that as crazy as it sounds, or might that actually work?
Any other suggestions? Why does IStandaloneDiffEditor have standalone in the name? Does that mean there is another way to create many diff editors that is more efficient?

Citate from your question:
I tried creating one Monaco instance per file, but that starts getting really sluggish around 30 instances.
The solution for your question
As you mentioned the perfomance was sluggish. This is because your server or may be your client has not enough memory. You have to add more memory to the server or may be to the client for more perfomance. Because I do not have enough information I can not say it is the server or the client. But this method is not efficient.
Citate from your question:
Why does IStandaloneDiffEditor have standalone in the name? Does that mean there is another way to create many diff editors that is more efficient?
Nothing from it. In Wikipedia I found the answer what standalone means:
Standalone software may refer to:
Computer software that can work offline, i.e. does not necessarily require network connection to function.
Software that is not a part of some bundled software.
A program that run as a separate computer process, not an add-on of an existing process.
Standalone program, a program that does not require operating system's services to run.
A portable application, which can be run without the need for installation procedure.
This means that standalone has nothing to do with single instance and you could have multiple instances of this editor. But you have to have more memory on your computer(s) to create 100 instances from this editor. And this it not efficient because you have 100 big JavaScript objects more in your memory.
On other services for display the difference between changed files they make it with DOM objects only or with DOM objects + one big instance from big JavaScript object which creates this objects, but not additional 100 big instances from big JavaScript objects.
Accordingly to this princip in this case you could use code from my recommended solution below and in background cleate only one instance from this difference editor. Then you have to put to this instance all your 100 files one after other and copy in each case from one file following DOM objects:
<div class="editor original showUnused" ...
<div class="editor modified showUnused" ...
This could you do for example with following code:
var diffPartContainars = document.querySelector('#container').querySelectorAll('.showUnused'),
editorOriginalPartHTML,
editorModifiedPartHTML;
for(var i = diffPartContainars.length; i--;)
{
var obj = diffPartContainars[i],
cln = obj.className;
if(cln.indexOf('editor original') > -1)
{
obj.removeAttribute('style');
editorOriginalPartHTML = obj.outerHTML;
}
if(cln.indexOf('editor modified') > -1)
{
obj.removeAttribute('style');
editorModifiedPartHTML = obj.outerHTML;
}
}
Then you have to delete from each editorOriginalPartHTML and editorModifiedPartHTML following DOM objects:
<div class="invisible scrollbar horizontal" ...
<canvas class="decorationsOverviewRuler" ...
<div class="visible scrollbar vertical" ...
and all other objects which you can not use. You can do it when you add editorOriginalPartHTML and editorModifiedPartHTML to your DOM. Then you can add around each from it one div object with suitable width, height and style="overflow:auto". And one thing you could do more: for each from this div objects you could add one onclick or onmouseover listener and then replace this div object view with your difference editor instance.
This is only one way in my opinion to be more efficient. Good luck!
The recommended efficient solution
The quickly, comfortable and efficient way to have only one instance of this editor and load a new sources on the click on the file names like below.
var diffEditor = null;
var filesContent =
{
'SomeJavaScriptFile.js':
{
originalContent: 'alert("heLLo world!")',
modifiedContent: 'alert("hello everyone!")',
type: 'text/javascript'
},
'AnotherJavaScriptFile.js':
{
originalContent: 'function open(str)\n{\n\talert(str)\n}',
modifiedContent: 'function output(value)\n{\n\tconsole.log(value)\n}',
type: 'text/javascript'
}
};
document.querySelector('#files').addEventListener('change', function(e)
{
var fileName = this.options[this.selectedIndex].text,
file = filesContent[fileName];
openInDiffEditor(file);
});
function openInDiffEditor(file)
{
if(!diffEditor)
diffEditor = monaco.editor.createDiffEditor(document.querySelector('#container'));
diffEditor.setModel({
original: monaco.editor.createModel(file.originalContent, file.type),
modified: monaco.editor.createModel(file.modifiedContent, file.type)
});
}
//open the first file in select list:
var firstFileName = document.querySelector('#files').options[0].text;
openInDiffEditor(filesContent[firstFileName]);
<p>Please select one file on the left list to see the file differences after changes.</p>
<select id="files" size="3">
<option selected>SomeJavaScriptFile.js</option>
<option>AnotherJavaScriptFile.js</option>
</select>
<div id="container" style="height:100%;"></div>
And file contents you have to load over AJAX. But in the case if you do not understand how to load it then ask me and I will write it.
The screenshot of this recommended solution

(JavaScript API 1.3 for Office) Set Value of Custom Properties

My client has decided to migrate to Office 2016 and porting portions of a business process to that client requires us to offer a replacement to the Document Information Panel, which is no longer available. The Backstage file information area isn't considered a sufficient user experience for the users in question, so we're endeavoring to replace the DIP with a Task Pane app.
This example: https://www.youtube.com/watch?v=LVGqpns0oT8&feature=share shows that the idea is, at least in theory, possible. We considered buying this app but can't find sufficient information to do so.
So we set about attempting to replicate the functionality we need in the DIP. It appears that we can successfully set Document Properties of standard types, such as strings, which looks something like this:
Word.context.run(function(context){
var properties = context.document.properties;
context.load(properties):
return context.sync().then(function(){
properties.title = properties.title + " Additional Title Text"; // once the sync goes off, this works.
return context.sync();
});
});
However, when we try to update an Document Property that's, for example, a Managed Metadata property defined by a SharePoint content type, the value in the proxy object loads and remains changed, but it seems to break its relationship to the actual document property. The code below demonstrates:
Word.context.run(function(context){
var properties = context.document.properties;
var customProperties = properties.customProperties;
context.load(properties):
context.load(customProperties);
return context.sync().then(function(){
var managedMetadataProperty = customProperties.getItem('MngdMetadata');
properties.title = properties.title + " Additional Title Text"; // once the sync goes off, this works.
context.load(managedMetadataProperty);
return context.sync().then(function(){
console.log(managedMetadataProperty.value) // let's say this looks like "10;#Label 1|64d2cd3d-57d4-4c23-9603-866d54ee74f1"
managedMetadataProperty.value = "11;#Label 2|cc3d57d4-4c23-72d4-3031-238b9100f52g"
return context.sync(); // now the value in the javascript object for managedMetadataProperty is updated, but the value in the document does not change.
});
});
});
The document property Managed Metadata Property never changes in the Word UI, nor does a change push back to the SharePoint. Say we save and close the document after making the update, then re-open it. The Property value has not visibly changed, however when we load the proxy object with 'context.load()', the value that's available reflects the changes we made on last run.
I'm unclear about why this would be. It seems like to circumvent this, I would need to make a call back to SharePoint to update the relevant field, but I don't know how I would instruct Word to refresh with the new information from SharePoint.

That's a great question.
The custom properties API gives you access to some built-in properties as well as custom properties. SP-related properties do NOT follow in this category from the API perspective. (and the same is true in VBA/VSTO/COM) To access those you need to use the CustomXmlParts functionalities. Here is a good example on how to use it in the Javascript API.
Also, FYI, the team is working right now in a feature to enable the DIP again, i don't have concrete dates or commitment, but you might get this functionality again out of the box soon.

Have you tried customPropertyCollectionObject.add(key, value) ?
It will replace existing kvp's in the customPropertiesCollectionObject.
Here is the documentation customPropertiesCollection

Trying to save open MS Access documents from JScript

I was hoping to save all open MS Access documents via a JScript run from the Windows Script Host.
So far I was able to obtain the MS Access object by calling:
var objAccess = GetObject('', "Access.Application");
But now I'm stumped. If it was MS Word, I'd enumerate all open documents in the .Documents property and call Documents.Item(n).SaveAs() method on each of them.
But how do you save-as all open documents in MS Access?

After you have your object variable set to an Access application instance with GetObject, use its Quit method with the acQuitSaveAll option (value = 1). Not sure about JScript; in VBScript, I can do it like this.
Dim objAccess
Set objAccess = GetObject(,"Access.Application")
WScript.Echo objAccess.CurrentDb.Name
objAccess.Quit(1) ' acQuitSaveAll
Set objAccess = Nothing
Note, when I used GetObject as in your example, objAccess was a new Access application instance rather than a reference to the instance which was running previously. So, with the GetObject line like this ...
Set objAccess = GetObject('', "Access.Application")
... the WScript.Echo line threw an error with CurrentDb.Name (because there was not a database open in that Access application instance.
This approach will save any changes to database objects (tables, forms, reports, etc.) which were in design mode but not saved. However if a user has any unsaved changes to data in a form, those changes will be discarded despite the acQuitSaveAll option. It seems that option only applies to objects, not data.
Edit: If that approach is not satisfactory, you can do something more sophisticated with VBA in your Access applications, as #Remou mentioned in his comment. An example is KickEmOff from Arvin Meyer. He also offers a sample database which demonstrates that code in action.
Edit2: Remou's comment got me thinking acQuitSaveNone (value = 2) should be safer than acQuitSaveAll ... the unsaved object changes would be discarded, but at least you would be less likely to save an object in a non-functional state.