We Keep Coding

Find all the subranges for a date range api request with capped max offset

I've to extract data from an API that has a max offset (1000) and limit (20). So it's possible that between two dates (from and to) there are more results than the returned from the request.
I'd like to get the complete set of data between those two dates so I'm trying to come up with a solution to accomplish this.
My idea is to start by making a request like:
https://api.com/?from=2021-07-20&to=2021-07-24&limit=20&offset=1000
If the request returns less than 20 elements the search is over, I've all the data available between the ranges. But if the request returns 20 elements, then it's probable that there's more data in this range, so I have to find a way to keep splitting ranges until this condition is false.
I've thought about splitting the ranges like:
from = from
to_1 = 2021-07-22
from_1 = 2021-07-22
to = 2021-07-24
And then pass those ranges to the recursive function until finding all the needed subranges.
The output would be something like:
[(2021-07-21,2021-07-22),(2021-07-22,2021-07-23),(2021-07-23, 2021-07-24)]
The problem with this solution is that I'm expanding the from's and to's so I can't manage to use a recursive function and I'm struggling about how to fix this issue.
Edit: I've added the "javascript" tag as the solution is going to be implemented in that language but ideas/pseudocode is welcome too.

Seems like you're looking for binary search.
Your recursive function should look like this (it's JS pseudo-code):
function get_ranges(from, to)
{
const items = request(from, to)
if (items.length < 20) {
return [[from, to]]
}
const date_between = calculate_date_between(from, to)
const left_ranges = get_ranges(from, date_between)
const right_ranges = get_ranges(date_between, to)
return [...left_ranges, ...right_ranges]
}
const result = get_ranges(min_date, max_date)

Call function saved as string on chosen object

I have something like:
var sFunction = 'my_function("param1", "param2")';
var oMyObject = ...;
And I want to combine it so the result would be equal to:
oMyObject.my_function("param1", "param2");
Would much appreciate any tips.
Remark
As many of you suggested to find a root cause and try not to deal with the problematic input here are some pieces of information about the origins of the "problem".
The sFunction comes from database, hardcoded in one of the columns. It is custom one which should be called on object retrieved basing on other parameters of sFunction's database record.
So being backed up by your comments I will try suggesting changing data model in hope that it is not too late for that. Thank you all for your help.

I am given that as an input, it may come from db or anywhere else. I just have to deal with it in described way.
As Luca noted, you're probably best off solving the problem that brought you to the point of having code in a string that you feel you need to evaluate at runtime. The number of use cases for doing that is very low.
For instance, instead of
sFunction = 'my_function("param1", "param2")';
perhaps you could have
call = {
f: "my_function",
params: ["param1", "param2"]
};
Then it's:
oMyObject[call.f].apply(oMyObject, call.params);
call could even start life as JSON text you parse -- live example:
var json =
'{' +
'"f": "my_function",' +
'"params": ["param1", "param2"]' +
'}';
var call = JSON.parse(json);
var oMyObject = {
my_function: function(p1, p2) {
console.log(p1, p2);
}
};
oMyObject[call.f].apply(oMyObject, call.params);
That's markedly safer than an arbitrary code execution.
You can do this with your sFunction (eval("oMyObject." + sFunction)), but consider:
It lets any arbitrary code in sFunction run.
If User A supplies the code and then you run it on User B's system, you're compromising User B's privacy. (I am not a lawyer, but you could be doing so in a way that violates a country's data protection or privacy laws.)
Now, if you're loading code from a DB and you know that the code in the DB can only be put there by trusted people (for instance, developers on your team, not end users of the system), that's fine, it's largely like running a script file. But there's almost certainly a better way to do it than delivering the code as a string and evaling it.
But if the code comes from "anywhere else", it's not fine; see bullet points above. The setup is fundamentally broken and better options are available. Take that information to your boss, and if necessary to his/her boss, and if necessary his/her boss, until you find someone who can change the requirement.

Here's a string hack that doesn't use eval(), but as I (and others) have said, this is not a good solution. The better solution would be to return the function name and any arguments as a comma delimited string, which would at least make this kind of solution more straight-forward.
var sFunction = 'my_function("param1", "param2")';
// The object would have to already have the function:
var oMyObject = {
my_function: function(x,y){
return x + y;
}
};
// Remove the last ")" and split the remainder into an array at the "("
var funcParts = sFunction.replace(")","").split("(");
// Split the second part (the arguments) into its own array
var funcArgs = funcParts[1].split(",");
// Pass the function name as a string key to the object and then pass the arguments to that
console.log(oMyObject[funcParts[0]](funcArgs[0], funcArgs[1]));
The bigger question is, what ultimately are you trying to accomplish as there is almost always a better approach than this.

To do a dynamic function call you can of course eval as I did in the comments, which is of course a terrible idea. Here is a quick-and-dirty alternative:
const dynamicCallMethod = (obj, s) => {
try {
const fname = s.match(/([$\w]+\(/);
const params = s.match(/("[\w$]+")/g);
return obj[fname](...params);
} catch (e) {
return e;
}
};
Note I still think there's any easier way to do this if you describe the scenario in more detail. The above will fail for any non-ascii characters, for instance.

Get DST/STD dates and times for a given timezone (Javascript)

I am developping an embedded website contained within a windows CE device.
See it as if it was the web configuration interface of your router.
Everything is contained within a really small footprint of memory, the entire website is less than 500KB with every script, html, css and icons.
We have to assume that the user that is going to 'browse' into that interface does not have access to the internet (LAN only) so no 'online' solution here.
I am looking for a solution so the user choose his timezone and the code will get all the DST/STD times and dates for the next 10-20 years at least and downloaded them to the device that will run autonomously after that and at specific dates, will change its time to DST/STD by itself. Current application is custom (not windowce api related) and needs the DST/STD date pairs.
iana.org is maintaining a db for like every location in the world. I also saw that moment-timezone (javascript interface) is 'packaging' this data in a very compact package 25kb zipped.
I need to know if it is possible to:
'Extract' a main tz list from this DB so the user can choose its own tz. I looked at their doc but example didn't work:
var itsTimeZones = moment.tz.names();
2- Extract the next 10-20 years of DST/STD dates/times for a chosen zone ? I haven't saw any documentation anywhere on this topic. But since it's kind of the purpose of such a database, i would say it must be burried in there somewhere, need a way to dig it out.
3- If moment timezone is not the right track to go, anybody has a solution that will fullfill that requirement ?
Thanxs

My solution for extracting DST list for a given zone from moment-timezone.
1. "theTz" is a numerical index from a change on a select in the webpage.
2. select is populated with "List".
var itsTz =
{
"List" : moment.tz.names(), // Get all zone 'Names'
"Transitions" : [],
};
function TimeZoneChanged(theTz)
{
var myZone = moment.tz.zone(itsTz.List[theTz]); // Get zone by its name from List
var myTime = moment.tz(moment(), myZone.name); // Start today
var myResult;
// Build transition list according to new zone selected
itsTz.Transitions = [];
do
{
myResult = GetNextTransition(myZone, myTime);
if(myResult != null)
{
itsTz.Transitions.push(moment(myResult).format("YYYY/MM/DD"));
myTime = moment.tz(myResult, myZone.name); // Get next from date found
}
}while(myResult != null);
}
function GetNextTransition(theZone, theTime)
{
var myResult;
for(var i = 0; i < theZone.untils.length; i++)
{
if(theZone.untils[i] > theTime)
{
theZone.untils[i] == Infinity ? myResult = null : myResult = new moment(theZone.untils[i]).format();
return myResult;
}
}
}
Results for 'America/Toronto:
2017/03/12,
2017/11/05,
2018/03/11,
2018/11/04,
2019/03/10,
2019/11/03,
2020/03/08,
2020/11/01,
2021/03/14,
2021/11/07,
2022/03/13,
2022/11/06,
2023/03/12,
2023/11/05,
2024/03/10,
2024/11/03,
2025/03/09,
2025/11/02,
2026/03/08,
2026/11/01,
2027/03/14,
2027/11/07,
2028/03/12,
2028/11/05,
2029/03/11,
2029/11/04,
2030/03/10,
2030/11/03,
2031/03/09,
2031/11/02,
2032/03/14,
2032/11/07,
2033/03/13,
2033/11/06,
2034/03/12,
2034/11/05,
2035/03/11,
2035/11/04,
2036/03/09,
2036/11/02,
2037/03/08,
2037/11/01,

Settnig duration values in jtsage datebox in mode durationflipbox

I'm having difficulty setting a time (duration) value in a datebox. A simple demonstration of the problem is if I do something like:
function initDuration() {
this.d['header Text'] = "Set";
this.d['headerText'] = "Set Duration";
var element = 'input#'+this.element[0].id;
var currentDt = $(element).datebox('getTheDate');
// ***************
var dt = $(element).datebox('parseDate', '%H:%M', this.element[0].value); // Where this.element[0].value = "01:00:00"
// ***************
$(element).datebox('setTheDate', this.element[0].value);
$(element).trigger('datebox', { 'method': 'doset' });
}
dt just contains the current date/time; i.e. jtsage didn't like it. The element is defined (in jade) as:
input.Duration(type="text" name="duration" form="form#{i}"
id="duration#{i}" value="#{map[i].duration}" data-role="datebox"
data-options=
'{"mode":"durationflipbox", "overrideDurationOrder":["h","i"],'
+' "overrideTimeFormat": "%l:%M", "minuteStep":15, "beforeOpenCallback": "initDuration"}')
Also I'm not sure how to change the flipbox title. The 2nd line in initDuration() sets the text for the button but the title still says 'Set Time'.
Because of the first problem the last 2 lines in initDuration() don't do what I want. i.e. they just use the current time, whatever that happens to be.

My apologies that this is going to be an incomplete answer, but it was going to be too long for a comment.
For the title - give "overrideHeaderText" a shot instead. It is entirely possible that I screwed this up at some point, it's not a feature I use in any of my own projects.
Next...
var dt = $(element).datebox('parseDate', '%H:%M', this.element[0].value); // Where this.element[0].value = "01:00:00"
I think I am reading you correctly that "dt" isn't containing what you are expecting. It's because 01:00:00 != %H:%M - to read this "format", you'd need to either use "%H:%M:%S" or "%H:%M:00" (the later ignoring the seconds field).
That said, I think what you are trying to do is set a duration, which, is a little different. There are a few ways to do it - and I'm noticing that there isn't a lot of support to do it functionally. The simplest method, is the set the value of the input, and let datebox handle the math - just be aware that the format you drop into the input must be exactly the same as the output format - it will read it when the control opens (or is initialized if the control is being shown inline - if you are doing it inline, and set the value "later", you can use the 'refresh' method to update it).
For what it's worth, if you really, really, really want to use the setTheDate method, duration modes work by comparing "theDate" (the publicly available date, i.e. setTheDate, getTheDate) with an internal initDate - which is not exposed to the API, but can be found here:
$(element).data('jtsage-datebox').initDate
So, in pseudo-code, for a duration of an hour
myNewDate = $(element).data( 'jtsage-datebox' ).initDate;
myNewDate.setHour( myNewDate.getHour() + 1 );
$(element).datebox( 'setTheDate', myNewDate );

Delete multiple documents

The following code is working but extremely slow. Up till the search function all goes well. First, the search function returns a sequence and not an array (why?!). Second, the array consists of nodes and I need URI's for the delete. And third, the deleteDocument function takes a string and not an array of URI's.
What would be the better way to do this? I need to delete year+ old documents.
Here I use xdmp.log in stead of document.delete just te be safe.
var now = new Date();
var yearBack = now.setDate(now.getDate() - 365);
var date = new Date(yearBack);
var b = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c = cts.search(b, ['unfiltered']).toArray();
for (i=0; i<fn.count(c); i++) {
xdmp.log(fn.documentUri(c[i]), "info");
};

Doing the same with cts.uris:
var now = new Date();
var yearBack = now.setDate(now.getDate() - 365);
var date = new Date(yearBack);
var b = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c = cts.uris("", [], b);
while (true) {
var uri = c.next();
if (uri.done == true){
break;
}
xdmp.log(uri.value, "info");
}
HTH!

Using toArray will work but is most likely were your slowness is. The cts.search() function returns an iterator. So All you have to do is loop over it and do your deleting until there is no more items in it. Also You might want to limit your search to 1,000 items. A transaction with a large number of deletes will take a while and might time out.
Here is an example of looping over the iterator
var now = new Date();
var yearBack = now.setDate(now.getDate() - 365);
var date = new Date(yearBack);
var b = cts.jsonPropertyRangeQuery("Dtm", "<", date);
var c = cts.search(b, ['unfiltered']);
while (true) {
var doc = c.next();
if (doc.done == true){
break;
}
xdmp.log(fn.documentUri(doc), "info");
}
here is an example if you wanted to limit to the first 1,000.
fn.subsequence(cts.search(b, ['unfiltered']), 1, 1000);

Several things to consider.
1) If you are searching for the purpose of deleting or anything that doesnt require the document body, using a search that returns URIs instead of nodes can be much faster. If that isnt convenient then getting the URI as close to the search expression can achieve similar results. You want to avoid having the server have to fetch and expand the document just to get the URI to delete it.
2) While there is full coverage in the JavaScript API's for all MarkLogic features, the JavaScript API's are based on the same underlying functions that the XQuery API's use. Its useful to understand that, and take a look at the equivalent XQuery API docs to get the big picture. For example Arrays vs Iterators - If the JS search API's returned Arrays it could be a huge performance problem because the underlying code is based on 'lazy evaluation' of sequences. For example a search could return 1 million rows but if you only look at the first one the server can often avoid accessing the remaining 999,999,999 documents. Similarly, as you iterate only the in scope referenced data needs to be in available. If they had to be put into an array then all results would have to be pre-fetched and put put in memory upfront.
3) Always keep in mind that operations which return lists of things may only be bounded by how big your database is. That is why cts.search() and other functions have built in 'pagination'. You should code for that from the start.
By reading the users guides you can get a better understanding of not only how to do something, but how to do it efficiently - or even at all - once your database becomes larger than memory. In general its a good idea to always code for paginated results - it is a lot more efficient and your code will still work just as well after you add 100 docs or a million.
4) take a look at xdmp.nodeUrl https://docs.marklogic.com/xdmp.nodeUri,
This function, unlike fn.documentUri(), will work on any node even if its not document node. If you can put this right next to the search instead of next to the delete then the system can optimize much better. The examples in the JavaScript guide are a good start https://docs.marklogic.com/guide/getting-started/javascript#chapter
In your case I suggest something like this to experiment with both pagination and extracting the URIs without having to expand the documents ..
var uris = []
for (var result of fn.subsequence(cts.search( ... ), 1 , 100 )
uris.push(xdmp.nodeUri(result))
for( i in uris )
xdmp.log( uris[i] )