I am using cloudant database to store my values and I specify my
document ID to be string+date(ddmmyy) (e.g foo071117). Recently I
need to filter the documents to show documents in the last 7 days
(e.g foo311017 to foo061117) and I have used views to do this. This is my code:
function (doc) {
var datearr = [];
var today = new Date()
for (var i =1; i<=7; i++){
var datesfull = new Date(today.getFullYear(), today.getMonth(), today.getDate() - i);
var dd = datesfull.getDate();
if (dd<10){
dd = '0'+dd;
}
var mm = datesfull.getMonth()+1;
if (mm<10){
mm ='0'+mm
}
var yy = datesfull.getFullYear().toString().substr(-2);
var dates = "foo" + dd + mm + yy;
datearr.push(dates);
}
if((datearr.indexOf(doc._id.slice(0,-1))>=0) || (datearr.indexOf(doc._id)>=0)){
emit(doc._id,doc);
}
}
The slice is for when my json files gets too big and I have to split the file into multiple files (e.g foo0711170, foo0711171)
This code works at the beginning as after running, cloudant view only showed 7 documents which I wanted. However, as the days passes, the old files remains instead of having just 7 documents. Is Cloudant supposed to refresh to show 7 documents in the View as specified in my code or is this due to how the data is stored?
*I can't query data for the past 7 days using URL as I am using powerBI which limits my ability to write a code to specify which document ID to extract so I can only use views as a filter
When you supply a design document to Cloudant with an index definition, the view is built and stored on disk. You may then query the view at a later time to get its results.
In your design document your are getting the time of "now", which in this case is the time when indexing is performed - not the time that the query is performed. It therefore cannot be used to extract data "in the last 7 days" (of the query time).
One solution would be to change the way your document _id is built. If you used: yyyymmdd+string, then your documents would be stored in date order because Cloudant has a primary index on the _id field.
You could then query the database's primary index, starting with a key 7 days in the past e.g.:
/mydb/_all_docs?startkey=20171101&include_docs=true
Related
Does anyone know how i can use moment js , to create a dynamic dates array , that contains todays date and dates of next 9 days.
so im creating a scheduler that will run on some days(not yet specified) when it runs its supposed to prepare some reports on todays date and next 9 days, my idea is to create an array of dates and map through them but i cant figure out how to create such an array using moment js , any ideas ?
i checked some legacy code and they are using the following to get present date
const now = moment().add(-moment().utcOffset(), 'minutes').toDate()
I think this is a good candidate for a traditional for loop solution, using an offset and the moment .add method to build an array adding the offset as you go, ending up with a ten-day array where the first element at index 0 holds the value of your now variable.
const moment = require('moment')
const date = moment().add(-moment().utcOffset(), 'minutes').toDate()
const reportDates = [];
for (let dayOffset = 0; dayOffset < 10; dayOffset++) {
const nextDate = moment(date).add(dayOffset, 'days').toDate();
reportDates.push(nextDate)
}
console.log(reportDates)
Output when I ran this snippet on my machine (# ~10:22 Central US time):
[
2022-11-29T22:22:51.788Z,
2022-11-30T22:22:51.788Z,
2022-12-01T22:22:51.788Z,
2022-12-02T22:22:51.788Z,
2022-12-03T22:22:51.788Z,
2022-12-04T22:22:51.788Z,
2022-12-05T22:22:51.788Z,
2022-12-06T22:22:51.788Z,
2022-12-07T22:22:51.788Z,
2022-12-08T22:22:51.788Z
]
The moment.js vs other libraries is another discussion, and if at all possible to use a different library I would recommend date-fns to avoid any confusion in the future from the mutability built into moment objects.
Good luck!
Issue in Qliksense SelectValues Field API when the field is Date format
Hi ,
Just to provide you background - We have multiple qlik apps opened in our web application and when the user makes the selections in primary app - we apply the common selection across all the apps opened. We have this strange issue with fields that are tagged as Date in the datamodel. Please check below example.
Version : June 2020 Patch 7 v13.82.15
I have issue with selectValues Field API when the field is in Date format.
1. Scenario 1 - Field : Month_Year is date datatype:
Let say, I have val array in which I have selected values of the date type field "Month_Year".
Var FieldName = "Month_Year"; // field in datamodel script : Date#(Month_Year,'MMM YYYY') AS Month_Year
var valArray[]= {
0:"Jan 2020"
1:"Feb 2020"
2:"Mar 2020"
}
app.Field(FieldName ).selectValues(valArray);
it is not selecting the value in desired "app" because the FieldName = Month_Year is derived as Date#(Month_Year,'MMM YYYY').
2. Scenario 2 - Field : Month_Year is TEXT() string datatype:
Let say, I have val array in which I have selected values of the text/string type field "Month_Year".
Var FieldName = "Month_Year"; // field in datamodel script : Text(Month_Year) AS Month_Year
var valArray[]= {
0:"Jan 2020"
1:"Feb 2020"
2:"Mar 2020"
}
app.Field(FieldName ).selectValues(valArray);
Now the same field is making the selections in the field if we bring the field as text(Month_Year) as Month_Year.
did anyone faced this issue in your mash up application?
Thats because under the hood Qlik is storing the values in dual representation - number and text (especially for fields that are flagged as date). The actual value for 01/01/2020 is actually 43831 and its displayed as 01 Jan 2020 (or whatever is the date format). So when trying to select Jan 2020 nothing is happening because such value do not exists.
When you "break" the dual representation (using the Text() function) then the selection is working but potentially there can be performance implications (based on the data volume in the app mostly)
Have a look at the dual() function documentation for more info about this behaviour.
In order to select something in this case you'll have to find the underlying values or element numbers and select them.
Personally I'm creating temporary listbox for the required filed and when I have to select I'm finding the qElemNumber of the required values and then selecting them in the listbox
// let values = ['array', 'of', 'values', 'to', 'select']
// create the session object
let sessionObj = await qDoc.createSessionObject(lbDef);
// get the layout so we can loop through the data pages
let layout = await sessionObj.getLayout();
// get the qElemNumber for the required values
// WARNING: if the field contains more than 1000 (i think) DISTINCT values then we'll have to loop through all the databased
let index = layout.field.qListObject.qDataPages[0].qMatrix.filter(function (m) {
return values.indexOf(m[0].qText) > -1
}).map(function (e) {
return e[0].qElemNumber
});
// perform the selection
let selection = await sessionObj.selectListObjectValues('/field/qListObjectDef', index, toggle);
I've used such approach here
I have a data-frame and I want to send it in my Django template.
dummy code in views.py:
def graphs(request):
df_new = pd.read_excel("/home/cms/cms/static/Sensorik.xlsx")
times = df_new.loc[:, df_new.columns[0]].to_json(orient='records')
# columns[0] contains datetime values
data_color = df_georgian.loc[:, df_georgian.columns[2]]
color = data_color.to_json(orient='records')
context = {
'times': times,
'data_color': color,
...
}
return render(request, 'graphs/graphs.html', context)
In my template, I get these data like the following:
<script>
var times = {{ times|safe }};
console.log('index: ', typeof(index));
var color = {{ data_color|safe }};
</script>
the color variable is totally ok but the times variable when get to JSON format turns from 2018-05-29 08:09:00 format to something like it:
element: 1528108200000
I want to be able to plot the color based on times to get a line graph with plotly. i.e. I want to show times as x-ticks of my plot.
any suggeston on
1- how to send datetime dataframe to django template?
or
2- how to convert element: 1528108200000 into a datetime in js to plot it as x-ticks?
Whenever you see something that should be a date or time expressed in a large number like the 1528108200000 format, that means it is or is similar to a UNIX timestamp—the number of seconds past January 1, 1970. In this case, the length of the timestamp indicates that it's milliseconds, not seconds, so it's not exactly a UNIX timestamp but is very close.
Milliseconds since January 1, 1970 is the way that JS internally stores Date objects, which is interesting because it means some sort of conversion is probably happening on the JS side, not the python side (where you'd usually get ISO format or UNIX Timestamps).
In any case, it's pretty easy to solve on the JS side because you can simply parse the number:
dateObject = new Date(1528108200000);
If you do that parsing and pass the data to Plotly as Date objects, Plotly should be able to recognize your dates.
A simple way to do that would be:
const times = [1528108200000, 1528108200000, 1528108200000]
const parsed = times.map(time => new Date(time))
assuming your times variable is just an array of these integers.
Some background: We have a shared Google Sheet to track our openings, screenings, and other events at a movie theater. We have a main tab ("Master") that contains all of our events and the details that go with them, and a tab for archiving ("Archive").
I would like to write a script within Google Sheets to detect events & screenings that are from yesterday and earlier based on the date (in column E), take the full row(s) (events) that meet that criteria, copy & paste them to the separate "Archive" tab, and then delete the row(s) from the "Master" tab.
Anything to point me in the right direction would be super helpful. I found a few similar responses to this but they're specific to Excel/VBA and I'm not familiar with that (or much Javascript, for that matter).
I suggested that you do some tutorials to familiarise yourself with how to write scripts.
In this answer, I will flesh out the steps that your code needs to address. You will find many existing topics on the same or similar question. This is merely in order to enable you to better search for the elements of code that you need. Consider that this may be just one way of achieving your outcome.
You have one spreadsheet with two sheets and you will refer to both sheets at different stages. getSheetByName(name) will enable you to create a re-usable variable for a sheet.
You will need to find the bottom row in each sheet. getLastRow() will help.
You want to find rows in "Master" for dates, so you need to get ALL values for "Master".
You'll start by defining the range - use getRange(row, column, numRows, numColumns), though this is just one of 5 ways to define a range.
Having defined the range you'll need the values in "Master" so that you can access the date field. Use getValues() in conjunction with the range that defined. FWIW, note how this is in plural because there are lots of values. If you just wanted a single cell, you'd use getValue().
You'll want to loop through the rows in "Master" and find those rows that have a date prior to today. The "Removing Duplicate Rows in a Spreadsheet" tutorial shows one way of looping, and you can read up on basic JavaScript "Loops and iteration".
In your scenario, there is a 'hitch' with looping. If one adopts the "usual" process, then one will loop from the first row to the last. However, you are deleting a row from "Master" and, as each row is deleted, the row numbers of the remaining rows will/may change; so the "usual" process won't do. What you need to do is two things: first) loop from the bottom of the range; this will ensure that the row numbers of remaining rows will never change; second) sort the data so that the oldest dates are at the bottom. So... now you will loop from the bottom to the top, and you will evaluate all the oldest dates without any risk that when you encounter a date greater than "today", there will be NO risk of further rows with a date less than "today". Of course, once the code is complete, you can always re-sort the data on "Master" back to any order that you might wish.
You need to compare the date in the row in "Master" with today's date and then build a if...else statement so that you can define what to do depending on the result. Comparing dates is sometimes easier said than done. This topic is relevant Checking if one date is greater than the other using Google Script and you can search on other topics for "Google Sheets Script date comparison".
When you find a date less than today, you want to copy the details of that row to "Archive". This is a two part process first) to gather there the data from the row on "Master", and second) to "copy" that data to "Archive". Gathering the data will have been covered in the tutorials. There are many options for copying the data to "Archive". You could append a row and use setValues to update the new values. An alternative is to accumulate the additional "Archive" data and add it to the "Archive" after the loops have been completed.
When you find a date less than today, you want to delete the row from "Master". There's a command for that: deleteRow(rowPosition).
You can process your function manually, on demand, or you may prefer it to be automated as a time-driven installable trigger. The option is yours.
There are many ways that you can combine these elements.
In preparing the summary above, I had to make sure that I was providing accurate and complete advice. So the following is but one approach to achieving your goal. It should be noted that my test data assumes that columns A and C are formatted for date and time respectively.
function so5710086103() {
// set up spreadsheet and sheets
var ss = SpreadsheetApp.getActiveSpreadsheet();
var master = ss.getSheetByName("Master");
var archive = ss.getSheetByName("Archive");
// get the last row and column of Master
var masterLR = master.getLastRow();
var masterLC = master.getLastColumn();
// get the last row and column of Archive
var archiveLR = archive.getLastRow();
var archiveLC = archive.getLastColumn();
//Logger.log("DEBUG: Last Row - Master = "+masterLR+", and Archive = "+archiveLR);
//Logger.log("DEBUG: Last Column - Master = "+masterLC+", and Archive = "+archiveLC);
// create a range, sort it and get the data from "Master"
var masterRange = master.getRange(2, 1, masterLR - 1, masterLC);
// sort master based on date
masterRange.sort({
column: 1,
ascending: false
});
// Logger.log("DEBUG: Master range = "+masterRange.getA1Notation());
var masterData = masterRange.getValues();
//Logger.log("DEBUG: Length of Master data = "+masterData.length);
// create a range and get the data from "Archive"
var archiveRange = archive.getRange(1, 1, archiveLR, archiveLC);
var archiveData = archiveRange.getValues();
// create a formatted date for today
var formattedToday = Utilities.formatDate(new(Date), 'GMT+10',
'dd MMMM yyyy');
// loop through the rows
// from bottom to top
for (var i = (+masterLR - 2); i > 0; i--) {
// convert cell dates to comparable format
var DBdate = Utilities.formatDate(masterData[i][0], 'GMT+10',
'dd MMMM yyyy');
var DBtime = Utilities.formatDate(masterData[i][2], 'GMT+10',
'hh:mm a');
//Logger.log("DEBUG: i = "+i+", DBdate = "+DBdate+", Today = "+formattedToday);
// clear the temporary row array
var archivecells = [];
if (DBdate < formattedToday) {
// the table date is less than today, so archive the data
// Logger.log("DEBUG: i = "+i+", DBdate = "+DBdate+", Today = "+formattedToday+" - DB value is less than Today. ACTION: Archive this row");
// copy the row cells to temporary row array
archivecells.push(DBdate);
archivecells.push(masterData[i][1]);
archivecells.push(DBtime);
archivecells.push(masterData[i][3]);
archivecells.push(masterData[i][4]);
// copy the temporary row array to archivedata
archiveData.push(archivecells);
// delete the Master Row
master.deleteRow(i + 2);
} else {
// the table date is NOT less than today, so do nothing
// Logger.log("DEBUG: i = "+i+", DBdate = "+DBdate+", Today = "+formattedToday+" - DB value is NOT less than Today. ACTION: Do nothing");
}
// update the accumulated data to Archive.
archive.getRange(1, 1, archiveData.length, archiveLC).setValues(
archiveData);
}
}
getSheetByName(name)
getLastRow()
getRange(row, column, numRows, numColumns)
getValues()
Loops and iteration
Javascript if...else
deleteRow(rowPosition)
Master - Before
Master - After
Archive - After
I'm building an API for my site (using Node.js and Mongoose) and I would like to incorporate pagination in it. My problem is the following: if the page size is for example 15 and I make a request for the first page so it sends me the first 15 items ordered by date of creation but then what if before I make the request for the second page, 15 new items are created in the database, the returned data will be the same as previously if I just use a skip on mongoose.
Is there a way to avoid doublons with mongoose? What I have at the moment is an "exclude" parameter in the query so it excludes all items already loaded but I'm thinking if there are lots of loaded items, the URL might be very long and I'm not sure that's a good thing...
Is there a better way to do this or do I have to just leave it with the risk of having doublons?
You can use mongoose-paginate-v2
And to prevent any duplicates from returned documents, you can pass (last document id & limit)
And your query should be like :
import { PaginateModel } from 'mongoose-paginate-v2';
constructor(
private readonly _paginateModel: PaginateModel<any>
) {}
const query = this._paginateModel
.find({ _id: { $lt: lastId } });
this._paginateModel.paginate(query, { limit: 10 });
You can use skip and limit function in which you can pass min and max value .
like in first page if you get 15 records values for min and max will be :
min = 0 and max = 15
for page 2
min = 15 and max = 30