I have been asked to count the number of tweets per hour by day (0 - 23) in a huge text file of random tweets. The date is not interesting, only the tweet per hour. I want to return them in a new array of objects. Each object should have properties hour and count like this:
{hour: x, count: y},
I've made a function where I'm declaring an empty array, in which I will put my data:
function(tweets) {
let result = [];
and I think I need to push them like this:
result.push({hour: x, count: y});
But I don't know how to extract the specific hour from my object (key and value).
in the huge, raw data file, each tweet is logged with a date like this:
created_at: "30-06-2015 14:27",
Any suggestions or experience? I'm currently learning about regex and for loops. Should I use them in this code or is there a smarter way?
Edit: as you asked for more details:
The raw data are object in an array with the following structure:
{
time: Date-object,
created_at: "30-06-2015 14:27",
fromUsername: "victor",
text: "asyl og integration",
lang: "da",
source: "Twitter for Android",
}
About extracting text I see good answer here. Instead of console.log add parsing and saving to your array.
About regexp - I think it should be something like
var re = /created_at: \"([^\"]*)\",/g;
What I would do is work from a different angle:
create an object with a dateTimeHour for the start of each hour that you care about. It should presumably be a limited timespan like for all tweets that happened before now:
So generate something that looks like this dynamically:
{
'2019-03-01T17:22:30Z': 0, // or simply '1552667443928'
'2019-03-01T18:22:30Z': 0,
'2019-03-01T19:22:30Z': 0,
'2019-03-01T20:22:30Z': 0,
...etc
}
Which you can do using current Date and then a loop to create additional previous date times:
const now = new Date()
// you can use a generator here or simply a while loop:
const dateTimes = {}
while(now > REQUIRED_DATE)
dateTimes[new Date(now.setHours(now.getHours() - 1))] = 0
Now you have an exhausted list of all the hours.
Then, check if the given tweet is within that hour:
check if item.created_at < currentHourBeingLooked because you should loop through the Object.keys(dateTimes).
Then, loop through each item in your list and check if it fits that dateTime if so increment dateTimes[currentHour]++.
So, the hardest part will be converting created_at to a normal looking date time string:
const [datePortion, timePortion] = "30-06-2015 14:27".split(' ')
const [day, month, year] = datePortion.split('-')
const [hour, minute] = timePortion.split(':')
now with all those date, month, year, hour, and minute you can build a time object in javascript:
It follows the formula:
From MDN:
new Date(year, monthIndex [, day [, hours [, minutes [, seconds [, milliseconds]]]]]);
AKA:
new Date(year, monthIndex, day, hours, minutes, seconds);
So for December 17, 2019 # 3:24am it'll be:
const = new Date(2019, 11, 17, 3, 24, 0);
I'll assume that you already know to use regex from the post pointed by Ralkov to get all of your created_at dates, and my answer will go from that.
You said the date is not important so once you have the string
'created_at: "30-06-2015 14:27"'
we need to get rid of everything except for the hour, i did it by extracting substrings, feel free to try other approaches, this is just to get you started.
var date = obj.substr(obj.indexOf(' ') + 1);
var time = date.substr(date.indexOf(' ') + 1);
var hour = time.substr(0, time.indexOf(':'));
will get yo the hour
"14"
Note that this only works for one day, you need to do some additional changes if you'd like to store tweet hour count for different days in the same data structure
When you write your for-loop use the following function each time you find a tweet and already extracted the hour, it stores a combination of value-pairs into a map variable defined outside the function, creating a new pair if necessary or just updates it with the new tweet count.
function newTweet(hour, tweetsPerHour) {
var tweetsThisHour = tweetsPerHour.get(hour);
tweetsThisHour = tweetsThisHour === undefined ? 0 : tweetsThisHour;
tweetsPerHour.set(hour, ++tweetsThisHour);
console.log(tweetsThisHour)
}
complete code:
var obj = 'created_at: "30-06-2015 14:27"';
var date = obj.substr(obj.indexOf(' ')+1);
var time = date.substr(date.indexOf(' ')+1);
var hour = time.substr(0, time.indexOf(':'));
var tweetsPerHour = new Map();
newTweet(hour, tweetsPerHour); //this is the extracted hour
newTweet("16", tweetsPerHour); //you can try different hours as well
newTweet("17", tweetsPerHour);
function newTweet(hour, tweetsPerHour) {
var tweetsThisHour = tweetsPerHour.get(hour);
tweetsThisHour = tweetsThisHour === undefined ? 0 : tweetsThisHour;
tweetsPerHour.set(hour, ++tweetsThisHour);
console.log(hour + " tweet count: " + tweetsThisHour)
}
what the code is doing is storing the hour and count of tweets in pairs:
[{"14":1} ,{"16":1}, {17:1}]
for example if you add "14" again it would update to
[{"14":2}, {"16":1}, {17:1}]
dig into JavaScript Map Objects as well.
Your code flow is something like the following:
Read .txt file
loop through dates -> get hour from date -> newTweet(hour,
tweetsPerHour).
Related
edit, I've been able to work around this problem by comparing the values of the dates rather than the dates themselves:
$scope.entries[0].formattedDate === $scope.days[0]
$scope.entries[i].formattedDate.valueOf() === $scope.days[0].valueOf() //true
The goal of this angularjs program is to:
Generate the days in the current week (as an array of Javascript Date Objects)
Obtain a bunch of JSON objects from mongodb. (These objects have a property called "date." This "date" property is in the following format: ISODate("2016-05-18T04:44:42.067Z")
Compare each of the mongo items to a generated day, tell the user if any match (only the day! disregard time)
This is my code to generate the days of the week, and it works well:
$scope.getWeek = function (num) {
var curr = new Date;
curr.setDate(curr.getDate() - (num * 7));
var days = [];
for (var i = 0; i < 7; i++) {
days.push(new Date(curr.setDate(curr.getDate() - curr.getDay() + i)));
days[i].setHours(0, 0, 0, 0);
}
$scope.days = days;
};
I only want to know if the day matches, and not the time. I am using this function to strip the time from every item retrieved from Mongodb:
var removeHours = function (date) {
date.setHours(0, 0, 0, 0);
return date;
};
When it comes to comparing the dates, I'm absolutely stuck! It seems like I've tried everything. When I console.log the two dates, they are identical. I'm probably not aware of something about JS objects and I'm hoping somebody can provide a simple, easy solution.
$scope.entries[i].formattedDate == $scope.days[0] // false
In case I have following:
var day = '29';
var month = '12';
var year = '2015';
var created_date = new Date(year, month, year); // first
var created_date = new Date(parseInt(year), parseInt(month), parseInt(year)); // second
Can I create new date as it described in "first" line? Or do I need to parse integer out of string first as it shown in "second" line?
Thank you in advance.
new Date("2015", "11", "23") // This work perfect, but it should be yy, mm, dd and not yy, mm, yy
You can do it in a both way except you have to -1 to the month
var day = '29';
var month = '12';
var year = '2015';
var created_date = new Date(year, month-1, day);
This will work correctly.
You know, you have entered year to the day parameter of Date()
var day = '29';
var month = '12';
var year = '2015';
var month1=parseInt(month)-1;
var created_date = new Date(year, month1.toString(), day);
console.log(created_date);
var created_date1 = new Date(parseInt(year), month1, parseInt(day));
console.log(created_date1);
both statements are correct in order to obtain new date but the format inside the new Date(); should be of type yy,mm,dd not in yy,mm,yy
The parameters of date are:
new Date(year, month, day, hours, minutes, seconds, milliseconds);
The only caveat to keep in mind is that January is month 0.
So if a user inputs 12 for December, or 10 for October, substract one.
Another caveat to remember is that dates may unexpectedly shift.
Doing new Date(2012,1,31) will result in 31 march instead of 31 February(because it only has 28/29 days). So always do some sanity checking.
As to your second question:
Javascript is type unsafe.
"22" is the same as 22 when fed into the Date object.
1 == true, 0 == false, "1" == 1 == true etc.. There are some caveats here and there. But usually it's not neccesary to typecast variables.
1 + true = 2
"22"/4+0.5 = 6
And so on. Only in cases when javascript clearly can't hack(for example addition of string numbers "22"+1 = "221") it should you do explicit typecasting. Or when you need something to be a number. This is in cases where a string starts with a number, but doesn't only contain numeric characters. Like "22" will be treated a s a number, but "22foo" will be a string. You need to explicitly cast that to an integer to get the number. parseInt('22foo') will result in a 22.
Doing sanity testing on user input is always a good practise. Is the variable what I expect it to be. With integers, if the user has a chance to feed in other characters it's wise to use the isNaN function to test.
You can use the is Not A Number function of javascript to test if a string is pure. isNaN('22') will return false because the answer on the question "is it not a number" is negative. isNaN('22foo') will return true. A little bit of around the twist logic. I find it useful to remember this particular function as a question to keep my sanity :-)
Like "22foo" + 2 will turn into "22foo2". so a parseInt("22foo")+2 would result the desired 24 output.
I'm building a calendar that takes a JS object input from an RSS feed. This RSS feed is generated automatically from a CMS used internally in my company. I cannot change anything on the CMS side nor the returned RSS feed. All I have access to is the object that is built from that feed.
The RSS feed joins the start date, end date, time, and title all in one string. I need to separate them into keys in my object so my calendar can display them.
The problem I'm running into is the RSS feed formats the string differently depending on how the event is setup in the CMS. For example:
"7/15/2013 8:00 PM - 9:00 PM Blah" //Date, Time, Title
"7/12/2013 Blue" //Date for all day event, Title
"7/6/2013 8:00 AM - 7/23/2013 9:00 AM Banana" //Long event - Start Date, Start Time, End Date, End Time, Title
As you can see, how different these are I'm having a hard time deciding how I should go about parsing these into my object. The object should look like this:
{
title: 'Banana',
start: new Date(2013, 7, 24, 10, 30),
end: new Date(2013, 7, 24, 11, 30),
allDay: false
}
My question comes down to this: What would be a the best way to approach this? Use regex, try to parse it manually with things like .indexOf("/"), build test cases for each one, or some other suggestion.
PS: A jQuery example is an acceptable answer as well.
Are you able to use a library like Datejs?
You might want to start splitting on ' - ' (with spaces). If you have one part, you know it's an all day event with a title. If it's two parts, you know it's a start/end event and there's a title in the second piece. After you parse out the title, you can use Datejs to create Date objects:
Date.parse('7/15/2013 8:00 PM')
From there you should have enough to build your JSON object.
Based on Jon's answer, this is what I ended up with:
parseEntries: function() {
//Rename to fit plugin requirements
for (var i = 0; i < Calendar.entries.length; i++) {
var entry = Calendar.entries[i];
//Rename
entry["url"] = entry["link"];
delete entry["link"];
var position = entry.title.indexOf(' - ');
if (position === -1) {
//All day event
entry.allDay = true;
var space = entry.title.indexOf(" "),
title = entry.title.substring(space + 1),
firstHalf = entry.title.slice(0, space); //Start date, no time because it's all day event
} else {
var firstHalf = entry.title.slice(0, position), //Start date/time
secondHalf = entry.title.substring(position + 3);
if (secondHalf.indexOf("AM") !== -1) {
var title = secondHalf.substring(secondHalf.indexOf("AM") + 3); //Title if has AM
} else {
var title = secondHalf.substring(secondHalf.indexOf("PM") + 3); //Title if has PM
}
secondHalf = secondHalf.slice(0, -(title.length + 1)); //End date/time
}
entry["start"] = firstHalf;
entry["end"] = secondHalf;
entry.title = title;
};
I have some code that looks like this:
<h3>Thursday Sep 10</h3>
<ul>great stuff in here</ul>
<h3>Friday Sep 18</h3>
<ul>other stuff in here</ul>
They have a basic jQuery hidey-showy thing happening: click on the H3, and the UL underneath expands or contracts. By default they are all showing/open.
I'd like to set it up so that any date that has passed (like Sep 10 in this example) is contracted by default.
I don't seem to be able to get my CMS to spit out a nice compare-able date (like epoch time) to use as an ID for the H3. Is there a way I can use js's Date() function to build something like that out of the existing information ("Thursday Sep 10"), or some other way entirely to get this done?
(All of the dates are in 2009 and the site won't exist in 2010, so hardcoding the year into any functions is A-OK if that's what needs to be done.)
Thanks!
The first hurdle is to convert your text dates into valid strings that can initialize JavaScript date objects.
The conversion process (including valid string formats) is covered on http://programming.top54u.com/post/Javascript-Convert-String-to-Date.aspx.
Unfortunately, your text representations are not valid inputs for the JavaScript Date constructor. You will need to write a function to convert your strings into a valid representation before using them in the Date constructor.
Once you have your Date objects initialized, you can compare them (see "Compare Two Dates" on http://www.w3schools.com/jS/js_obj_date.asp).
I agree with Mayo. If you had access to the date time value and wanted to convert it to a string similar to what you have ("Thursday Sep 10"), you could use this.
Keep it simple; you don't really need a proper date parser to compare your dates. I've got the feeling that you're looking for something quick and dirty:
var now = new Date();
var nowHash = now.getMonth() << 5 + now.getDate();
$("h3").each(function (e) {
var tokens = $(e).html().split(" ");
var hash = [ "Jan", "Feb", ..., "Dec" ].indexOf(tokens[1]) << 5 + tokens[2];
if (hash < nowHash) {
// contract the <ul> that follows
}
});
Month * 32 (or shift 5 bits to left) + day is enough to give you a unique hash of a date that can be compared with other dates.
Mayo is correct about having no quick way. So here is my take on DIY solution.
Basically you extract the date and compare the date yourself.
var aMonths = { "Jan":0, "Feb":1 ..., "Sep":8, ... };
var aToday = new Date();
var aH3Text = .....; // Get the value via jQuery
var aDataElements = aH3Text.split("");
var aMonthIndex = aMonths[aDataElements[1]];
var aDateIndex = aDataElements[2];
var aH3Date = new Date();
aH3Date.setFullYear(2009, aMonthIndex, aDateIndex);
if (aH3Date > today) {
// BEFORE today
} else {
// AFTER today
}
Hope this helps.
My date objects in JavaScript are always represented by UTC +2 because of where I am located. Hence like this
Mon Sep 28 10:00:00 UTC+0200 2009
Problem is doing a JSON.stringify converts the above date to
2009-09-28T08:00:00Z (notice 2 hours missing i.e. 8 instead of 10)
What I need is for the date and time to be honoured but it's not, hence it should be
2009-09-28T10:00:00Z (this is how it should be)
Basically I use this:
var jsonData = JSON.stringify(jsonObject);
I tried passing a replacer parameter (second parameter on stringify) but the problem is that the value has already been processed.
I also tried using toString() and toUTCString() on the date object, but these don't give me what I want either..
Can anyone help me?
Recently I have run into the same issue. And it was resolved using the following code:
x = new Date();
let hoursDiff = x.getHours() - x.getTimezoneOffset() / 60;
let minutesDiff = (x.getHours() - x.getTimezoneOffset()) % 60;
x.setHours(hoursDiff);
x.setMinutes(minutesDiff);
JSON uses the Date.prototype.toISOString function which does not represent local time -- it represents time in unmodified UTC -- if you look at your date output you can see you're at UTC+2 hours, which is why the JSON string changes by two hours, but if this allows the same time to be represented correctly across multiple time zones.
date.toJSON() prints the UTC-Date into a String formatted (So adds the offset with it when converts it to JSON format).
date = new Date();
new Date(date.getTime() - (date.getTimezoneOffset() * 60000)).toJSON();
Just for the record, remember that the last "Z" in "2009-09-28T08:00:00Z" means that the time is indeed in UTC.
See http://en.wikipedia.org/wiki/ISO_8601 for details.
Out-of-the-box solution to force JSON.stringify ignore timezones:
Pure javascript (based on Anatoliy answer):
// Before: JSON.stringify apply timezone offset
const date = new Date();
let string = JSON.stringify(date);
console.log(string);
// After: JSON.stringify keeps date as-is!
Date.prototype.toJSON = function(){
const hoursDiff = this.getHours() - this.getTimezoneOffset() / 60;
this.setHours(hoursDiff);
return this.toISOString();
};
string = JSON.stringify(date);
console.log(string);
Using moment + moment-timezone libraries:
const date = new Date();
let string = JSON.stringify(date);
console.log(string);
Date.prototype.toJSON = function(){
return moment(this).format("YYYY-MM-DDTHH:mm:ss:ms");;
};
string = JSON.stringify(date);
console.log(string);
<html>
<header>
<script src="https://momentjs.com/downloads/moment.min.js"></script>
<script src="https://momentjs.com/downloads/moment-timezone-with-data-10-year-range.min.js"></script>
</header>
</html>
Here is another answer (and personally I think it's more appropriate)
var currentDate = new Date();
currentDate = JSON.stringify(currentDate);
// Now currentDate is in a different format... oh gosh what do we do...
currentDate = new Date(JSON.parse(currentDate));
// Now currentDate is back to its original form :)
you can use moment.js to format with local time:
Date.prototype.toISOString = function () {
return moment(this).format("YYYY-MM-DDTHH:mm:ss");
};
I'm a little late but you can always overwrite the toJson function in case of a Date using Prototype like so:
Date.prototype.toJSON = function(){
return Util.getDateTimeString(this);
};
In my case, Util.getDateTimeString(this) return a string like this: "2017-01-19T00:00:00Z"
I run into this a bit working with legacy stuff where they only work on east coast US and don't store dates in UTC, it's all EST. I have to filter on the dates based on user input in the browser so must pass the date in local time in JSON format.
Just to elaborate on this solution already posted - this is what I use:
// Could be picked by user in date picker - local JS date
date = new Date();
// Create new Date from milliseconds of user input date (date.getTime() returns milliseconds)
// Subtract milliseconds that will be offset by toJSON before calling it
new Date(date.getTime() - (date.getTimezoneOffset() * 60000)).toJSON();
So my understanding is this will go ahead and subtract time (in milliseconds (hence 60000) from the starting date based on the timezone offset (returns minutes) - in anticipation for the addition of time toJSON() is going to add.
JavaScript normally convert local timezone to UTC .
date = new Date();
date.setMinutes(date.getMinutes()-date.getTimezoneOffset())
JSON.stringify(date)
Usually you want dates to be presented to each user in his own local time-
that is why we use GMT (UTC).
Use Date.parse(jsondatestring) to get the local time string,
unless you want your local time shown to each visitor.
In that case, use Anatoly's method.
Got around this issue by using the moment.js library (the non-timezone version).
var newMinDate = moment(datePicker.selectedDates[0]);
var newMaxDate = moment(datePicker.selectedDates[1]);
// Define the data to ask the server for
var dataToGet = {"ArduinoDeviceIdentifier":"Temperatures",
"StartDate":newMinDate.format('YYYY-MM-DD HH:mm'),
"EndDate":newMaxDate.format('YYYY-MM-DD HH:mm')
};
alert(JSON.stringify(dataToGet));
I was using the flatpickr.min.js library. The time of the resulting JSON object created matches the local time provided but the date picker.
Here is something really neat and simple (atleast I believe so :)) and requires no manipulation of date to be cloned or overloading any of browser's native functions like toJSON (reference: How to JSON stringify a javascript Date and preserve timezone, courtsy Shawson)
Pass a replacer function to JSON.stringify that stringifies stuff to your heart's content!!! This way you don't have to do hour and minute diffs or any other manipulations.
I have put in console.logs to see intermediate results so it is clear what is going on and how recursion is working. That reveals something worthy of notice: value param to replacer is already converted to ISO date format :). Use this[key] to work with original data.
var replacer = function(key, value)
{
var returnVal = value;
if(this[key] instanceof Date)
{
console.log("replacer called with key - ", key, " value - ", value, this[key]);
returnVal = this[key].toString();
/* Above line does not strictly speaking clone the date as in the cloned object
* it is a string in same format as the original but not a Date object. I tried
* multiple things but was unable to cause a Date object being created in the
* clone.
* Please Heeeeelp someone here!
returnVal = new Date(JSON.parse(JSON.stringify(this[key]))); //OR
returnVal = new Date(this[key]); //OR
returnVal = this[key]; //careful, returning original obj so may have potential side effect
*/
}
console.log("returning value: ", returnVal);
/* if undefined is returned, the key is not at all added to the new object(i.e. clone),
* so return null. null !== undefined but both are falsy and can be used as such*/
return this[key] === undefined ? null : returnVal;
};
ab = {prop1: "p1", prop2: [1, "str2", {p1: "p1inner", p2: undefined, p3: null, p4date: new Date()}]};
var abstr = JSON.stringify(ab, replacer);
var abcloned = JSON.parse(abstr);
console.log("ab is: ", ab);
console.log("abcloned is: ", abcloned);
/* abcloned is:
* {
"prop1": "p1",
"prop2": [
1,
"str2",
{
"p1": "p1inner",
"p2": null,
"p3": null,
"p4date": "Tue Jun 11 2019 18:47:50 GMT+0530 (India Standard Time)"
}
]
}
Note p4date is string not Date object but format and timezone are completely preserved.
*/
I ran into the same problem.
The way I resolvet it was:
var currentTime = new Date();
Console.log(currentTime); //Return: Wed Sep 15 13:52:09 GMT-05:00 2021
Console.log(JSON.stringify(currentTime)); //Return: "2021-09-15T18:52:09.891Z"
var currentTimeFixed = new Date(currentTime.setHours(currentTime.getHours() - (currentTime.getUTCHours() - currentTime.getHours())));
Console.log(JSON.stringify(currentTimeFixed)); //Return: "2021-09-15T13:52:09.891Z"
I wrote the following code blog where it makes service calls.. it will try to serializable the json in every post submission, it will format to local date it again.
protected async post(endPoint: string, data, panelName?: string, hasSuccessMessage: boolean = false): Promise<Observable<any>> {
const options = this.InitHeader(true);
const url: string = this._baseUrl + endPoint;
Date.prototype.toJSON = function () {
return moment(this).format("YYYY-MM-DDThh:mm:00.000Z");;
};
return await this._http.post(url, data, options).pipe(map(response => {
return this.Map<any>(response, null);
}));
}
All boils down to if your server backend is timezone-agnostic or not.
If it is not, then you need to assume that timezone of server is the same as client, or transfer information about client's timezone and include that also into calculations.
a PostgreSQL backend based example:
select '2009-09-28T08:00:00Z'::timestamp -> '2009-09-28 08:00:00' (wrong for 10am)
select '2009-09-28T08:00:00Z'::timestamptz -> '2009-09-28 10:00:00+02'
select '2009-09-28T08:00:00Z'::timestamptz::timestamp -> '2009-09-28 10:00:00'
The last one is probably what you want to use in database, if you are not willing properly implement timezone logic.
Instead of toJSON, you can use format function which always gives the correct date and time + GMT
This is the most robust display option. It takes a string of tokens
and replaces them with their corresponding values.
I tried this in angular 8 :
create Model :
export class Model { YourDate: string | Date; }
in your component
model : Model;
model.YourDate = new Date();
send Date to your API for saving
When loading your data from API you will make this :
model.YourDate = new Date(model.YourDate+"Z");
you will get your date correctly with your time zone.
In this case I think you need transform the date to UNIX timestamp
timestamp = testDate.getTime();
strJson = JSON.stringify(timestamp);
After that you can re use it to create a date object and format it. Example with javascript and toLocaleDateString ( https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Objets_globaux/Date/toLocaleDateString )
newDateObject = new Date(JSON.parse(strJson));
newDateObject = newDateObject.toLocalDateStrin([
"fr-FR",
]);
If you use stringify to use AJAX, now it's not useful. You just need to send timestamp and get it in your script:
$newDateObject = new \DateTime();
$newDateObject->setTimestamp(round($timestamp/1000));
Be aware that getTime() will return a time in milliseconds and the PHP function setTimestamp take time in seconds. It's why you need to divide by 1000 and round.
In Angular place the following in index.js script section:
setTimeout(function (){
Date.prototype.toJSON = function(){
return new Date(this).toLocaleDateString("en-US") + " "+new Date(this).toLocaleTimeString();
}},1000);