I've been banging my head against the wall trying to get a JavaScript equivalent to this php snippet:
<?php
$id = 'uniqueID'
$now = round(time()/60);
$lock = md5($now . $id);
?>
I've been trying variations on this:
var timeInMin = new Date().getTime() / 60000;
var timestamp = Math.round(timeInMin);
var key = md5(timestamp + 'uniqueID');
utilizing an md5 script from here
I merely need lock and key to match. Seems simple to me. What am I doing wrong?
As said before me, if time not matching it will not create the same hash. What I do in situations like that is to find way to pass the time from php to the client side so they can use the same exact time.
PHP side:
<?php
$id = 'uniqueID';
$now = round(time()/60);
$lock = md5($now . $id);
print $lock;
setcookie("time",$now);
?>
Client Side:
<script type="text/javascript">
var timestamp = getCookie("time");
var key = md5(timestamp + 'uniqueID');
console.log(key);
</script>
Note that getCookie is a shortcut function
The following example is here to present the idea in a simple form. I would not use time as the name of the cookie nor give access to the vars (wrap in function). Uglify scripts goes a long way in cases like this.
To put it in concrete terms (my comment was slightly facetious):
PHP is a server-side language. When your browser fires a request for a page over the internet (or even to a listening port on your local machine), the instance of apache running on the server (or your local machine) executes the PHP on the page, then spits it back out to the browser.
The browser then executes any JavaScript on the page, we refer to this as client-side.
Because you are using Math.round, if it takes more than 30 seconds between the time your server executes the PHP (server-side) and the time your browser starts executing the relevant Javascript (client-side) then the time in minutes will be different. Using Math.floor instead would give you 59 seconds of wiggle room but that's still dicey, especially on mobile.
Even assuming the page executes the JavaScript immediately after loading and loads pretty quickly 30 seconds of latency is not totally off the table, and on mobile neither is 59.
I've seen some answers to this that refer the askee to other libraries (like phantom.js), but I'm here wondering if it is at all possible to do this in just node.js?
Considering my code below. It requests a webpage using request, then using cheerio it explores the dom to scrape the page for data. It works flawlessly and if everything had gone as planned, I believe it would have outputted a file as i imagined in my head.
The problem is that the page I am requesting in order to scrape, build the table im looking at asynchronously using either ajax or jsonp, i'm not entirely sure how .jsp pages work.
So here I am trying to find a way to "wait" for this data to load before I scrape the data for my new file.
var cheerio = require('cheerio'),
request = require('request'),
fs = require('fs');
// Go to the page in question
request({
method: 'GET',
url: 'http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp'
}, function(err, response, body) {
if (err) return console.error(err);
// Tell Cherrio to load the HTML
$ = cheerio.load(body);
// Create an empty object to write to the file later
var toSort = {}
// Itterate over DOM and fill the toSort object
$('#emb table td.list_right').each(function() {
var row = $(this).parent();
toSort[$(this).text()] = {
[$("#lastdate").text()]: $(row).find(".idx1").html(),
[$("#currdate").text()]: $(row).find(".idx2").html()
}
});
//Write/overwrite a new file
var stream = fs.createWriteStream("/tmp/shipping.txt");
var toWrite = "";
stream.once('open', function(fd) {
toWrite += "{\r\n"
for(i in toSort){
toWrite += "\t" + i + ": { \r\n";
for(j in toSort[i]){
toWrite += "\t\t" + j + ":" + toSort[i][j] + ",\r\n";
}
toWrite += "\t" + "}, \r\n";
}
toWrite += "}"
stream.write(toWrite)
stream.end();
});
});
The expected result is a text file with information formatted like a JSON object.
It should look something like different instances of this
"QINHUANGDAO - GUANGZHOU (50,000-60,000DWT)": {
"2016-09-29": 26.7,
"2016-09-30": 26.8,
},
But since the name is the only thing that doesn't load async, (the dates and values are async) I get a messed up object.
I tried Actually just setting a setTimeout in various places in the code. The script will only be touched by developers that can afford to run the script several times if it fails a few times. So while not ideal, even a setTimeout (up to maybe 5 seconds) would be good enough.
It turns out the settimeouts don't work. I suspect that once I request the page, I'm stuck with the snapshot of the page "as is" when I receive it, and I'm in fact not looking at a live thing I can wait for to load its dynamic content.
I've wondered investigating how to intercept the packages as they come, but I don't understand HTTP well enough to know where to start.
The setTimeout will not make any difference even if you increase it to an hour. The problem here is that you are making a request against this url:
http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp
and their server returns back the html and in this html there are the js and css imports. This is the end of your case, you just have the html and that's it. Instead the browser knows how to use and to parse the html document, so it is able to understand the javascript scripts and to execute/run them and this is exactly your problem. Your program is not able to understand that has something to do with the HTML contents. You need to find or to write a scraper that is able to run javascript. I just found this similar issue on stackoverflow:
Web-scraping JavaScript page with Python
The guy there suggests https://github.com/niklasb/dryscrape and it seems that this tool is able to run javascript. It is written in python though.
You are trying to scrape the original page that doesn't include the data you need.
When the page is loaded, browser evaluates JS code it includes, and this code knows where and how to get the data.
The first option is to evaluate the same code, like PhantomJS do.
The other (and you seem to be interested in it) is to investigate the page's network activity and to understand what additional requests you should perform to get the data you need.
In your case, these are:
http://index.chineseshipping.com.cn/servlet/cbfiDailyGetContrast?SpecifiedDate=&jc=jsonp1475577615267&_=1475577619626
and
http://index.chineseshipping.com.cn/servlet/allGetCurrentComposites?date=Tue%20Oct%2004%202016%2013:40:20%20GMT+0300%20(MSK)&jc=jsonp1475577615268&_=1475577620325
In both requests:
_ is a decache parameter to prevent caching.
jc is a name of a JS wrapper function which should be invoked with the result (https://en.wikipedia.org/wiki/JSONP)
So, scrapping the table template at http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp and performing two additional requests you will be able to combine them into the same data structure you see in the browser.
[IE9] Consider the following code:
var scr = document.createElement("script");
scr.src = "http://collector.bonzai.mobi/expfreq?hours=1&freq=1&adid=abc&sn=source1&pub=publisher";
scr.onload = function(){
alert("openPage " + openPage);
}
document.body.appendChild(scr);
The problem is when I put this code directly in a <script> tag it works fine (means every time it contact server for retreiving the script). But if I keep the same code inside document.onclick function the IE9 is taking the script from cache 2nd time on wards. (I can see the 304 and <1ms)
PS : I cannot use Date.now() or any sort of cache bursting mechanism to append at the end of URL because the backend implementation depends of Cache (ETag).
Also I would suggest you clear your cache in IE browser before attempting each code change.
Any advice would be of great help. Thanks.
I got the solution for the problem. Setting "Cache-Control:private, must-revalidate" header from server solved the problem.
a solution from the client side could be :
scr.src = "http://collector.bonzai.mobi/expfreq?hours=1&freq=1&adid=abc&sn=source1&pub=publisher×tamp=" + new Date().getTime();
So each request won't be cached.
I'm currently programming in JSP and Javascript. (I am by no means an expert in either). Right now, what I want is for a Javascript function to be called repeatedly and one of the variables to be queried from the database repeatedly (it is the date that the page was last modified). If this variable is greater than when the page was loaded, I want the page to refresh.
What I have so far:
...
<body onload="Javascript:refreshMethod()">
<script type="text/JavaScript">
<!--
function refreshMethod()
{
var interval = setInterval("timedRefresh()", 10000);
}
function timedRefresh() {
var currenttime = '<%=currentTime%>';
var feedlastmodified = '<%=EventManager.getFeedLastModified(eventID)%>';
var currenttimeint = parseInt(currenttime);
var feedlastmodifiedint = parseInt(feedlastmodified);
if(feedlastmodifiedint > currenttimeint)
{
alert(feedlastmodifiedint);
setTimeout("location.reload(true);",timeoutPeriod);
}
if(feedlastmodifiedint < currenttimeint)
{
alert(feedlastmodifiedint + " : " + currenttimeint);
}
}
// -->
</script>
The problem is that everytime the timedRefresh runs, the feedlastModifiedInt never changes (even if it has been changed).
Any help would be appreciated,
Thanks.
The JSP code within the <% ... %> tags runs only once, on the server-side, when the page is loaded. If you look at the source of the page in the browser, you will find that these values have already been placed within the JavaScript code, and thus they will not change during each timer interval.
To update the data as you are expecting, you can use AJAX. You can find plenty of tutorials online.
JSP and JavaScript doesn't run in sync as you seem to expect from the coding. JSP runs at webserver, produces a bunch of characters which should continue as HTML/CSS/JS and the webserver sends it as a HTTP response to the webbrowser as response to a HTTP request initiated by the webbrowser. Finally HTML/CSS/JS runs at the webbrowser.
If you rightclick the page in webbrowser and choose View Source, you'll probably understand what I mean. There's no single line of Java/JSP code. It has already done its job of generating the HTML/CSS/JS. The only communication way between Java/JSP and JavaScript is HTTP.
You need to move this job to some servlet in the server side and let JS invoke this asynchronously ("in the background"). This is also known as "Ajax". Here's a kickoff example with a little help of jQuery.
<script src="http://code.jquery.com/jquery-latest.min.js"></script>
<script>
$(document).ready(function() {
var refreshInterval = setInterval(function() {
$.getJSON('refreshServlet', function(refresh) {
if (refresh) {
clearInterval(refreshInterval);
location.reload(true);
}
});
}, 10000);
});
</script>
Where the doGet() method of the servlet which is mapped on an url-pattern of /refreshServlet roughly look like this:
response.setContentType("application/json");
if (EventManager.getFeedLastModified(eventID) > currentTime) {
response.getWriter().write("true");
} else {
response.getWriter().write("false");
}
See also:
Communication between Java/JSP/JSF and JavaScript
How can i get the current time? (in JavaScript)
Not the time of your computer like:
now = new Date;
now_string = addZero(now.getHours()) + ":" + addZero(now.getMinutes()) + ":" + addZero(now.getSeconds());
But the real accurate world time?
Do i need to connect to to a server (most likely yes, which one? and how can i retrieve time from it?)
All the searches I do from google return the (new Date).getHours().
Edit:
I want to avoid showing an incorrect time if the user has a wrong time in his computer.
You can use JSON[P] and access a time API:
(The code below should work perfectly, just tested it...)
function getTime(zone, success) {
var url = 'http://json-time.appspot.com/time.json?tz=' + zone,
ud = 'json' + (+new Date());
window[ud]= function(o){
success && success(new Date(o.datetime));
};
document.getElementsByTagName('head')[0].appendChild((function(){
var s = document.createElement('script');
s.type = 'text/javascript';
s.src = url + '&callback=' + ud;
return s;
})());
}
getTime('GMT', function(time){
// This is where you do whatever you want with the time:
alert(time);
});
First, to get the accurate GMT time you need a source that you trust. This means some server somewhere. Javascript can generally only make HTTP calls, and only to the server hosting the page in question (same origin policy). Thus that server has to be your source for GMT time.
I would configure your webserver to use NTP to synchronize its clock with GMT, and have the webserver tell the script what time it is, by writing a variable to the page. Or else make and XmlHttpRequest back to the server when you need to know the time. The downside is that this will be inaccurate due to the latency involved: the server determines the time, writes it to the response, the response travels over the network, and the javascript executes whenever the client's cpu gives it a timeslice, etc. On a slow link you can expect seconds of delay if the page is big. You might be able to save some time by determining how far off from GMT the user's clock is, and just adjusting all the time calculations by that offset. Of course if the user's clock is slow or fast (not just late or early) or if the user changes the time on their PC then your offset is blown.
Also keep in mind that the client can change the data so don't trust any timestamps they send you.
Edit:
JimmyP's answer is very simple and easy to use: use Javascript to add a <script> element which calls a url such as http://json-time.appspot.com/time.json?tz=GMT. This is easier than doing this yourself because the json-time.appspot.com server works as a source of GMT time, and provides this data in a way that lets you work around the same-origin policy. I would recommend that for simple sites. However it has one major drawback: the json-time.appspot.com site can execute arbitrary code on your user's pages. This means that if the operators of that site want to profile your users, or hijack their data, they can do that trivially. Even if you trust the operators you need to also trust that they have not been hacked or compromised. For a business site or any site with high reliability concerns I'd recommend hosting the time solution yourself.
Edit 2:
JimmyP's answer has a comment which suggests that the json-time app has some limitations in terms of the number of requests it can support. This means if you need reliability you should host the time server yourself. However, it should be easy enough to add a page on your server which responds with the same format of data. Basically your server takes a query such as
http://json-time.appspot.com/time.json?tz=America/Chicago&callback=foo
and returns a string such as
foo({
"tz": "America\/Chicago",
"hour": 15,
"datetime": "Thu, 09 Apr 2009 15:07:01 -0500",
"second": 1,
"error": false,
"minute": 7
})
Note the foo() which wraps the JSON object; this corresponds to the callback=foo in the query. This means when the script is loaded into the page it will call your foo function, which can do whatever it wants with the time. Server-side programming for this example is a separate question.
Why don't you send the time with every page? For example somewhere in the html:
<span id="time" style="display:none;">
2009-03-03T23:32:12
</span>
Then you could run a Javascript while the site loads and interpret the date. This would reduce the amount of work the network has to do. You can store the corresponding local time and calculate the offset every time you need it.
You could use getTimezoneOffset to get the offset between the local date and the GMT one, and then do the math. But this will only be as accurate as the user's clock.
If you want an accurate time, you should connect to a NTP server. Because of the Same Origin Policy, you can't make a request with JS to another server then yours. I'd suggest you to create a server-side script that connects to the NTP server (in PHP, or whatever language you want) and return the accurate date. Then, use an AJAX request to read this time.
A little addition to Mr. Shiny and New and James answers
Here is a PHP script which you can place on own server and use instead of json-time.appspot.com
<?php
header('Content-Type: application/json');
header("Expires: Tue, 01 Jan 1990 00:00:00 GMT");
header("Cache-Control: no-store, no-cache, must-revalidate, max-age=0");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");
$error = "false";
$tz = $_GET['tz'];
if ( !in_array($tz, DateTimeZone::listIdentifiers())) {
$error = 'invalid time zone';
$tz = 'UTC';
}
date_default_timezone_set($tz);
?>
<?php echo htmlspecialchars($_GET['callback'], ENT_QUOTES, 'UTF-8' ); ?>({
"tz": "<?php echo $tz ?>",
"hour": <?php echo date('G'); ?>,
"datetime": "<?php echo date(DATE_RFC2822); ?>",
"second": <?php echo intval(date('s')); ?>,
"error": "<?php echo $error; ?>",
"minute": <?php echo intval(date('i')); ?>
})
Like Mr. Shiny and New said, you need a server somewhere with correct time. It can be the server where you site are or some other server that sends the correct time in a format that you can read.
If you want to use the date several times on your page, one or more seconds apart, you probably don't want to get the time from the server every time, but instead cache the difference and use the clients clock. If that is the case, here is one of many solutions:
var MyDate = new function() {
this.offset = 0;
this.calibrate = function (UTC_msec) {
//Ignore if not a finite number
if (!isFinite(UTC_msec)) return;
// Calculate the difference between client and provided time
this.offset = UTC_msec - new Date().valueOf();
//If the difference is less than 60 sec, use the clients clock as is.
if (Math.abs(this.offset) < 60000) this.offset = 0;
}
this.now = function () {
var time = new Date();
time.setTime(this.offset + time.getTime());
return time;
}
}();
Include it on your page and let your server side script produce a row like:
MyDate.calibrate(1233189138181);
where the number is the current time in milliseconds since 1 Jan 1970. You can also use your favorite framework for AJAX and have it call the function above. Or you can use the solution JimmyP suggested. I have rewritten JimmyPs solution to be included in my solution. Just copy and paste the following inside the function above:
this.calibrate_json = function (data) {
if (typeof data === "object") {
this.calibrate (new Date(data.datetime).valueOf() );
} else {
var script = document.createElement("script");
script.type="text/javascript";
script.src=(data||"http://json-time.appspot.com/time.json?tz=UTC") +
"&callback=MyDate.calibrate_json";
document.getElementsByTagName('head')[0].appendChild(script);
}
}
this.calibrate_json(); //request calibration with json
Notice that if you change the name of the function from MyDate you have to update the callback in this.calibrate_json on the line script.src.
Explanation:
Mydate.offset is the current offset between the server time and the clients clock in milliseconds.
Mydate.calibrate( x ); is a function that sets a new offset. It expects the input to be the current time in milliseconds since 1 Jan 1970. If the difference between the server and client clock is less than 60 seconds, the clients clock will be used.
Mydate.now() is a function that returns a date object that has the current calibrated time.
Mydate.calibrate_json( data ) is a function that either takes an url to a resource that gives back a datetime reply, or an object with the current time (used as a callback). If nothing is supplied, it will use a default url to get the time. The url must have a question mark "?" in it.
Simple example of how to update an element with the current time every second:
setInterval(
function () {
var element = document.getElementById("time");
if (!element) return;
function lz(v) {
return v < 10 ? "0" + v : v;
}
var time = MyDate.now();
element.innerHTML = time.getFullYear() + "-" +
lz(time.getMonth() + 1) + "-" +
lz(time.getDate()) + " " +
lz(time.getHours()) + ":" +
lz(time.getMinutes()) + ":" +
lz(time.getSeconds())
;
},1000);
I stumbled upon another solution for those who do not have access to the server side of things. An answer to the question Getting the default server time in jQuery?.
Basically, you can grab the "Date" header by doing an AJAX HEAD request to "/". Not all servers support this and it may take some jiggery-pockery to get it working with a particular server.
Check out the real answer for more details.
IMO, simplest solution is spinning up an AWS Lambda (or Google Serverless) and attaching it via API Gateway, giving you a timeserver without managing any infrastructure. My Lambda code:
import datetime
import json
def lambda_handler(event, context):
dt = datetime.datetime.utcnow()
return {
'statusCode': 200,
'body': json.dumps({
'iso_time': dt.strftime('%Y-%m-%dT%H:%M:%SZ')
})
}
We've used an API from EarthTools with much success:
EarthTools
The service returns XML with information such as the current GMT & timezone offsets (when supplying a latitude & longitude). We used a WebClient in conjunction with an XMLDocument in the VB backend of our ASP.NET page to read & interpret the XML.
Since the service related to the James's answer seems no longer working, I found another good rest API which can be used for this purpose, and it works fine.
The source is this site: timeapi and the code that you can use is simply this (using jQuery library and jsonp callback provided by the API):
$.getJSON("http://www.timeapi.org/utc/now.json?callback=?",function(json){
//console.log(json)
alert(json.dateString);
});
With this you will have the date and hour in the fomat like
September 02, 2016 09:59:42 GMT+0100
If you want to manipulate it in javascript, to have the hours, just transform it in a Date object like this:
new Date(json.dateString)
So, for example, you can write like this:
$.getJSON("http://www.timeapi.org/utc/now.json?callback=?",function(json){
var now = new Date(json.dateString);
var hours = now.toLocaleTimeString();
alert(hours)
});
in this way you will have the exact local time in hours.
Your can do this in PHP
<?php
$json = file_get_contents( "http://json-time.appspot.com/time.json?tz=GMT" );
$obj = json_decode($json);
$timenow= $obj->{'datetime'};
$hournow= $obj->{'hour'};
$minutenow= $obj->{'minute'};
$secondnow= $obj->{'second'};
?>
$timenow contains your full date & time: "Mon, 05 Mar 2012 01:57:51 +0000", $hournow: "1"
$minutenow: "57".
If your javascript is served from web server then you can use this simple function to request time from this server. It will work if server returns Date header in HTTP response.
function getServerTime()
{
var req = new XMLHttpRequest();
req.open('GET', document.location, false);
req.send(null);
return new Date(req.getResponseHeader("Date"));
}
Bacause of caching this may return wrong time. To fix this you can use random URL:
req.open('GET', "/time?r=" + Math.floor(Math.random()*10000000), false);