We Keep Coding

Scraping a site with PHP which has variables in double curly brackets instead of text [duplicate]

This question already has answers here:
Scrape web page data generated by javascript
(2 answers)
Closed 8 years ago.
I am stuck with a scraping task in my project.
i want to grab the data from the link in $html , all table content of tr and td , here i am trying to grab the link but it only shows javascript: self.close()
<?php
include("simple_html_dom.php");
$html = file_get_html('http://www.areacodelocations.info/allcities.php?ac=201');
foreach($html->find('a') as $element)
echo $element->href . '<br>';
?>

Usually, this kind of pages load a bunch of Javascript (jQuery, etc.), which then builds the interface and retrieves the data to be displayed from a data source.
So what you need to do is open that page in Firefox or similar, with a tool such as Firebug in order to see what requests are actually being done. If you're lucky, you will find it directly in the list of XHR requests. As in this case:
http://www.govliquidation.com/json/buyer_ux/salescalendar.js
Notice that this course of action may infringe on some license or terms of use. Clear this with the webmaster/data source/copyright owner before proceeding: detecting and forbidding this kind of scraping is very easy, and identifying you is probably only slightly less so.
Anyway, if you issue the same call in PHP, you can directly scrape the data (provided there is no session/authentication issue, as seems the case here) with very simple code:
<?php
$url = "http://www.govliquidation.com/json/buyer_ux/salescalendar.js";
$json = file_get_contents($url);
$data = json_decode($json);
?>
This yields a data object that you can inspect and convert in CSV by simple looping.
stdClass Object
(
[result] => stdClass Object
(
[events] => Array
(
[0] => stdClass Object
(
[yahoo_dur] => 11300
[closing_today] => 0
[language_code] => en
[mixed_id] => 9297
[event_id] => 9297
[close_meridian] => PM
[commercial_sale_flag] => 0
[close_time] => 01/06/2014
[award_time_unixtime] => 1389070800
[category] => Tires, Parts & Components
[open_time_unixtime] => 1388638800
[yahoo_date] => 20140102T000000Z
[open_time] => 01/02/2014
[event_close_time] => 2014-01-06 17:00:00
[display_event_id] => 9297
[type_code] => X3
[title] => Truck Drive Axles # Killeen, TX
[special_flag] => 1
[demil_flag] => 0
[google_close] => 20140106
[event_open_time] => 2014-01-02 00:00:00
[google_open] => 20140102
[third_party_url] =>
[bid_package_flag] => 0
[is_open] => 1
[fda_count] => 0
[close_time_unixtime] => 1389045600
You retrieve $data->result->events, use fputcsv() on its items converted to array form, and Bob's your uncle.

In the case of the second site, you have a table with several TR elements, and you want to catch the first two TD children of each TR.
By inspecting the source code you see something like this:
<tr>
<td> Allendale</td>
<td> Eastern Time
</td>
</tr>
<tr>
<td> Alpine</td>
<td> Eastern Time
</td>
So you just grab all the TR's
<?php
include("simple_html_dom.php");
$html = file_get_html('http://www.areacodelocations.info/allcities.php?ac=201');
$fp = fopen('output.csv', 'w');
if (!$fp) die("Cannot open output CSV - permission problems maybe?");
foreach($html->find('tr') as $tr) {
$csv = array(); // Start empty. A new CSV row for each TR.
// Now find the TD children of $tr. They will make up a row.
foreach($tr->find('td') as $td) {
// Get TD's innertext, but
$csv[] = $td->innertext;
}
fputcsv($fp, $csv);
}
fclose($fp);
?>
You will notice that the CSV text is "dirty". That is because the actual text is:
<td> Alpine</td>
<td> Eastern Time[CARRIAGE RETURN HERE]
</td>
So to have "Alpine" and "Eastern Time", you have to replace
$csv[] = $td->innertext;
with something like
$csv[] = strip(
html_entity_decode (
$td->innertext,
ENT_COMPAT | ENT_HTML401,
'UTF-8'
)
);
Check out the PHP man page for html_entity_decode() about character set encoding and entity handling. The above ought to work -- and an ought and fifty cents will get you a cup of coffee :-)

PHP POST for HTML Select using another POST

This ain't something not working but just I'm confusing about how to do it, I want to fetch the values from my DB based on users' preferences that been chosen earlier.
These are the steps will be taken for my process:
User will select from images (will add HTML images then will whip it upon another selection)
Will continue till reaching last stage
Last stage will have 3 Select (dropdown menus) and 2 of them will change the content according to what user's chooses (like country and state dd)
My PHP:
else if ($_POST["data_key"]=="last")
{
$final_arr;
$fetcher_theme = $_POST["themeid"];
$fetcher_category= $_POST["themecategory"];
$fetcher_product= $_POST["themeproduct"];
$fetcher_cover = $_POST["ctitle"];
$myquery="SELECT DISTINCT Layout.* FROM Layout,Products,Occasion, Cover, Theme
WHERE Layout.product=$fetcher_product
AND Layout.occasion=$fetcher_category
AND Layout.theme=$fetcher_theme
AND Layout.cover=$fetcher_cover;";
$results=$DB->fetchAll($myquery);
foreach ($results as $row) {
$row["current"]="size";
unset($row["pixfizzId"]);
$final_arr[]=$row;
}
echo json_encode($final_arr);
}
else if ($_POST["data_key"]=="size")
{
$final_arr;
$fetcher = $_POST["selected_id"];
$fetcher_theme = $_POST["themeid"];
$fetcher_category= $_POST["themecategory"];
$fetcher_product= $_POST["themeproduct"];
$fetcher_cover = $_POST["ctitle"];
$fetcher_size = $_POST["stitle"];
$myquery="SELECT DISTINCT Size.stitle FROM Layout,Products,Occasion, Size, Theme
WHERE Layout.product=$fetcher_product
AND Layout.occasion=$fetcher_category
AND Layout.theme=$fetcher_theme
AND Layout.size=$fetcher_size
AND Layout.size=Size.id";
$results=$DB->fetchAll($myquery);
foreach ($results as $row) {
$row["current"]="finishing";
unset($row["pixfizzId"]);
$final_arr[]=$row;
}
echo json_encode($final_arr);
}
else if ($_POST["data_key"]=="finishing")
{
$final_arr;
$fetcher = $_POST["selected_id"];
$fetcher_theme = $_POST["themeid"];
$fetcher_category= $_POST["themecategory"];
$fetcher_product= $_POST["themeproduct"];
$fetcher_cover = $_POST["ctitle"];
$fetcher_size = $_POST["stitle"];
$fetcher_finishing = $_POST["ftitle"];
$myquery="SELECT DISTINCT Finishing.ftitle FROM Layout,Products,Occasion, Size, Cover, finishing, Theme
WHERE Layout.product=$fetcher_product
AND Layout.occasion=$fetcher_category
AND Layout.theme=$fetcher_theme
AND Layout.size=$fetcher_size
AND Layout.cover=$fetcher_cover
AND Layout.finishing=$fetcher_finishing";
$results=$DB->fetchAll($myquery);
foreach ($results as $row) {
$row["current"]="finishing";
unset($row["pixfizzId"]);
$final_arr[]=$row;
}
echo json_encode($final_arr);
}}
My Engine JS (what I assign):
myItem.setId(jsonData[i].id);
myItem.setImg(jsonData[i].image);
myItem.setTitle(jsonData[i].title);
My Selects in JS (printing HTML):
myString +="<a>Sizes: </a><br><select id='sizesSelect' style=' width:200px'></select><br><br>";
myString +="<a>Cover: </a><br><select id='coverSelect' style=' width:200px'><br></select><br><br>";
myString +="<a>Finishing: </a><br><select id='finishingSelect' style=' width:200px'></select><br><br><br>";
Appending to Select in JS:
myString +="<script>$('#sizesSelect').append('<option val="+i+">"+this.getSize()+"</option>')</script>";
Now I need to know how can I post again to my PHP server to fetch the the values to other selects (refer for the img).
Select Size -> Update Covers -> Select Covers -> Update Finishing -> Select Finishing

Instead of using SQL to store and present previous user choices, you could keep the previous choices as $_POST data by re-adding the choises to the HTML Form as hidden input fields. Example: <input type="hidden" text="foo" name="<?=htmlspecialchars($_POST['foo'])?>">

Timepiker value is not going to php database (Datatype : datetime) variable?

I take time piker to take time from my web page.
This is my front end code.
<input type="text" class="form-control timepicker" id="event_start_timepicker" placeholder="select time" name="startAt">
I use Laravel 5 frame work . so I access this data in my controller class. I want to sent this value in my phpMyAdmin database datetime varable.
This is my controller class code.
public function store(Request $request) {
//
$event = Request::all();
//dd($event['startAt']); out put : "02:45 PM" //data is comming correctly
//try to format data
$s_time = date_create($event['startAt'])->format('Y-m-d h:i:s A');
$e_time = date_create($event['endAt'])->format('Y-m-d h:i:s A');
//reassigning format data
$event['startAt']=$s_time;
$event['endAt'] =$e_time;
Event::create($event);
return redirect('event');
}
first time I run without formate , I check database and I see datetime variable value is 0000-00-00 00:00:00 . so , I formate but still same result other all details passed correctly. expect some answer. please help me.
this is script.
//Timepicker
$("#event_start_timepicker").timepicker({
showInputs: false,
format:"yyyy-mm-dd h:i:s A"
});
$("#event_end_timepicker").timepicker({
showInputs: false,
format:"yyyy-mm-dd h:i:s A"
});

conditionally format php output with javascript

I would like to conditionally apply (based on date and time) with Javascript different CSS styles on a <div> element that contains data output with Php from a Mysql database. The php generates a list of events into various <div>s, where a <div> would receive different coloring depending on the date/time it holds relative to the actual day. (So if the event date/time is in the future, the color would be blue, if in the past, grey.)
I understand a PHP solution would use the server's date/time, while javascript would use the browser's date/time, so I selected javascript for the conditional formatting based on date/time.
I also understand that javascript needs to wait until the whole page loads, including the result of the php query.
(I also understand that I know very little about javascript and php.)
Below is the html and php code I have. This php queries the first 10 events and presents them in a list ordered by date in a bootstrap table. The div #timecode is which I would like to conditionally format (this sample php code does not contain the pagination php code, and returns the first 10 records from the database):
<div class="row">
<div class="venuelista">
<div class="col-sm-9 col-md-9 col-lg-9">
<table class="table">
<tbody>
<?php
while ($row = mysql_fetch_assoc($rs_result)) {
?>
<tr>
<td class="col-md-7">
<div id="timecode" padding="right:5px"><?php echo $row['esemeny_disp_mm_dd']; ?></div>
<div class="eventListTitle"><?php echo $row['esemeny_cim']; ?><br><?php echo $row['esemeny_jelleg']; ?></div>
</td>
<td class="col-md-5">
<div class="eventListTitle"><?php echo $row['esemeny_helyszin']; ?>
</div>
</td>
</tr>
<?php
};
?>
</tbody>
</table>
</div>
</div>
In the Mysql database I have and 'event date' field and a separate 'event time' field.
And this is the javascript I found on the net, maybe on this site, to format the div #timecode on the basis of its date/time (this is just testing if it works against a fix date - it does):
<script>
$(document).ready(function() {
var curtime = new Date(),
curday = curtime.getDate(),
curmonth = curtime.getMonth()+1;
if(curmonth == 7 && curday == 20)
$('#timecode').addClass('circle-disappear-blue');
else
$('#timecode').addClass('circle-disappear-grey');
});
</script>
Problem: javascript only formats the very first php result of the list. I thought the $(document).ready(function() would allow it to format all 10 list items.
Again I am sorry for malformatting my question - I am not on mutual understanding with the code sample function.
Any insight into this is appreciated.
EDIT:
The javascript now looks like this
<script>
$(document).ready(function() {
$('#mytableid tr td:first-child').addClass('circle-disappear-blue');
var curtime = new Date(),
curday = curtime.getDate(),
curmonth = curtime.getMonth()+1;
if(curmonth == 7 && curday == 19)
$('#blue-condtional').addClass('circle-disappear-blue');
else
$('#blue-condtional').addClass('circle-disappear-grey');
});
</script>

add id to your table for example
<table class="table" id="mytableid">
then use proper selector
$('#mytableid tr td:first-child').addClass('circle-disappear-blue');
the other option is to add class to types that you want change color let's say "blue-condtional"
if(curmonth == 7 && curday == 20) {
$('.blue-condtional').addClass('circle-disappear-blue');
} else {
$('.blue-condtional').addClass('circle-disappear-grey');
}
and change
<div id="timecode" padding="right:5px">
to
<div class="blue-condtional">
btw using inline styles is not good habbit.

Angular performance issues, ng-repeat, unwanted callbacks

I'm facing some serious performance issues in my angular app, surely due to my bad ng-repeat implementation but I don't know how to do better. I've tried some other combinations but none give me the result I want, nor better performance. There have been some infinite loops, undefined variables due to asynch and $digest aborting.
This is how it should be looking (so that's the working result, but not optimized for now) : http://scr.hu/0ei9/8bpw9
Days with blue cells = user is present, red cell = absent. Beneath are the "shifts" per day (each shift can be done on various days).
When I console.log some of the functions, they are called way too many times than needed, for example isPresent (for checking if the user submited the day as available) is called 5x180 (180 being probably 5 users x 30 days of month) and getDayOfWeek (used in template for the shifts row) 28+840+812...there are 10 shifts. Whereas I would normally need to call isPresent only 1x180 for all cells and getDayOfWeek 1x30 days of month for one shifts row. That said, I don't really get why its called sooo many times and how to optimize it.
In addition, when I click on a blue cell, the setShift() function is called. I copied the function here but don't analyze it, I only wanted to show that it nowhere calls any of the previous functions, however I console.log an impressive 28x "days of the week" + 180 "is present" and + 812 "day of the week". No Idea where it comes from... to be frank, everything i click calls this same amout - if I change the month, year, click on an independent button...
I omitted some classes and css for better code reading.
Thanks for your help.
Edit
after commenting and decommenting some of the code I see that the most of those unwanted logs when I call the setShift() function come from the last sub-call : showPopover() where I open a $modal. When commented, there is like 1/3 of what I'm used to see. Before the modal appears there is the rest of this stuff. Furthermore I think that the temlpateUrl might be the cause because when commented it does not log all those houndreds of 'is present' and 'day of week'. And when I click anywhere on the modal, it recalls all those functions. Any ideas?
JS angular functions
$scope.isPresent = function(day, month, year, userId) {
console.log('is present')
var date = new Date(year, month - 1, day + 1)
var day = moment(date).format("D-M-YYYY");
for (var i = 0; i < $scope.presences.length; i++) {
var presentDay = moment($scope.presences[i].start).format("D-M-YYYY")
if (presentDay == day && userId == $scope.presences[i].coursierId) {
return true
}
}
}
$scope.getDayOfWeek = function(day, month, year) {
console.log('day of the week')
var date = new Date(parseInt(year), month - 1, day + 1)
var dayId = date.getDay();
return dayId;
}
/*
used for a limited ng-repeat
*/
$scope.getTimes = function(n) {
return new Array(n);
};
/*
Shows a list of sorted out shifts with corresponding
hours, cities and capabilities of coursier.
*/
$scope.setShift = function(day, month, year, userId, event) {
var date = new Date(year, month - 1, day)
var day = moment(date).format("D-M-YYYY");
var dayOfWeek = moment(date).day()
//SORT SHIFTs BY DAY OF WEEK clicked
var day_shifts = $scope.sortShiftByDayOfWeek(dayOfWeek)
//console.log(day_shifts)
//check if the day clicked is an dispo present day
for (var i = 0; i < $scope.presences.length; i++) {
var dispoDay = moment($scope.presences[i].start).format("D-M-YYYY")
//if yes, check the presence user id and the cell of user clicked
if (dispoDay == day) {
//then get all the shifts that fit into this coursier's time range
if ($scope.presences[i].coursierId == userId) {
var dispo = $scope.presences[i];
var dispoHours = $scope.getDispoHoursAndDay(dispo);
var time_shifts = $scope.compareDiposHoursAndShift(dispoHours, day_shifts);
//then sort the shifts by the dispo's and shift's cities
var time_city_shifts = $scope.compareDispoCityAndShift(time_shifts, dispo);
var time_city_able_shifts = $scope.compareUserAndShift(time_city_shifts, userId);
$scope.showPopover(time_city_able_shifts, event);
}
};
};
}
###Calendar table
<table class="table table-bordered">
<!--days of month-->
<tr class="mainHeader">
<th>coursier/jour</th>
##calendar.days = 30 days of month, for example
<th class="monthDay" ng-repeat=" numDay in [] | range: calendar.days">{{$index+1}} {{calendar.daysNames[$index]}}
</th>
</tr>
<!-- user name and days -->
<tr ng-repeat="user in coursiers">
<!-- <td class="coursierName">nom coursier</td> -->
<td>
{{user.name}}
</td>
<td ng-click="setShift($index+1, monthNum,year, user._id, $event)" ng-repeat="numDay in getTimes(calendar.days) track by $index" ng-class=" isPresent($index, monthNum,year, user._id) == true ?
'present' : isAbsent($index, monthNum,year, user._id) == true ?
'absent' : '' ">
</td>
</tr>
<tr>
<td>Shifts par jour</td>
<td class="shiftsParJourCell" ng-repeat="day in getTimes(calendar.days) track by $index">
<span ng-repeat="shift in shifts">
<span ng-repeat="jour in shift.jours">
{{jour.id ==
getDayOfWeek($parent.$parent.$index, monthNum, year) ? shift.nom : ''}}
</span>
</span>
</td>
</tr>
</table>

After a while, in my case the problem was that I wasn't using directives.
The scopes were having unwanted interactions (still don't know why) but after separing my html in angular directives there are no issues so far.