We Keep Coding

Use Javascript to Scrape table by selecting some other elements and clicking on button

I'm working with financial data and i want to scrape data from this site web using javascript and add the sript.js to my index.html file (https://www.sikafinance.com/marches/historiques?s=BRVMAG).
I want to scrape the data of a table from above site that takes four arguments
•Ticker who is BRVMAG in the minimal example
•dlPeriod
•datefrom
•dateto
And finally click on the button
btcChange ="OK".
After trying the code below with R I get the table but how can a achieve the same script with fetch?
I would like to be able to get the other tables when I change the start and end date using now javascript in Visual Studio Code.
Since yersterday i'm looking for a reponse searching every here but now result.
Does anyone have any idea how to recover the whole table and return table in of html?
The bottom images show what I noticed when i inspected their site.
I think either the whole table is available and it does a filter for the dates(gap between dates must not exceed 3 months)
library(httr)
library(rvest)
first_date<-as.Date("2022-02-01")
end_date <- as.Date("2022-03-29")
query_params <- list(dlPeriod = "Journalière",
datefrom = first_date,
dateto = end_date,
btnChange = "OK")
parameter_response <- GET("https://www.sikafinance.com/marches/historiques?s=BRVMAG", query_params)
parameter_response1<- httr::content(parameter_response, as = "text", encoding = "UTF-8")
parameter_response2 <- read_html(parameter_response1)%>%
html_node('#tblhistos')%>%
html_table()
parameter_response2
# Date Clôture `Plus bas` `Plus haut` Ouverture `Volume Titres` `Volume FCFA` `Variation %`
# <chr> <chr> <chr> <chr> <chr> <chr> <int> <chr>
# 1 29/04/2022 312,09 312,09 312,09 312,09 - 0 2,53%
# 2 28/04/2022 304,38 304,38 304,38 304,38 - 0 0,00%
# 3 27/04/2022 304,38 304,38 304,38 304,38 - 0 2,69%
# 4 26/04/2022 296,42 296,42 296,42 296,42 - 0 0,81%
# 5 25/04/2022 294,05 294,05 294,05 294,05 - 0 1,34%
# 6 22/04/2022 290,17 290,17 290,17 290,17 - 0 0,36%

How to keep a form from submitting when creating a Django model object with many-to-many field selection that is doesn't exist in the related model?

I'm using Django to create a web app. I have multiple Django models with relationships between each other. One of my models is called Type and another is called TestEquipment. The Type model has a many-to-many relationship with TestEquipment.
To allow the user to create a new Type, I have an html form and to select which TestEquipment will be associated with that Type I have am using searchable dropdown with "chips" (javascript) which loads all TestEquipment and allows you to search and select multiple objects to add. When the user submits the form, the selected TestEquipment is added.
Everything works great other than when the user enters text and presses enter, instead of selecting from the dropdown, a chip is added with that text. When the form submits it tries to add a TestEquipment that doesn't exists.
I would like to find a way to either not allow adding an objects that doesn't exist or throw an alert "must select from existing test equipment"; somehow I have to ensure that the form does not submit to my constructor and create the new Type if text is added to the field.
I've tried to find an answer to this and absolutely have had no luck. Any help is REALLY appreciated!
Django Models code:
class TestEquipment(models.Model):
name = models.CharField(max_length=64, unique=True)
notes = models.TextField(null=True, blank=True)
def __str__(self):
return f"{self.name}"
class Type(models.Model):
name = models.CharField(max_length=64, unique=True)
type_folder = models.URLField(null = True, blank = True)
type_test_equipment = models.ManyToManyField(TestEquipment, blank=True, related_name="type_test_equipment")
type_notes = models.TextField(null=True, blank=True)
test_sheet = models.URLField(null=True, blank=True)
type_test_guide = models.URLField(max_length=300, null=True, blank=True)
objects = TypeManager()
def __str__(self):
return f"{self.name}"
Views.py code:
def create_type_view(request):
if not request.user.is_authenticated:
return render(request, "jobs/login.html", {"message": None})
test_equipments = TestEquipment.objects.all()
equipment_types = Type.objects.all()
#pass in existing types of equipment
context= {
"equipment_types": equipment_types,
"test_equipments": test_equipments
}
if request.user.is_authenticated:
return render(request, "jobs/create_type.html", context)
create_type.html code (pulls in materialize styling and javascript):
<div>
<form id="type_form" class="col s12" action="{% url 'create_type' %}" method="post">
{% csrf_token %}
<div class="row">
<div class="col s12">
<h6 style="color:#808080">Select Type-Specific Test Equipment</h6>
<div class="row">
<div class="input-field col s12">
<div id="test_equipment-chips" class="chips chips-autocomplete">
</div>
<input id="test_equipment" name="test_equipment" style="visibility: hidden;" value="">
</div>
</div>
</div>
</div>
<div class="row">
<input id="submit-btn" type="submit" value="Confirm Add" onclick="onSubmit();" />
</div>
</form>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
<script>
function onSubmit() {
var chipInstance = M.Chips.getInstance($("#test_equipment-chips"));
var value = Array.from(chipInstance.chipsData.map(x=>x.tag)).join(",");
$("#test_equipment").val(value);
}
$('#test_equipment-chips').chips({
autocompleteOptions: {
data: {
{% for test_equipment in test_equipments %}
'{{test_equipment}}': null,
{% endfor %}
},
limit: Infinity,
minLength: 0,
}
});
</script>
code for constructor:
def create_type(request):
#extract type name from form
type_name = request.POST["type_name"]
#create new type object
new_type=Type.objects.create_type(type_name)
#fill in all properties that were submitted in the form
new_type.type_folder = request.POST["type_folder"]
new_type.test_sheet = request.POST["test_sheet"]
new_type.type_test_guide = request.POST["type_test_guide"]
new_type.type_notes = request.POST["type_notes"]
equipment_list=request.POST["test_equipment"].split(",")
for equipment_name in equipment_list:
equipment=TestEquipment.objects.get(name=equipment_name)
new_type.type_test_equipment.add(equipment.pk)
#save to database
new_type.save()
return HttpResponseRedirect(reverse("jobs"))

So the most important thing to do there is to perform a check in your python because you shouldn't perform a get request to the database like that unless you know it can't fail. As you're seeing, it just ends up with 500 errors when objects don't exist.
So job 1 would be to do this in your view;
for equipment_name in equipment_list:
try:
equipment=TestEquipment.objects.get(name=equipment_name)
except TestEquipment.DoesNotExist:
# Log the error somehow if you like?
print(f"Equipment not found with the name {equipment_name}")
else:
# This only executes when the exception wasn't raised
new_type.type_test_equipment.add(equipment.pk)
Using the above code, it'd just ignore invalid input, which suits a minimum viable option. If you wanted to provide an error in the event of invalid input however you should run through all values in equipment_list to ensure they exist before adding them to the relationship. That way you could render the form again with the errors attached.
I've had a look at the demo/docs for chips and it doesn't look like it can help prevent invalid inputs, but you could do some validation in javascript yourself to prevent submission.
You should also have a look at using django's forms because your view is directly accessing post data. By using forms you can handle validation logic outside of your view & there's a bunch more benefits from them. If you're yet to find out about forms, give this page a read; https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Forms

transferring value from a google map to a rchart in shiny application

I have a shiny app in which I plot a map with plotGoogleMaps and a rChart, with values related
to some markers in the map, rendered via rPlot.
The app user can click on a marker in the map to show a tooltip.
I'd like that when he clicks on the marker, the pertinent value in the chart would be highlighted.
Anybody knows how to perform this task?
Thank you.

#jdharrison
The R-code can be like this:
suppressPackageStartupMessages(library(googleVis))
## Hurricane Andrew (1992) storm track with Google Maps
AndrewMap <- gvisMap(Andrew, "LatLong", "Tip", options = list(showTip = TRUE,
showLine = TRUE, enableScrollWheel = TRUE, mapType = "hybrid", useMapTypeControl = TRUE))
print(AndrewMap, "chart")
The example was taken from here. You can download the package here. In this example, library googleVis is used, but I think that the answer can be similar.
Shiny example
server.R
# Author: Denis Petrov
# Date: 2014-07-04
# Example is based on http://rpubs.com/gallery/googleVis
library (shiny)
suppressPackageStartupMessages (library (googleVis))
# Define server logic required to summarize and view the selected dataset
shinyServer ( function (input, output, session)
{
GetDataAndrew <- function ()
{
# Hurricane Andrew (1992) storm track with Google Maps
AndrewMap <- gvisMap (Andrew, "LatLong", "Tip",
options = list(showTip = TRUE,
showLine = TRUE,
enableScrollWheel = TRUE,
mapType = "hybrid",
useMapTypeControl = TRUE))
return (AndrewMap)
}
output$viewchart <- renderGvis({
GetDataAndrew ()
})
output$info <- renderPrint ({
cat ('Hurricane\n')
cat ('Pressure=937\n')
cat ('Speed=120')
})
}
)
ui.R
# Author: Denis Petrov
# Date: 2014-07-04
# Define UI for dataset viewer application
shinyUI(pageWithSidebar(
# Application title
headerPanel('Hurricane Andrew (1992)'),
# Sidebar with controls to provide a caption, select a dataset, and
# specify the number of observations to view. Note that changes made
# to the caption in the textInput control are updated in the output
# area immediately as you type
sidebarPanel(
p ('Hurricane Andrew (1992) storm track with Google Maps'),
p ('I would like the following output be changed based on
selected pin.'),
verbatimTextOutput ('info')
),
# Show the caption, a summary of the dataset and an HTML table with
# the requested number of observations
mainPanel(
htmlOutput ('viewchart')
)
))

Web text input save into database without form- Django Javascript

I have an sqlite3 database setup with django 1.6 where I want web users to be able to enter text and see it in 3 live tables (high, low and latest) all on the same page. The one page ideally should have text entry, voting, display of the three tables updated when new entries go in or are voted on, and a search if possible if all that gets sorted (personal project, not commercial). I am also on win7 64 if that matters...
Currently I have: a working database the three tables displaying in a web page without update and some web text input via js (and failing to save to the database in Django).
I shied away from forms at first, as they seem to want separate html pages for input. Asking an experienced django coder, he helped me out with some javascript for text entry on the page. He said I didn't need to do it via forms and POST, GET, as the text was just going in with the initial score of 0 and the current datetime.
My issue now is that I cannot get the entered text to save into the database without error.
Since I need to program this in 2-3 weeks and am new to django (and oblivious to javascript, though I've done some Processing with PHP), my questions are;
Am I missing something obvious with the text input save to database?
-and-
Is there a way to have all this using forms and GET, POST in one page
so I can avoid a lot of javascript (unless it is truly easier)?
I am going to start to try to build this with Forms at the moment, but hope for a bit of guidance on best practice from wiser heads.
Here's the code so far:
urls.py
from django.conf.urls import patterns, include, url
from django.contrib import admin
admin.autodiscover()
urlpatterns = patterns('',
url(r'^admin/', include(admin.site.urls)),
url(r'^i/$', 'entries.views.index'),
url(r'^add/(.*)$', 'entries.views.add'),
)
Models.py
from django.db import models
import datetime
from django.utils import timezone
class Entry(models.Model):
text = models.CharField(max_length=15)
score = models.IntegerField(default=0)
pub_date = models.DateTimeField('date published')
def __unicode__(self):
return self.text
def was_published_recently(self):
return self.pub_date >= timezone.now() - datetime.timedelta(days=1)
was_published_recently.admin_order_field = 'pub_date'
was_published_recently.boolean = True
was_published_recently.short_description = 'Published recently?'
index.html
<html>
<head>
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
</head>
<body>
<ul>
{% for entry in latest_entry_list %}
<li>{{ entry.text }} &nbsp{{ entry.score }}</li>
{% endfor %}
</ul>
<ul>
{% for entry in high_entry_list %}
<li>{{ entry.text }} &nbsp{{ entry.score }}</li>
{% endfor %}
</ul>
<ul>
{% for entry in low_entry_list %}
<li>{{ entry.text }} &nbsp{{ entry.score }}</li>
{% endfor %}
</ul>
<style type="text/css" media="screen">
div h2 span { color: #ff0000; }
div span { color: #00ff00; }
#box { width: 400px; height: 400px; }
#h { color: #ff0000; }
</style>
<h3 id="h">title</h3>
<p>message: {{ text }}</p>
<input type="text" name="word" value="" id="input"/>
<script type="text/javascript" src="{{STATIC_URL}}post.js"></script>
</body>
post.js
console.log("hi from js");
$(document).ready(function() {
$("#input").bind("keypress", function(e) {
//enter key pressed
if (e.keyCode == 13) {
var args = {};
var text = $("#input").val();
$.get("/add/" + text, args).done(function(data) {
console.log("message: " + data);
});
}
});
});
views.py
from django.shortcuts import render
from django.http import HttpResponse
from entries.models import Entry
from django.db import models
import datetime
from django.utils import timezone
def index(request):
context = {
'latest_entry_list': Entry.objects.order_by('-pub_date')[:10],
'high_entry_list': Entry.objects.order_by('-score')[:10],
'low_entry_list': Entry.objects.order_by('score')[:10],
}
return render(request, 'entries/index.html', context);
def add(request, thingtoadd):
#created_date = models.DateTimeField('date published', default=datetime.now)
#created_score = '0'
#created_text = 'test'
#e = Entry(text=created_text, score=created_score,pub_date=created_date)
#e.save()
return HttpResponse('done')
I am unsure of defining the fields for populating the Entry....does the above look right?
I can uncomment the e=Entry(etc...) without error,
but when I uncomment the e.save(), the error is:
GET http://127.0.0.1:8000/add/a 500 (INTERNAL SERVER ERROR) jquery.min.js:4
send jquery.min.js:4
n.extend.ajax jquery.min.js:4
n.(anonymous function) jquery.min.js:4
(anonymous function) post.js:15
n.event.dispatch jquery.min.js:3
r.handle
I will be getting on with trying to do this in forms, but wonder if there is some good advice as to if that is possible - I would ideally like to avoid js extras as I am very unfamiliar with it and it would be another level of unknowns at this point. Any input greatly appreciated...

Your mistake in view function add:
created_date = models.DateTimeField('date published', default=datetime.now)
It must be value assign:
created_date = datetime.now()
Not field definition.
In advance you could specify auto_now_add=True in your model: https://docs.djangoproject.com/en/dev/ref/models/fields/#datefield
In that case field will be filled automatically.
Additional:
It is error in urls.py
You should do some fixes:
urls.py:
url(r'^add/$', 'entries.views.add'),
post.js
$("#input").bind("keypress", function(e) {
//enter key pressed
if (e.keyCode == 13) {
var text = $("#input").val();
var args = {'text': text};
$.get("/add/", args).done(function(data) {
console.log("message: " + data);
});
}
});
views.py
def add(request):
created_date = default=datetime.now()
created_score = '0'
created_text = request.GET.get('text')
e = Entry(text=created_text, score=created_score,pub_date=created_date)
e.save()
return HttpResponse('done')
Update - Solution
The solution in addition to the changes below was to add 'from datetime import datetime' in views....

Prevent datepicker values from being stripped from params

I'm using the bootstrap-datepicker.js (http://www.eyecon.ro/bootstrap-datepicker/) to set the date range for a table in a rails view.
calls.html.erb
<form class="form-horizontal daterange">
<input type="text" data-date-format="yyyymmdd" id="start-date" name="start">
<span class="add-on">-</span>
<input type="text" data-date-format="yyyymmdd" id="end-date" name="end">
<button type="submit" class="btn submit-date">Submit</button>
</form>
<script type="text/javascript">
$(document).ready(function() {
$('#start-date').datepicker({
format: 'yyyy-mm-dd'
});
$('#end-date').datepicker({
format: 'yyyy-mm-dd'
});
});
</script>
It passes params[:start] and params[:end] into the URL that I use to set the date range for Twilio API calls.
Ex. /twilio/calls/?start=2013-03-01&end=2014-01-06
Ever since I created the following routes to paginate the api calls:
routes.rb
match 'twilio/calls' => 'twilio#calls', :as => :twilio_page
match 'twilio/calls/:page_id' => 'twilio#calls', :as => :twilio_page
Whenever I advance to the next page of results, it strips out the date range values.
How do I preserve the :start/:end parameters?
twilio_controller.rb Update - added start/end data
if params[:start].nil?
#start_date = DateTime.parse((Date.today - 7).to_s).strftime("%Y-%m-%d")
#end_date = DateTime.parse((Date.today - 1).to_s).strftime("%Y-%m-%d")
else
#start_date = params[:start]
#end_date = params[:end]
end
#user = current_user
#account_sid = #user.twilio_account_sid
#auth_token = #user.twilio_auth_token
#page_size = 5
#page = params[:page_id] || 0
#sub_account_client = Twilio::REST::Client.new(#account_sid, #auth_token)
#subaccount = #sub_account_client.account
#recordings = #subaccount.recordings
#recordingslist = #recordings.list({:page_size => #page_size, :page => #page, :"date_created<" => #end_date, :"date_created>" => #start_date})

According to the canonical Rails guides:
The params will also include any parameters from the query string.
Therefore, you want to eliminate the first route, which doesn't allow for any parameters:
# config/routes.rb
match 'twilio/calls/:page_id' => 'twilio#calls', :as => :twilio_page
To illustrate, suppose you are trying to access /twilio/calls/1?start=2013-03-01&end=2014-01-06. The query string is actually parsed, such that you have access to not only the page_id parameter, but also start and end.
If you want to maintain a default route, e.g., twilio/calls, you can declare a route redirect as follows:
# config/routes.rb
get '/twilio/calls', to: redirect('/twilio/calls/1')
This will automatically reroute all requests for /twilio/calls to /twilio/calls/1.
UPDATE:
To clarify, URL query parameters are parameterized and fed to the corresponding controller action as members of the params hash. Therefore, given the path /twilio/calls/1?start=2013-03-01&end=2014-01-06, you'll have access to params[:page_id], params[:start], and params[:end]. You'll need to pass the latter two to your API call in your action:
#recordingslist = #recordings.list({:page_size => #page_size,
:page => #page,
:"date_created<" => params[:start],
:"date_created>" => params[:end]
})
UPDATE:
In order to persist the query parameters across page views, you can concatenate the argument passed to twilio_page_path with the query string:
twilio_page_path((#page.to_i + 1).to_s + "?start=" + params[:start] + "&end=" + params[:end]) %>