Since DeepL updated their API which also their website (https://www.deepl.com/translator) uses, an error appears when you're requesting a translation via Java or Python. It says "Too many requests." A year ago the answer here (Using DeepL API to translate text) from EmilioK worked.
Because of their update, the API URL changed to "https://www2.deepl.com/jsonrpc". But the response is {"jsonrpc": "2.0","error":{"code":1042901,"message":"Too many requests."}}. Translating via the website works, so they seem to have implemented a background check. I already tried to debug their JavaScript code, but I don't understand what I'm doing wrong. At last I tried to re-build the requests by analysing the network traffic from the homepage. Didn't work either.
Besides, it seems like others are having the same problem (https://github.com/EmilioK97/pydeepl/issues/12, https://github.com/vsetka/deepl-translator/issues/9).
I used Java 8 with the Apache HttpClient 4.5.8:
private static final String BASE_URL = "https://www2.deepl.com/jsonrpc";
private static AtomicInteger requestId = new AtomicInteger(10000 * (int) Math.round(10000 * Math.random()));
private static final CloseableHttpClient HTTP_CLIENT = HttpClients.createDefault();
public void example() {
HttpResponse httpResponse = HTTP_CLIENT.execute(generateRequest("Hello", "EN", "DE"), getContext());
JSONObject response = new JSONObject(EntityUtils.toString(httpResponse.getEntity()));
System.out.println(response.toString());
}
HttpContext getContext() {
CookieStore cookieStore = new BasicCookieStore();
BasicClientCookie cookie = new BasicClientCookie("LMTBID", "3389513b-e369-4810-a2f3-73b9405e0b0d|403992a1240e1eca6c6e98b428849a3c");
cookie.setDomain(".deepl.com");
cookie.setPath("/");
cookie.setAttribute(ClientCookie.DOMAIN_ATTR, "true");
cookieStore.addCookie(cookie);
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
return localContext;
}
HttpPost generateRequest(String text, String from, String to) {
HttpPost request = new HttpPost(BASE_URL);
StringEntity params = new StringEntity(getJsonRequest(text, from, to).toString(), StandardCharsets.UTF_8);
request.addHeader(HttpHeaders.HOST, "www2.deepl.com");
request.addHeader(HttpHeaders.USER_AGENT, "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0");
request.addHeader(HttpHeaders.ACCEPT, "*/*");
request.addHeader(HttpHeaders.ACCEPT_LANGUAGE, "de,en-US;q=0.7,en;q=0.3");
request.addHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate, br");
request.addHeader(HttpHeaders.REFERER, "https://www.deepl.com/translator");
request.addHeader(HttpHeaders.CONTENT_TYPE, "text/plain");
request.addHeader(HttpHeaders.CONNECTION, "keep-alive");
request.addHeader("X-Requested-With", "XMLHttpRequest");
request.addHeader("Origin", "https://www.deepl.com");
request.addHeader("DNT", "1");
request.addHeader("TE", "Trailers");
request.setEntity(params);
request.setConfig(requestConfig);
return request;
}
private static JSONObject getJsonRequest(String text, String from, String to) {
JSONObject send = new JSONObject();
send.put("id", requestId.incrementAndGet());
send.put("jsonrpc", "2.0");
send.put("method", "LMT_handle_jobs");
JSONObject paramsObject = new JSONObject();
JSONArray jobsArray = new JSONArray();
JSONObject jobsObject = new JSONObject();
jobsObject.put("kind", "default");
jobsObject.put("quality", "fast");
jobsObject.put("raw_en_context_after", new JSONArray());
jobsObject.put("raw_en_context_before", new JSONArray());
jobsObject.put("raw_en_sentence", text);
jobsArray.put(jobsObject);
JSONObject langObject = new JSONObject();
JSONArray userPreferredLangsArray = new JSONArray();
userPreferredLangsArray.put(to);
userPreferredLangsArray.put(from);
langObject.put("source_lang_user_selected", from.getLanguageCode());
langObject.put("target_lang", to);
langObject.put("user_preferred_langs", userPreferredLangsArray);
paramsObject.put("jobs", jobsArray);
paramsObject.put("lang", langObject);
paramsObject.put("priority", -1);
paramsObject.put("timestamp", System.currentTimeMillis());
send.put("params", paramsObject);
return send;
}
The result should look like this, but I don't remember exactly:
{"result":{"source_lang":"EN","target_lang":"DE","translations":[{"beams":[{"postprocessed_sentence":"Hallo","score":0.5,"totalLogProb":0.3,"num_symbols":1},{"postprocessed_sentence":"Guten Tag","score":0.3,"totalLogProb":0.7,"num_symbols":2}]}]}}
I can't tell the length of the text items you are trying to translate, but I had to send in groups of text of max 1000 at a time. Also I have a subscription and not using the free side.
Related
I have two applications: one web client and one java server. The web client sends an HTTP POST request with a JSON body of data, and the server should receive the data and display it on the screen. The problem is that when the server reads the body of the request, there is nothing to read. What is wrong?
Edit: I realized that the problem is on the browser side (since I could send and read an HTTP POST request from some other website), but I still don't know what the problem is. Is this related to the browser running the code? When I use Chrome I get the described problem. When I use Firefox or IE the java server isn't even notified; it doesn't even run the handle method that is supposed to run when it gets an HTTP request.
It worked to read data at the server when I coded the content type as url encoded. I think it was: x-www-form-urlencoded. But I want to send data as JSON.
I use XMLHttpRequest for the web client, as you can see below.The web client:
<!DOCTYPE html>
<html>
<body>
<script>
function handleInput(){
var title = "title";
var reviewer = "reviewer";
const xhr = new XMLHttpRequest();
var searchInfo = {
title:title,
reviewer:reviewer
};
xhr.open('POST', 'http://localhost:8001');
xhr.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');
xhr.send(JSON.stringify(searchInfo));
}
</script>
</body>
</html
The server:
import java.io.IOException;
import java.net.InetSocketAddress;
import com.sun.net.httpserver.HttpContext;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
public class Server {
public static void main(String[] args) {
try {
HttpServer server = HttpServer.create(new InetSocketAddress("localhost", 8001), 0);
HttpHandler handler = new MyHttpHandler();
HttpContext context = server.createContext("/");
context.setHandler(handler);
server.start();
System.out.println("Server started on port 8001");
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
The servers HTTP handler:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
public class MyHttpHandler implements HttpHandler {
#Override
public void handle(HttpExchange httpExchange) throws IOException {
InputStream is = httpExchange.getRequestBody();
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8));
StringBuilder content = new StringBuilder();
String line;
while ((line = br.readLine()) != null) { br.readLine is always null!!!
content.append(line);
content.append("\n");
}
System.out.println("Content: " + content.toString());
}
}
As your server (java side, live in localhost:8001) and client (xhttp side, live in somewhere else) are separated. The request is considered as cross origin requests (CORs).
There are 2 types of cross origin requests - Safe and unsafe:
Only the below Content-Type are considered as SAFE in cross site request:
application/x-www-form-urlencoded
multipart/form-data
text/plain
So, application/json; is considered UNSAFE.
For safe request, there is not much restriction on server so you can get the response back while using application/x-www-form-urlencoded
client <=> POST request <=> Server
For unsafe request, there is a preflight request (a special OPTIONS action) to ask the server
if client origin is allowed and
what can the client do
before making the real request, in your case the POST request.
client <=> pre-flight (OPTIONS) request <=> Server
client <=> POST request <=> Server
So the problem is there is no handling of preflight request and the request cannot be processed by server.
To deal with it, you need to change your handler as follow:
import java.io.*;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
public class MyHttpHandler implements HttpHandler {
#Override
public void handle(HttpExchange httpExchange) throws IOException {
Headers headers = httpExchange.getResponseHeaders();
System.out.println(httpExchange.getRequestMethod());
headers.add("Access-Control-Allow-Origin", "*"); // * means allow all origin, in production, it should be the origin you trust e.g. http://client.abc.com
if (httpExchange.getRequestMethod().equalsIgnoreCase("OPTIONS")) {
headers.add("Access-Control-Allow-Headers", "Content-Type"); // allow clients to pass in content-type
headers.add("Access-Control-Allow-Methods", "GET,POST,PUT,DELETE");
httpExchange.sendResponseHeaders(204, -1);
return;
}
InputStream is = httpExchange.getRequestBody();
BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8));
StringBuilder content = new StringBuilder();
String line;
while ((line = br.readLine()) != null) { // br.readLine is always null!!!
content.append(line);
content.append("\n");
}
System.out.println("Content: " + content.toString());
// added response as well to complete the request
String response = "Good" ;
httpExchange.sendResponseHeaders(200, response.length());
OutputStream os = httpExchange.getResponseBody();
os.write(response.getBytes(StandardCharsets.UTF_8));
os.close();
httpExchange.close();
}
}
HTML
<!DOCTYPE html>
<html>
<body>
<button onclick="handleInput()">Click</button>
<script>
function handleInput() {
var title = "title";
var reviewer = "reviewer";
const xhr = new XMLHttpRequest();
xhr.onload = function() {
console.log('done')
}
var searchInfo = {
title: title,
reviewer: reviewer,
};
xhr.open("POST", "http://localhost:8001");
xhr.setRequestHeader("Content-Type", "application/json; charset=UTF-8");
xhr.send(JSON.stringify(searchInfo));
}
</script>
</body>
</html>
To further understanding CORs:
https://javascript.info/fetch-crossorigin
For your html
<form name="searchForm" id="searchForm" onSubmit="handleInput(event)">
...
</form>
<script>
function handleInput(e){
e.preventDefault()
...
}
</script>
Let me explain a bit about my system. I have a third party software that communicates with a library/API (C#) of mine. This library communicates with my FrontEnd (JS) and, of course, this Front End communicates with my BackEnd (Java).
My problem is that I need my library/API to compress an XML that comes from third-party software, I pass it to the FrontEnd and it passes it to the BackEnd. But because this XML is too large, I need the library/API to compress it and the BackEnd unzip it. When the buffer reaches my BackEnd the decompressor jumps a "Not in GZip format" exception. I get the impression that the problem is in the Parse, but I can not find the solution. I would appreciate any help.
Library/API Code (C#):
public string operationXML(String xmlIN){
try
{
[...]
byte[] binaryData = Encoding.UTF8.GetBytes(xmlUtil.ObjectSerializer(objOUT, null));
//xmlUtil.ObjectSerializer returns a XML in String format
return Encoding.UTF8.GetString(CompressGZip(bynaryData));
}
catch()[...]
}
public byte[] CompressGZip(byte[] raw){
using (MemoryStream memory = new MemoryStream()){
using (GZipStream gzip = new GZipStream(memory, CompressionMode.Compress, true)){
gzip.Write(raw, 0, raw.Length);
}
return memory.ToArray();
}
}
FrontEnd (JS) Calling the library/API and Passing to BackEnd:
var response = petition("operationXML", responseObject.responseText);
conn.request({
url: gContextPath + "/MostradorGrid.json",
method: 'POST',
params: {
"action": "processXML",
"response": response,
"idUser": gIdUser
},
success: function (responseObject) {[...]
BackEnd Controller (Java):
case processXML:{
if (ParamMap.containsKey("idUser") && ParamMap.containsKey("response")) {
int idUser= NumberUtils.toInt(ParamMap.get("idUser")[0]);
byte[] xmlgzip= ParamMap.get("response")[0].getBytes("UTF8");
grid271 grid271 = (grid271) getWebApplicationContext().getBean("grid271");
response.getWriter().println(grid271.xmlprocess(xmlgzip, idUser));
}
}
BackEnd grid271.xmlprocess
[...]
byte[] temp = compressor.dGzip(xmlgzip);
String data= compressor.bytesToString( temp , "UTF_8", true);
[...]
BackEnd compressor
public byte[] dGzip(byte[] contentBytes) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
try {
IOUtils.copy(new GZIPInputStream(new ByteArrayInputStream(contentBytes)), out);
} catch (IOException e) {
throw new RuntimeException(e);
}
return out.toByteArray();
}
In IOUtils.copy(new GZIPInputStream(new ByteArrayInputStream(contentBytes)), out); is when the Exception jumps: (java.Util.zip.zipException) java.util.zip.ZipException: Not in GZIP format
Any ideas?
Thanks in advance
I'm trying to write an old native Android app with Ionic and I need help for the http request. I'm newbie in AngularJS (js too).
My Android code has a function like:
String address = "http://www.example.com";
DefaultHttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(address);
List<NameValuePair> pairs = new ArrayList<NameValuePair>();
pairs.add(new BasicNameValuePair("param", sParam));
try {
post.setEntity(new UrlEncodedFormEntity(pairs));
HttpResponse response = client.execute(post);
BufferedReader rd = new BufferedReader(new InputStreamReader(responsegetEntity().getContent()));
StringBuilder sBuilder = new StringBuilder();
String sLine = "";
while ((sLine = rd.readLine()) != null) {
sBuilder.append(sLine).append("\n");
}
String sContent = sBuilder.toString();
(...parsing sContent...)
} catch (Exception e) {
//something
}
and if there are more then one page I call a function like
String address = "http://www.example.com/result.do?page="+ iPage;
HttpPost post = new HttpPost(address);
List<NameValuePair> pairs = new ArrayList<NameValuePair>();
pairs.add(new BasicNameValuePair("param", sParam));
Cookie ck = client.getCookieStore().getCookies().get(0);
pairs.add(new BasicNameValuePair("param_ck", ck.getValue()));
try {
post.setEntity(new UrlEncodedFormEntity(pairs));
HttpResponse response = client.execute(post);
(..parsing..)
}
So, I read the html content of a webpage (I'm not the owner) and I do somethings with that.
I tried $http.post but I'm not sure if it's the same
.factory('Service', function($q,$http) {
return {
getResult: function(param) {
var q = $q.defer();
var address = "http://www.example.com";
var sParam = "param";
$http({
url: address,
method: "POST",
data: {
'param' : sParam
}
})
.then(function(response) {
(...)
q.resolve(position);
},
function(error) {
(...)
q.reject(error);
});
return q.promise;
}
};
});
PS: I get the
No 'Access-Control-Allow-Origin' header is present on the requested resource.
with that.
Can you help me?
I am not entirely sure why you don't get a similar error with your Android code, or even if you were supposed to, as I am not familiar with native Android itself. But the reason that you get this with Angular in Ionic is that the server requires to implement CORS to get rid of that.
From MDN:
A resource makes a cross-origin HTTP request when it requests a resource from a different domain than the one which the first resource itself serves. For example, an HTML page served from http://domain-a.com makes an src request for http://domain-b.com/image.jpg. Many pages on the web today load resources like CSS stylesheets, images and scripts from separate domains.
What ways are there for a website to detect automated connections made through C# Httpwebrequest?
Such as c#'s default user-agent? Operating System? or what..
have this problem with any other language, just C#?
I'm being blocked from accessing a certain website using Httpwebrequest, I don't
Also it's definitely not my IP address & nor are there any faults in my code as I've tested connections to other websites which work just fine.. Also I stated above I can make connections to the website using C++, C, Vb.net, Java, Python & so on, there is also no difference in header information either.
EDIT:
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create ("http://services.runescape.com/m=hiscore_oldschool/overall.ws");
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "user1=Zezima&submit=Search";
byte[] byteArray = Encoding.UTF8.GetBytes (postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream ();
// Write the data to the request stream.
dataStream.Write (byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close ();
// Get the response.
WebResponse response = request.GetResponse ();
// Display the status.
Console.WriteLine (((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream ();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader (dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd ();
// Display the content.
Console.WriteLine (responseFromServer);
// Clean up the streams.
reader.Close ();
dataStream.Close ();
response.Close ();
private const string Url = "http://services.runescape.com/m=hiscore_oldschool/overall.ws";
private static HttpWebRequest BuildWebRequest()
{
var request = WebRequest.Create(Url) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.Timeout = 40000;
request.ServicePoint.Expect100Continue = true;
string body = "user1=Zezima&submit=Search";
byte[] bytes = Encoding.Default.GetBytes(body);
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
return request;
}
static void Main(string[] args)
{
try
{
HttpWebRequest request = BuildWebRequest();
var response = request.GetResponse() as HttpWebResponse;
var responseContent = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.Write("Success - " + response.StatusCode);
}
catch (Exception e)
{
Console.Write(e);
}
}
I can take the response from the website. It is not empty.
I have this ASP.Net Web Application and in one of my Pages I'm using JQuery JQGrid, I want to be able to export the JQGrid to Excel Sheet, so using JavaScript I Collected the Values in an Array and then called a WebService to fill the values in a DataGrid, I successfully done all that, except for the exporting in the WebService no file download appears, and this is my WebService code:
[WebMethod]
public void GetData(object[] Values)
{
DT_ToFill.Columns.Add("CaseID");
DT_ToFill.Columns.Add("SR");
foreach (object Value in Values)
{
Dictionary<string, object> DictVals = new Dictionary<string, object>();
DictVals = (Dictionary<string, object>)Value;
string CaseID = DictVals["CaseID"].ToString();
string SR = DictVals["SR"].ToString();
object[] Obj = new object[2] { CaseID, SR };
DT_ToFill.Rows.Add(Obj);
}
ExportToExcel(DT_ToFill, Context.Response);
}
[WebMethod]
public string HelloWorld()
{
return "Hello World";
}
public void ExportToExcel(DataTable dt, HttpResponse Response)
{
GridView GridView1 = new GridView();
GridView1.DataSource = dt;
Response.Buffer = true;
Response.AddHeader("Content-Disposition", "inline;filename=Schedule_ExcelSheet.xls");
Response.Charset = "";
Response.ContentType = "application/ms-excel";
Response.Charset = "UTF-8";
Response.ContentEncoding = Encoding.UTF8;
StringWriter sw = new StringWriter();
HtmlTextWriter ht = new HtmlTextWriter(sw);
GridView1.AllowPaging = false;
GridView1.DataBind();
GridView1.RenderControl(ht);
Response.Output.Write(sw.ToString());
Response.Flush();
Response.End();
}
public void VerifyRenderingInServerForm(Control control)
{
}
Help Please...
Thanks
Sounds like you are sending the excel file back as the response to the ajax request? If so, I don't think you can make a file download work through ajax.
Maybe send the data via ajax like you are doing now, but then do either a new window or an iframe (append it to document.body) and set the src to be a url that returns the excel file. That should trigger the file download. This is what I usually do.