• 10 dec 2017: forum version update. In case of issues use this topic.
  • 30 nov 2017: pilight moved servers. In case of issues use this topic.
Hello There, Guest! Login Register

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
http library doesn't properly handle big response
If the data returned from an http request is long, the "size" parameter of the callback is correctly showing the actual length of the data received. The data itself however is an empty string.

I have put some debug lines in http.c  The examples below show the result of logging the contents of the "buf" string,  right at the beginning of the read_cb function. When the response data was 110 bytes long , I saw correct results (size=110 and data with a string length of 110):

"HTTP/1.1 200 OK
Content-Length: 110
Connection: close
Date: Mon, 25 Jun 2018 17:25:14 GMT
Server: lighttpd/1.4.35

19:25:14.347 -
Version: 2.0 -
Local IP: -
Remote IP: -
SSID: prive-3 -
RSSI: -60"

But when the response data was 396 bytes long, "size" was indeed 396, but "buf" contained only a part of the header followed by an empty line and no real data at all.

"HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 396
Connection: close


The expected result was an http query string with a number of name=value pairs.

The examples shown above are results from real web "services".

I cannot say from which data size the results will start to go wrong, but obviously it must be somewhere between 110 and 396.

So apparently the software calling the read_cb function in http.c doesn't handle long responses properly.
Can you provide me with a test case? An url that always fails?
In practice absulutely useless, but when calling http://www.google.nl the result is always an emptry string.

I also tried it with https://jsonplaceholder.typicode.com/users which returns a big json result,with Transfer-encoding: chunked. Here the request succeeds, but a segfault occurs in devices_update when I try to update a device with that result. Now that is a different issue that I wil investigate further first. 

So iIt looks like a chunked response is handled correctly. Only big responses that are not chunked are giving the empty response.
The issue is the http response that doesn't communicate a Content-Length. Those were not handled correctly. I will push a fix soon.

... and pushed.
I can confirm that the issue has been fixed. Thanks again.

As I said before, if I try to update a device with a long http response (or in act any very long string) this results in a segfault.
This was an issue that I earlier told about in a diferent thread. When you told that you would rewrite the events parser,  I decided to waited for that.

I will revive that thread now to try and get this solved too.
I have found yet another issue with http, although I cannot say that it is a real bug. But it is anoying, so I woulld like to discuss it with you.

The issue is that for some webservers, pilight's http is closing the "sending" socket too early.

What happens is that pilight sends the request and apparently closes the "request socket" as soon as the header of the request is received. Some webservers are interpretating closing of the "request socket" as "so you dont want to talk with me anymore" and are closing the "response socket" immediately. The effect is that the reponse body is never being received by pilight.

If I do the same request using other clients (such as a browser in Windows, wget or python cgi in Linux) the response is always complete, so including the body. My interpretation is that those clients keep the "request socket" open until the full response has been received, or until the request times out.

Although imho, a webserver should not immediately close the "response socket" when the "request socket" is being closed, that is what some of them do.

As a work around I am using a python cgi script working as a kind of proxy, forwarding the request from pilight to the webserver and returning the response to pilight.

I wonder if there is a  specific reason why the "request socket" is being closed right after the response header has been received and, if not so, if it can be kept open longer.
As always. I need test cases.
It appears that the problem occurs mostly, if not always, with embedded webservers with little or no configuration options.

I can give you access to one of those that demonstrates the issue, but I don't want to publish the url on the forum. If you wish I can send you an email with that url.

If you call the url with a browser, you will see one line of data. If you call it with http get, the response of the request ("data") is an empty string, though "code" is 200 and "size" correctly matches the content length.

Btw. the time out value for http requests is very often too low. You will have to increase that value as I did in order to be able to test this, otherwise you will probably get 408 errors most of the time.
Either mail me with the url or built a unit test in the http test that covers the issue.
LUA HTTP library has been added just now. Good luck with your modules and awaiting your feedback.

Possibly Related Threads...
Thread Author Replies Views Last Post
  ][solved]Segfault when retrieving big chunked http message Niek 21 6,509 11-29-2018, 03:17 PM
Last Post: curlymo
  http code 301 causes segfault Niek 3 3,494 08-14-2018, 06:57 PM
Last Post: curlymo
  no callback on some http requests Niek 20 7,026 07-28-2018, 10:22 AM
Last Post: Niek

Forum Jump:

Browsing: 1 Guest(s)