Real Software Forums

The forum for Real Studio and other Real Software products.
[ REAL Software Website | Board Index ]
It is currently Sat Oct 20, 2018 11:10 pm
xojo

All times are UTC - 5 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: 2012R1 HTTPSocket Bug
PostPosted: Wed Sep 12, 2012 12:11 pm 
Offline
User avatar

Joined: Fri Oct 28, 2005 7:05 am
Posts: 565
Location: Emsworth, UK
As mentioned on this thread, viewtopic.php?f=7&t=45282
there is a bug in the HTTPSocket code shipped with 2012R1. I said I would write up my own tests, so here you go.


Warning: Check the thread title. If you happen to be reading this in 3 months or more, looking for a fix for your HTTPSocket problem, do not assume this is it. Hopefully RS will fix this issue quickly.

For testing purposes, I used 2011R4.3 Enterprise (R4.3) and 2012R1 Enterprise (R1) on Windows 7 x 64. As the original fault was in the Mac forum, I think we can safely assume the issue is across supported platforms. Network sniffing was conducted with Wireshark 1.8 and the URI used to test was http://173.8.248.213:8086/weather.json

Using HTTPSocket.Get(URI), the .PageReceived event fires in R4.3 but not in R1.

The first difference between R4.3 and R1 is the request sent to the server
R4.3 sends "GET /weather.json HTTP/1.0"
R1 sends "GET /weather.json HTTP/1.1"

At the HTTP layer this is what the server should respond with. Hopefully you can follow my colorisation.

#0 Transfer-Encoding:chunked[CR][LF]
#1 <other headers>
#2 Accept-Ranges:bytes[CR][LF]
#3 [CR][LF]
#4 27f8
#5 { "station":
#6 <data continues>
#7 }[CR][LF]
#8 0[CR][LF]
#9 [CR][LF]

Note the "27f8" and "0" characters I have emphasised in bold. These are the boundary delimiters for the data 'chunk' and are sent as ASCII. 0x27fe is the length of the chunk and 0[CR][LF] marks the end of the chunk. Everything between these boundaries is the data.

When .PageReceived fires in R4.3 the string passed as the "content" parameter, includes the chunk boundary marks.

Comparative testing

First of all, both R4.3 and R1 receive the headers just fine.

When calling .Get(URI) in either R4.3 or R1, the .ReceiveProgress event fires. The first observed difference is with the bytesReceived value passed to .ReceiveProgress
R4.3, bytesReceived increments.
R1, bytesReceived is always 0.

As HTTPSocket is subclassed from TCPSocket, I assumed .ReceiveProgress was probably just .DataAvailable being re-raised after RS have done their processing. To see what might be occurring , I populated the event and had a look at HTTPSocket.Lookahead.

Observed differences in .ProgressReceived behaviour;
newData parameter
R4.3, contains the new data
R1, always contains an empty string

HTTPSocket.Lookahead.LenB
R4.3, is always 0.
R1, increments on subsequent calls.

R1, the .ProgressReceived event appears to be raised once too often, when there has been no changed to .Lookahead.LenB

So, what happens when "call me.Read(me.Lookahead.LenB)" is inserted in the R1 .ReceiveProgress event handler?
The bytesReceived parameter starts incrementing, the .PageReceived event fires with the "content" parameter populated. However, unlike R4.3 the "content" parameter does not contains the chunk boundary markers.

It appears RS are failing to read the input buffer during DataAvailable, in some circumstances. It's easy done. Personally I would recommend waiting on RS to fix the issue, rather than shipping R1 code using the HTTPSocket or relying on my test results for production use.

I did some further fiddling, using .SendRequest("Get", URI) which seemed to indicate the un-necessary call to .ReceiveProgress might be wiping out the "content" buffer and leaving 7 bytes, including the 0 boundary mark, in the input buffer. Those test were not conducted so methodically, are inconclusive and I won't be writing them up.

Warning: On no account should you fiddle with .Lookahead and .Read in production code using the HTTPSocket. It can cause strange and unpredictable results. On second thoughts please do, as it might generate enough calls to RS to convince them to provide some way to hide events and methods that are overriden in subclasses (TFIC).

HTH

_________________
Yes it's me in the avatar


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Sat Sep 15, 2012 10:21 am 
Offline

Joined: Wed Mar 01, 2006 6:48 pm
Posts: 199
Thank you for documenting this. I was the original poster in the thread you mention. Do I still need to log a feedback bug report or have you done so and I just missed it?

_________________
Colorado, USA
Primary: BioMedical/Custom Applications - www.sysdyn.com
Secondary: Handheld/Custom Applications - www.mrhswco.com


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Sat Sep 15, 2012 10:30 am 
Offline
User avatar

Joined: Sat Nov 11, 2006 2:43 pm
Posts: 1221
Location: This poster has left the forums
I believe a number of different feedback cases are open concerning this.

Some cases were considered resolved, but I subsequently provided some sample code to Real, who were able to replicate the problem. A question remains as to the nature of the bug and whether it is some of the servers interpretation of the HTTP/1.0 spec, and their expectation of the ordering of the headers. That question aside, I'm told a fix is in the works.

_________________
%Invalidforumsignatureexception% user signature not found


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Sat Sep 15, 2012 11:03 am 
Offline

Joined: Wed Mar 01, 2006 6:48 pm
Posts: 199
Awesome! Thanks

_________________
Colorado, USA
Primary: BioMedical/Custom Applications - www.sysdyn.com
Secondary: Handheld/Custom Applications - www.mrhswco.com


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Mon Sep 24, 2012 3:06 am 
Offline
User avatar

Joined: Fri Oct 28, 2005 7:05 am
Posts: 565
Location: Emsworth, UK
pony wrote:
I believe a number of different feedback cases are open concerning this.
I hope RS are reading the forums too.

Quote:
A question remains as to the nature of the bug and whether it is some of the servers interpretation of the HTTP/1.0 spec, and their expectation of the ordering of the headers.


There should be no question. HTTP headers are not ordinal - A dictionary rather than an array. HTTP headers can be sent or received in any order and there should be no expectation or assumption of order, made by the client or the server. That's been a feature of the HTTP spec since at least HTTP/0.94.

Quote:
That question aside, I'm told a fix is in the works.
Shame they did not (think) to include a method for declaring HTTP/1.0, reverting to the functionality of prior RS versions, as it is still valid.

Let's hope they get a fix out soon.

_________________
Yes it's me in the avatar


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Mon Sep 24, 2012 7:53 am 
Offline
User avatar

Joined: Sat Nov 11, 2006 2:43 pm
Posts: 1221
Location: This poster has left the forums
msssltd wrote:
There should be no question. HTTP headers are not ordinal - A dictionary rather than an array. HTTP headers can be sent or received in any order and there should be no expectation or assumption of order, made by the client or the server. That's been a feature of the HTTP spec since at least HTTP/0.94.

It was claimed one or more of the servers in the example I sent was requiring the headers be in a certain order. I could neither validate or refute that claim. The servers are USPS Federal Government Servers. It was opined that one or more of them (they are load balanced) was causing the problem by not answering if the headers were not in a certain order. I was not in any way convinced.


msssltd wrote:
Let's hope they get a fix out soon.

Marked as fixed. Still not working. I've been asked for another example project, but the first one I sent still exhibits the behavior. Not sure what to do. It appeared they decided the bug was quashed without any request for feedback from me. Very frustrating.

_________________
%Invalidforumsignatureexception% user signature not found


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Mon Sep 24, 2012 3:01 pm 
Offline
User avatar

Joined: Sat Nov 11, 2006 2:43 pm
Posts: 1221
Location: This poster has left the forums
I knocked up a dirty server to simply display my headers.

The behavior of TCPSocket1.SetPostContent has changed between 2011R4.3 and 2012
The following
TCPSocket1.SetPostContent ("Just a long string of text that says nothing much at all.","")

creates two different requests when executed in both versions.

In 2011R4.3 the header parameter included "Content-type:" (the parameter was included with an empty value)
Image
In 2012 the header parameter is omitted completely if it does not have a value.
Image

_________________
%Invalidforumsignatureexception% user signature not found


Top
 Profile  
Reply with quote  
 Post subject: Re: 2012R1 HTTPSocket Bug
PostPosted: Sat Sep 29, 2012 6:18 am 
Offline
User avatar

Joined: Fri Oct 28, 2005 7:05 am
Posts: 565
Location: Emsworth, UK
pony wrote:
In 2012 the header parameter is omitted completely if it does not have a value.
Which is (probably) a better interpretation of the HTTP protocol specs. HTTP headers are name, value pairs. A null header value is never valid, as a null value is implied by the absence of the name.

When you declare a HTTP header name, you should always assign a corresponding value.
When you send a POST request you should always declare the Content-Type header.
When you don't do what you should do, behaviour is no longer defined by the protocol.

Have you tried setting Content-Type to "text/plain"

There is a list of content types on Wikipedia
http://en.wikipedia.org/wiki/Internet_media_type
Pick whichever one suits.

_________________
Yes it's me in the avatar


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 5 hours


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group