Real Software Forums

The forum for Real Studio and other Real Software products.
[ REAL Software Website | Board Index ]
It is currently Mon Dec 10, 2018 8:26 am
xojo

All times are UTC - 5 hours




Post new topic Reply to topic  [ 10 posts ] 
Author Message
 Post subject: How many bytes should/can I write to a TCPsocket at once?
PostPosted: Fri Sep 07, 2012 12:02 pm 
Offline
User avatar

Joined: Mon Apr 02, 2007 2:08 am
Posts: 1225
Location: San Francisco, CA, USA
I'm using a TCPSocket to upload data to a server, but the TCPSocket is still sending the data long after the TCPSocket.SendComplete event fires! Packets are still being sent over the wire and the server sends its 'transfer complete' response as soon as the packets stop being sent.

My assumption right now is that the TCPSocket is buffering/queuing my writes. At the moment, I'm writing 64k of data before calling TCPSocket.Flush:
MySocket.Write(MyBinaryStream.Read(64 * 1024))
MySocket.Flush


Should I use 32k? 16k? I can't just dump the whole file at once since the file may be tens of GB in size. Is there something obvious that I'm missing?

_________________
Boredom Software


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Sep 12, 2012 10:28 am 
Offline
User avatar

Joined: Fri Oct 28, 2005 7:05 am
Posts: 565
Location: Emsworth, UK
charonn0 wrote:
My assumption right now is that the TCPSocket is buffering/queuing my writes.
You're application is sending data to an OS protocol stack, which then takes care of delivering it across the network. I don't know, because I have never tested it, but if you are getting a .SendComplete before all bytes have been sent to the wire, I can only imagine it indicates the data has been sent to the protocol stack, which is doing it's own buffering.

Quote:
At the moment, I'm writing 64k of data before calling TCPSocket.Flush:
Flush immediately empties the transmit buffer. You should avoid calling flush, unless you are intending to abandon the connection.
E.g. The far end disconnected before the outgoing transmission was complete.

Could it be the call to Flush is causing SendComplete to fire prematurely?

Quote:
Should I use 32k? 16k?
Whatever you send above the MTU size (usually around 1500 bytes), will get carved into multiple packets by the Protocol Stack, so this issue is unlikely to go away just by reducing the chunk size.

Quote:
Is there something obvious that I'm missing?
At the application layer (sockets), how the TCPStack delivers data lower down the stack, is not your application's concern. However, applications which transfer large amounts of data often implement their own progress and acknowledgement protocols. If your application is connected to someone else's server application, you need to comply with the server application's protocol (FTP, SMTP, HTTP etc). If it's your own server application, you might need to implement your own application layer protocol.

_________________
Yes it's me in the avatar


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Sep 12, 2012 11:59 am 
Offline
User avatar

Joined: Mon Apr 02, 2007 2:08 am
Posts: 1225
Location: San Francisco, CA, USA
msssltd wrote:
Could it be the call to Flush is causing SendComplete to fire prematurely?

I thought about that too, and a I removed the Flush line with no ill effects, though it hasn't solved the issue.

Quote:
At the application layer (sockets), how the TCPStack delivers data lower down the stack, is not your application's concern. However, applications which transfer large amounts of data often implement their own progress and acknowledgement protocols. If your application is connected to someone else's server application, you need to comply with the server application's protocol (FTP, SMTP, HTTP etc). If it's your own server application, you might need to implement your own application layer protocol.


The remote server is an FTP server, and FTP specifies that response 226 should be sent by the server to the client when an upload finishes. I'm getting 226 and the file transfers successfully, but the TCPSocket's SendComplete fires before response 226 is sent, and the length of the delay is correlative to the size of the file: larger file, longer delay.

For now, I'm going to disregard the TCPSocket's event and simply wait for a response code from the server.

Thanks for your thoughts :)

_________________
Boredom Software


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Sep 12, 2012 1:07 pm 
Offline

Joined: Fri Jan 06, 2006 3:21 pm
Posts: 12388
Location: Portland, OR USA
Quote:
The remote server is an FTP server, and FTP specifies that response 226 should be sent by the server to the client when an upload finishes. I'm getting 226 and the file transfers successfully, but the TCPSocket's SendComplete fires before response 226 is sent, and the length of the delay is correlative to the size of the file: larger file, longer delay.

That sounds normal. SendComplete just means the data has left your application. There could be many things that transpire before you get an acknowledgement from the remote side.


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Sep 12, 2012 1:13 pm 
Offline

Joined: Tue Mar 23, 2010 8:44 pm
Posts: 673
timhare wrote:
Quote:
The remote server is an FTP server, and FTP specifies that response 226 should be sent by the server to the client when an upload finishes. I'm getting 226 and the file transfers successfully, but the TCPSocket's SendComplete fires before response 226 is sent, and the length of the delay is correlative to the size of the file: larger file, longer delay.

That sounds normal. SendComplete just means the data has left your application. There could be many things that transpire before you get an acknowledgement from the remote side.



When sending larger files for speed try using larger data chunks such as 2mb-5mb, maybe larger. I'm not sure if anything larger than 5mb will benefit but you can test.
The trade off would be that your app would now probably require more ram.

MySocket.Write(MyBinaryStream.Read(2 * 1024 * 1024)) ' 2mb chunk.


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Tue Oct 30, 2012 12:40 am 
Offline

Joined: Mon Aug 15, 2011 10:25 pm
Posts: 293
I had wrote a FTP library in Runtime Revolution and had noticed that there could still be data transferring even after getting the 226 response code and just ended up waiting for the server to close the socket after the file transmission..


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Tue Oct 30, 2012 12:24 pm 
Offline
User avatar

Joined: Mon Apr 02, 2007 2:08 am
Posts: 1225
Location: San Francisco, CA, USA
shaosean wrote:
I had wrote a FTP library in Runtime Revolution and had noticed that there could still be data transferring even after getting the 226 response code and just ended up waiting for the server to close the socket after the file transmission..

That's pretty much what I'm seeing, too. I'll try your suggestion, thanks!

_________________
Boredom Software


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Oct 31, 2012 12:37 am 
Offline

Joined: Mon Aug 15, 2011 10:25 pm
Posts: 293
Here are a few other little tidbits that I discovered..

* When doing a passive LIST or RETR command, only the Mac-based Rumpus server correctly sends the 150 reply before sending the listing/file data
* Some servers will reply with 150 and send a blank directory listing for directories that do not exist
* The 226 reply to LIST, RETR, STOR and ABOR is sometimes sent before the transaction is completed, therefore you are better off checking to ensure that the data connection has closed instead of relying on this status code
* The reply to the PASV command can receive one of two formats to determine the data connection
- the old format (12,34,56,78,123,34); where the first four items are the IP Address and the last two are used to create the port number ((123 * 256) + 34)
- the "new" format ||||58011; where the IP Address is the same as the control socket and the number passed is the port

_________________
Real Studio 2012r1.1 | MacBook Pro i5, 10.6.8 | Windows 7


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Wed Oct 31, 2012 4:01 am 
Offline
User avatar

Joined: Wed May 10, 2006 2:42 pm
Posts: 2985
Location: Germany
I think you can easily pass a 5 MB buffer to TCPSocket.
The OS will make the packaging.
And I never use flush on sockets.

Greetings
Christian

_________________
See you in Orlando, Florida for Real World 2013
More details and registration here:
http://www.realsoftware.com/community/realworld.php


Top
 Profile  
Reply with quote  
 Post subject: Re: How many bytes should/can I write to a TCPsocket at once
PostPosted: Thu Nov 01, 2012 11:24 pm 
Offline

Joined: Sat May 26, 2007 11:37 pm
Posts: 52
Location: Tumwater, Wa
The short answer to a long explanation, the OS Stack could be buffering on both ends - so yes


The Server that you are sending data to, could be implementing TCP / Flow control with a low window size.
the window size is advertised between the two stations that are communicating with TCP.

example, client wants to send data to a server via TCP,

the sever could advertise that it can only receive 100 bytes at a time from the client.

- so the client can only send 100bytes at a time

the Server receives the Data, places it in its buffer, and then sends the ACK to the client,

Then the servers ip/stack processes the data and then sends it up the OSI layers ending at the application

But the server doesn't have to immediately transfer the data out of the buffer.

the client will also advertise a receive window to the server.

when a buffer gets full, one technique a receiving machine can do, to control the flow of data is to re-advertise a receive window of 0, which tells the transmitting machine to hold off on sending data,.. until the receiving machine re-advertises a window size larger than 0.


Also the Nagles theorem could be playing a piece into this as well.
which implements congestion control for ip/tcp.
essentially, Nagles theorem combines small out going segments/packets together and send them out all at once. specifically long as there is sent segments/packets that there isn't an acknowledgement the sender should keep buffering its ouput until it has a full payload of 1460, so the payload can be sent all at once.

you've got about 40bytes overhead with TCP and IPv4, the header for tcp is about 20,and the header for ipv4 is about 20, this gives you a payload of about 1460 to use, as the system will tack on 40bytes to give you an mtu of 1500.

I personally like to use wireshark when I'm experiencing application slowness/weirdness over a network, as the packet capture will really show you whats happing on the wire.

~ Good Luck ~ if you decide to use a packet capture application and need help in interpretation, feel free to drop a note..


Nux :)

_________________
• program, n.: A magic spell cast over a computer allowing it to turn one's input into error messages.
tr.v. To engage in a pastime similar to banging one's head against a wall, but with fewer opportunities for reward


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC - 5 hours


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group