/prog/ I come to you for help with my superior ENTERPRISE(tm) programming.
The test environment is Windows XP running under VirtualBox and I'm using Python to execute the program.
Basically I want to send n amount of data with socket.send and have a reliable estimate on how long it has taken to send n bytes. I am using sockets with a timeout of 15 seconds. Let me explain in my own words why I haven't been able to do what I want:
Apparently send works with network buffers. It just adds what I'm trying to send to the network buffer and returns the amount of data it passed to this buffer, only ever blocking/sleeping when the buffer is full. So in other words, send doesn't actually send anything. So even when I send large amounts of data, 100MB or more, send returns instantly, WTF.
Shouldn't the buffer be a reasonable size, and if send really does work like this how am I to determine how many bytes have been actually sent?
Note: I don't want to use a protocol, i.e. have the client return how many bytes received because I'm working on a socket class.
Note: I don't have this problem with recv it seems . . .
1. Don't.
2. Use a raw IP socket and implement TCP yourself.
4. Sniff packets and read the sequence numbers off your connection.
8. 100MB network buffers, really?
>>2
and anon. Raw sockets really won't work well on Windows. They've been severely limited. Still, thanks for the reply.
Name:
Anonymous2011-01-15 2:36
Well, firstly, socket.send() calls the underlying send(), so assuming you really are checking the return value (have you tried sendall()?), it's all passed on to the OS.
It could be VirtualBox overcommitting its virtual hardware - a bad case of what the hip kids apparently call `bufferbloat' or something these days - have you checked whether the the memory is taken from the virtual machine or the host? Those 100MB have to come from somewhere.
Name:
Anonymous2011-01-15 2:51
>>5
I just don't understand though. Why would the buffer be so big if XP can't keep up with emptying it fast enough? Could this possibly be a bug with VirtualBox? I guess I don't understand what's going on here . . .
Let me rewrite my code to use sendall. Thanks a lot for replying to me. You are saving this neckbeard some hell.
Does it really matter? I'm sure it's some toy program, just show how much you've given to the OS, and trust the OS to do its job and send the data. Obviously if it can't you'll get an error, otherwise your estimate of how much has been sent (based on the amount given to the OS) will be accurate enough[citation needed].
Name:
Anonymous2011-01-15 15:27
>>4 When Microsoft released Windows XP in 2001 with raw socket support implemented in the Winsock interface, the media criticized Microsoft asserting that raw sockets are only of use to hackers to perform TCP reset attacks. Three years after the Windows XP release, Microsoft silently limited Winsock's raw socket support in a non-removable hotfix and offered no further support or workarounds for applications that used them.
Oh dear, that's retarded.
>>13 asserting that [some technology is] only of use to [criminals] to perform [illicit acts]
This is exactly how stupid people ruin your day. It's selfish too.
What is this retarded shit.
How are you going to time the sending of the data?
What process is at the distant end?
A posix send will return -1 on error, else the number of bytes actaully written. If you fill the buffer, it will return a number less than what you requested to send, and you'll have to call send again starting at the appropriate offset in the data passed to your first send.
if it returns the same as your input buffer, then it was all "sent".
the delays you speak of at the OS level - from arriving in the OS's buffer for that socket, to actual sending is not important. it's not going to wait that long for a tcp socket. so dont worry about that.
OP again. As said before I really appreciate the replies. I'm going to work hard on this problem today and think a lot about your replies.
Some more information for /prog/. I'm sending the data to a remote host on the internet. Say 100MB worth. My upstream for this connection is 0.2 MB/s so I do not understand why send would immediately return.
I need to understand the problem more.
Name:
Anonymous2011-01-15 22:59
>>17
IMO you might as well deal with a bit of buffer. If you're not saturating the throughput capacity your measurements are going to reveal more about the scheduler and system performance than anything else. If you do saturate then I/O will get held up and you will get a meaningful measurement. Your OS might choose to express this through scheduling, but the result will match up with your output about as accurately as you can hope for.
Name:
Anonymous2011-01-15 23:07
>>18
I don't really understand. Are you telling me to try reduce the network buffer sizes?
Shitty test code I wrote specifically to demonstrate the issue:
import socket
import time
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP)
print "Send buffer size (sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF))", sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF)
sock.connect(("www.google.com", 80))
#buf = "GET /dicks.html HTTP/1.1\r\n"
#buf += "Host: www.google.com\r\n\r\n"
buf = "x" * ((1024 * 1024) * 10)
print "time.clock()", time.clock()
print "sent:", sock.send(buf), "bytes"
print "time.clock()", time.clock()
sock.close()
Now am I right in assuming the maximum amount of data send should be able to send each call is the same amount returned from sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF) and if so why are the results different.
Look at this log file I found online for example which is very similar to my code:
[quote]
0811 15:58 chronis:~% python !$
python scratch/sendtst.py
connected from ('208.210.124.49', 1258)
sent 17520/10000000 chars
done
0811 15:58 chronis:~%
[/quote]
You see here it sent an amount very close to the size of my send buffer!
Here is his code:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect("chronis.pobox.com", 8010)
sent = sock.send(s)
if sent != len(s):
print "sent %d/%d chars" % (sent, len(s))
else:
print "sent all chars"
>>19
No, I'm saying you should just live with the fact that it's buffered. Why are you trying to take a measurement anyway? If it has anything to do with time estimations the data will be sent before you can get enough samples to make an estimate anyway. If it is to measure throughput capacity you can't do that without saturating anyhow.
Name:
Anonymous2011-01-16 1:01
>>22
I need a measurement because I want to limit the upstream.
You can try writing a wrapper over the socket which will do its own buffering and then call send or sendall or w/e and calculate/limit the speeds.
But it might very well be that your code doesn't work because you're running it in a VM.
Name:
Anonymous2011-01-16 7:08
OP here. I'm bumping with my solution.
So I kept going on and on about buffers. I ended up turning them off completely, but only for send. Now it works perfectly!
In Python I do that by using (after I create a socket):
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 0)
So this means send isn't buffered at all. You told me to accept the fact that these functions are buffed, alas' I have done it my way and it works. No more buffering for send.
With this implementation, send actually seems to block until all of the data you have passed it has really been sent, thus it can even be used to measure average upstream if you fully saturate the upstream for at least 4 seconds.
I'm sure send may not send all the bytes you pass it so a loop will probably still be needed.
I hope this helps someone as /prog/ has helped me.
I should note this does not work on my Fedora box. I really do hope it works on all the Windows OS' . . . though I don't count on it, have yet to test it. My Python version is 2.6.6 by the way.
I am Heron of Alexandria. I ahev a large beard and I create mathematical formulas. If you don't repost this comment on 10 other pages, I will use my primitive steam engine to induce mould in your walls.
Name:
Anonymous2011-01-16 22:13
i am a heron. i haev a long neck and i pick fish out of the water w/ my beak. if you dont post in this thread i will fly into your kitchen tonight and make a mess of your pots and pans
I am oberon. i am king of fairies and a powerful sorcerer. If you dont post in this thread i will cast fire into your linen closet and singe your bed sheets