NETBRAKE readme
Copyright(C) 2001 Salvatore Sanfilippo All right reserved
Distribution of this document is unlimited, providing that you include the above copyright notice.
OVERVIEW
Netbrake is an utility to limit the bandwidth used by a process. It does not require to modify the kernel, nor to be root in order to work. It is useful when you need to download a very big file from a fast server to avoid a network congestion that will result in a too slow web/irc/... experience. I use it mostly to download big files (like the kernel source code) with wget.
Netbrake also implements a very simple HTTP filesystem extension, so you can use the standard text utils against some URL, like:
cat http://www.google.com
grep -i href 'http://www.google.com/search?q=foo'
and get the expected result. While this support is not complete it may be useful sometime.
Netbrake consists of a little dynamic library (libnetbrake.so). This library provides wrappers for important system calls, this makes possible the bandwidth limitation. To override the libc syscalls wrappers with the netbrake's ones you need to use LD_PRELOAD. To improve the easy of use a simple program (netbrake), that exports the right LD_PRELOAD environ variable and run the specified command (the one you need to limit) is included in the distribution.
Netbrake is free software under the terms of the X11 license, see the LICENSE file in the distribution for more information.
SUPPORTED SYSTEMS
Linux/glibc under all archs (tested only on i386 and sparc)
Probably other POSIX systems with glibc and a working LD_PRELOAD with minor changes.
INSTALL
$ ./configure
$ make
$ make install
SIMPLE USAGE
To run a program with the required bandwidth limitation just use the netbrake program. The usage is the following:
netbrake wget http://www.kernel.org/....
you can specify some option to change the behaviour: This is the help screen of the 'netbrake' utility.
Usage: netbrake [options] program [program args]
List of available options:
+ --bps -r <bps> Set the max bytes per second limit
--notcp -T Don't limit TCP sockets (usually limited)
--noudp -U Don't limit UDP sockets (usually limited)
* --rcvbuf -b <bufsize> Set the RCVBUF size (default is 8k)
* --debug -d Enable debugging messages
* --limit-getc -g Limit getc() and fgetc() (default is NOT)
* --httpfs -w Enable the HTTP filesystem.
--httphdr -H Include the HTTP header with HTTP fs
--bpsinfo -p Print on stderr info about bandwidth usage
--limit -l Enable the bandwidth limitation. It is ON
for default but you can use this to override
the NETBRAKE_NETLIMIT=0 environ var.
--help -h Show this help
+ = note that the default BPS limit is 2k for second (2048 bytes) * = use it only if you know what you are doing
I'll try to explain better what this switchs can do:
-r <bytes per second> to set the maximun bandwidth to use. This is probably the main option for the user, example:
netbrake -r 2048 wget http://www.kernel.org/...
will se the max bandwidth to 2k per second, that is the default bandwidth (if -b is not used).
-d switch to turn on the debug mode.
This is not useful if you aren't a programmer, but you should use it if you want to report a bug.
-T to turn off the TCP sockets bandwidth limitation
-U to turn off the UDP sockets bandwidth limitation
-b to set the RCVBUF for the socket. Don't touch this unless you
know what you are doing.
-w Enable the HTTP filesystem. Example:
$ netbrake -w md5sum http://www.google.com d6102f7fc57d4bd279c7fa24c7b99e86 http://www.google.com
As shown the HTTP filesystem provide a way to use http URL with normal text utils. You need to specify the URL as http://HOST[:PORT]/PATH.
This support is for default NOT compiled, ask yes to the right question when you run ./configure to enable it.
-H includes the reply HTTP header with the HTTP filesystem,
usually you don't want this.
-p print the current and average bandwidth used by the limited
process. Note that the information displaied is about the
limited sockets and not about all the traffic generated
by the application.
-l Turn ON the bandwidth limitation in the case you exported the
NETBRAKE_NETLIMIT to "0".
-h shows the shown help screen and exit.
HOW IT WORKS
Libnetbrake wrappers syscalls like socket(2), read(2), write(2) and so on. Once the first call to socket(2) is trapped libnetbrake intializes the internal stuff and starts tracing all the AF_INET family socket descriptors. Every socket that is relevant is set in a bitmap so we have a way to take the list of the socket descriptors we need to trace without to waste too CPU.
For ever read/write operation a list with the few last time/bytes information is compiled. This allows libnetbrake to get an idea about the (almost) 'instantaneous' speed. this speed is for default computed using the last 10 samples. If the application breaks this limit it is put in sleep to adjust the download speed.
If the other side of the connection is sending the data too fast for our 'bps' limit the process will be put in sleep often, this will fill the socket buffer and our system will publish a TCP window of 0 soon. When the application will read new data from the socket, the socket buffer will be again ready to accept new data, so the TCP stack will send some ACK with a non-zero window to the other and. And so on.
To make the download speed very precise we also limit in the same way (putting the process in sleep) the average speed of the download. This will correct errors with the instantaneous BPS guessing.
BTW the socket recv buffer can be very big, for example in linux is 87380 bytes for default, so for little files the limitation is not effective since the buffer will be filled receiving the data ASAP anyway. To fix this problem libnetbrake will set a smaller socket recv buffer for every AF_INET socket() the application will open.
The HTTP filesystem uses the same trick, it isn't a full implementation and may not work with many applications, but I tested it with many standard unix utils without to experiment too problems.
The HTTP filesystem wrappers the open(2) and stat(2) syscalls. when the application try to open a file in read-only and that file contains a string like http://... netbrake open an HTTP connection against the specified web server/port and do an HTTP request, than pass the descriptor as it was returned by the real open(2). If the request was successful the application will read the HTTP server reply like a file, unless it try to do operations that aren't allowed for a socket, like lseek(2).
The same for the stat(2) syscall. netbrake will do an HTTP 'HEAD' request to get the size and the last modification date of the remote file, to fill the "struct stat" structure. The inode is computed hashing the URL itself.
TESTED APPLICATIONS COMMENTS
wget (this is my main target)
linux ftp client
lftp
w3m (needs --limit-getc)
lynx
tcl
APPLICATION THAT SEEMS TO WORK WITH THE HTTP FILESYSTEM
cat sed md5sum tail head grep vim cp ls diff xv
KNOWN PROBLEMS
netbrake can't get a real bandwidth control. It reaches its goal using hacks, so probably it will not work with some application.
It can't be used against multithread applications, the behaviour is undefined, but it shouldn't be hard to fix.
The http filesystem is experimental and doesn't work with all the applications, expecially the one that instead to call open/stat /path/file do something like cd /path open file.
SEE ALSO
CONTACT INFO
For feedbacks and bug reports write to <antirez@invece.org>
NOTE
The project is abandoned because I finally got ADSL, so 0.2 (that fixes a compilation bug in new versions of libc) is probably the last version of netbrake from me.