Back to files

Download Machine

          simple, non interactive, non graphical, batch download tool
              for FTP, FSP and HTTP downloads with resume support
                     Licensed as Open Source Software (GPL)

                         Radim Kolar (hsn/at/netmag.cz)
                        http://dmachine.sourceforge.net/

How to use:

Configure it by editing dmachine.cnf. Normally not needed.
Run the background download daemon: java dmachine
Put some URLs into queue file:

Open queue file (default name is queue) with any ASCII editor and place required URL to it, one per line. I use copy & paste between Netscape and text editor.

Smart Cache proxy server can be configured to wrote download requests to the queue file. see http://scache.sourceforge.net

4. Save queue file and wait :)

5. Queue file(s) is checked by daemon, so you can add more URL to it and save it.

6. DM has also drop directory support. This is a special directory; any

file in this directory will be processed as queue and then deleted.

Special commands in queue file

Specifying an alternative URLs for file

Download Machine uses only filename part of URL. If you are place:

http://www.server.com/file.zip (and)
http://mirror.server.com/mirrored/file.zip

into queue file, DM will thinks that there are alternative URLs for the same file.

If the same file is on the net with different filename you can use: http://www.server.com/stuff/file-1.2.3.zip#file123.zip http://www.server.org/stuff/file-123.zip#file123.zip

2. Custom Referer header

Some servers checks Referer header. Download Machine sends fake Referer header, which works in most time, but sometimes you will need a custom Referer header.

If you place into queue file:
Referer http://216.42.31.135/ http://www.zephyr.com/ every request to http://216.... will gets second (zephyr) URL as Referer.

3. Custom cookie header

Some servers needs Cookie header to be sent. Cookies are often used for authorization purposes and sometimes server can refuse a download if no valid cookie was found in request. First argument is server name and second is cookie value in format name=value[; ...]

cookie http://www.fileplanet.com/ auth=542345123456 every request to http://216.... will gets second (zephyr) URL as Referer.

4. Check also other directory if file is already downloaded

check_also <regexp> <directory>

example: check_also *.mp3 /home/hsn/mp3/Krishna_Das/ 5. Save file to alternate directory

save_to <regexp> <directory>

example

save_to *.mp3 /home/hsn/mp3/Krishna_Das/

Limitations

Access to Password-protected HTTP/FTP files fails for sure because Java 1.1 can't parse URL with passwords. Send me your reports about that if you want...

Future

Make file info's data persistent between dm runs Support lastmod for FTP
Caching open FTP connections
Support for scheduled downloads
Support for min,max CPS