Virendra Rajput (BkVirendra)

home me my projects

Directory Downloader in Python

By Virendra Rajput Tagged:

Recently, while browsing for some Facebook Timeline covers that I wanted for my Facebook Profile. I came across hundreds of covers that I would love to have on my hard-disk as my Timeline Cover collection (yeah, I have a Timeline Cover collection).

And then, I came across a few websites that allowed directory browsing. So I started saving the images manually from the index. And well those directories had thousands of images, and downloading them manually would suck (being a #hacker you always want everything to be automated).

So I started hacking a script, that would carry out this task for me. And in just 15 minutes, I cracked it. And had fun, downloading entire webserver directories in minutes.

You can use this Python script to download entire directories (if the webserver has indexes open).

This script also makes use of Beautifulsoup, you can install it, by using the following command:

pip install beautifulsoup4  # if you have pip installed

 easy_install BeautifulSoup4 # if you have easy_install

For using the script, you need to pass the directory url as a commandline argument to the script, for Eg. 

For downloading the directory at

$ python

The code:

import urllib2
import sys
import os

from bs4 import BeautifulSoup
from urlparse import urlparse

def downloader(urls, grab_url, foldername):
    if not os.path.exists(foldername):
        print "\""+ foldername + "\" does not exist!"
        print "Creating \"" + foldername + "\"..." 
    for cover in urls:
            print "Downloading item " + cover + "..."
            print grab_url + cover
            img = urllib2.urlopen(grab_url + cover)
            output = open(foldername + "/" + cover,''wb'')
            print cover + "... downloaded!!"
        except Exception, e:

def main(url):
    urls = []
    print "Fetching the page..."
    page = urllib2.urlopen(url).read()
    print "Fetching completed!"
    soup = BeautifulSoup(page)
    print "Grabbing the objects of the page..."
    lis = soup.find_all("li")
    for item in lis:
    domain = urlparse(url)
    downloader(urls, url, domain.netloc)
    print "All files have been successfully downloaded!"

if __name__ == "__main__":

You can also fork it on Github here.