Comments on: Using Python to Grab Images From a Web Site

By: michaelangela

michaelangela — Thu, 16 Oct 2008 01:01:15 +0000

Ah… forgot to add the

| xargs wget -q

at the end of the curl,egrep line to do the actual downloading.

By: Ramblings

Ramblings — Wed, 15 Oct 2008 17:29:57 +0000

grep instead of python?…

…

By: michaelangela

michaelangela — Wed, 15 Oct 2008 17:21:08 +0000

Great tip using grep. I was going to use python, too. My box doesn’t have the -P option but I used egrep to similar effect. Also, the items I needed were in tags so I had to remove those with a sed command. The images were also retrieved from a database and didn’t have an extension so I did a one liner loop to rename those.

curl -s http://domain.tld/feed | egrep -o “.*” | egrep -o “(http.*)” | sed -e ’s/]*>//g’
for f in *; do mv ./”$f” “${f}.jpg”; done

By: ThinkingSerious

ThinkingSerious — Fri, 02 May 2008 22:06:25 +0000

Great example. Thanks for your contribution.

By: btruelove

btruelove — Fri, 02 May 2008 17:42:49 +0000

Overkill IMO. A less trivial example is going to get large and messy quickly. What about when the XML file in on an FTP, or requires authentication, needs some cookie, has to filter downloads by file size, wants to use a proxy, spawn multiple processes and so on? A more apt tool is curl or wget (I’m lazy so I used both). Also, when you keep it at the shell it’s more natural to pull in other shell commands when needed.

curl -s http://99designs.com/contests/6999/feed | grep -Po “src=\”.*(png|jpg)” | grep -o “http.*” | xargs wget -q