wget -q http://www.sina.com.cn -O - | \
tr "\t\r\n'" ' "' | \
grep -i -o '<a[^>]\+href[ ]*=[ \t]*"\(ht\|f\)tps\?:[^"]\+"' | \
sed -e 's/^.*"\([^"]\+\)".*$/\1/g' \
| \
xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective};%{http_code};%{time_total};%{time_namelookup};%{time_connect};%{size_download};%{speed_download}\n'

拼凑了一下 stackoverflow 上的例子，能达到效果，@jeffjade 你可以试试。
参考：
http://stackoverflow.com/questions/2804467/spider-a-website-and-return-urls-only
http://stackoverflow.com/questions/6136022/script-to-get-the-http-status-code-of-a-list-of-urls