====== 抓取与正则 ======

===== 抓取=====

urllib 和 urllib2 中设置http超时
<code python>
import socket		
socket.setdefaulttimeout(5)
</code>

最简洁的抓取语句：<code> urllib.urlopen(url).read() </code>
<code python>u = urllib.urlopen(url)
c = u.read() 
u.close
</code>

POST数据：
<code python>u = urllib.urlopen('http://url', urllib.urlencode({'gtalk':fromid,'msg':content}))
c = u.read()
u.close</code>

传递cookies
<code python>
import cookielib, urllib2, urllib

cj = cookielib.CookieJar()
opener.addheaders=[("Cookie","dV9pZA**=Mg**; iCast2_1470_1097hO=0_2_20160; _iCast2_1470_1097hO=1")]
url_post = 'http://test.soften.cn'
content = opener.open(url_post, urllib.urlencode(posts)).read()
opener.close()

print content
</code>
===== 正则 =====
匹配：
<code python>
m = re.search('tags=(.*?)">', c, re.I+re.S+re.M)
if m:
    print m.group(1)
</code>

替换：
<code python>
pattern = re.compile('<style.*?</style>|<script.*?</script>', re.S | re.I)
html = re.sub(pattern, ' ', html)
</code>

===== URL编码 =====

<code python>
m = {'name' : 'somebody'; 'gender' : 'male'}
s = urllib.urlencode(m)
print s
##gender=male&name=somebody
</code>
<code python>
content = "zhongwen zifu"
urllib.quote(content)
#urllib.unquote(content)
</code>


===== 资料 =====
  * http://www.woodpecker.org.cn/diveintopython/http_web_services/index.html
  * [[http://wiki.ubuntu.org.cn/Python%E6%AD%A3%E5%88%99%E8%A1%A8%E8%BE%BE%E5%BC%8F%E6%93%8D%E4%BD%9C%E6%8C%87%E5%8D%97|Python正则表达式操作指南]]