|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了,ebay、amazon等一些站,采集先看robots.txt,从robots.txt中找到sitemap的索引,剩下的就想办法搞吧
6 Y' `7 e7 `3 c! R# Ahttp://www.ebay.com/robots.txt # sitemaps - SRPs& T5 G/ P- _2 ^& T
Sitemap: http://www.ebay.com/lst/SRP_US_index.xml
7 L; M3 h y8 d2 t% Y: R7 e! t. GSitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml. j* a, |" V1 u( B& l
$ r. ^: ?9 B, k5 `9 ~1 E
# Guides sitemaps$ C2 K4 z( ?1 a: h, j9 m4 y
Sitemap: http://www.ebay.com/lst/GUIDES-0-index.xml
# b4 m u7 r) ~ n2 k- G$ C7 ~; T$ L. \' w$ ^
# SSRP sitemaps
* }& l$ G% u4 MSitemap: http://www.ebay.com/lst/SSRP-0-index.xml
3 G8 ]& j0 q9 V) o2 i# b. B# ^
# k6 g, W+ z, c! l( U#Stores Sitemaps& U8 |9 ~3 [0 o
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml
9 T. B2 }4 ^- }4 X ^2 Y4 o1 |9 z1 {; |4 C+ ~4 Y
#BHP Sitemaps
/ c) C& T1 I# NSitemap: http://www.ebay.com/lst/BHP-0-index.xml
' v5 A' u5 k# Y1 R* B3 Y+ h
2 E( c9 Z9 L4 ^7 C) @; S#Collections
/ a2 a0 N2 \+ C2 b2 k+ { a' ~# lSitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml
8 s+ g3 Y# |) \( H+ w2 K4 s& O" ^7 _' r; k) N3 } Q
#VI# O/ Y( h' q1 |+ d1 ^' _
Sitemap: http://www.ebay.com/lst/VI-0-index.xml
6 @$ G7 c& Q/ P# H+ h6 h: s+ c* s. h1 h0 Z6 w( M
#PRP
8 P# i9 ?& c) X5 F6 g+ U/ dSitemap: http://www.ebay.com/lst/PRP-0-index.xml $ T& O7 ?$ ~+ m5 x3 b: k
, a, _ }1 k1 i& A7 A
|
|