最新消息：20210816 当前crifan.com域名已被污染，为防止失联，请关注（页面右下角的）公众号
    你的位置：在路上 > 工作和技术 > Web > Crawl_EmulateLogin > 【记录】给BlogsToWordPress添加支持导出网易的心情随笔

    【记录】给BlogsToWordPress添加支持导出网易的心情随笔

    Crawl_EmulateLogin crifan 5790浏览 0评论

    【背景】

    之前的

    BlogsToWordpress

    不支持网易的心情随笔。

    现在去添加此功能。

    【解决过程】

    1.结果使用：

    BlogsToWordpress.py -s http://blog.163.com/ni_chen

    竟然结果连第一个帖子地址都找不到了。

    2.所以去用Firebug调试网易博客，发现，原先的获得对应的帖子信息的访问，从之前的GET变成现在的POST了。

    所以，把旧的GET的代码，

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    getBlogUrl = genGetBlogsUrl(userId, startBlogIdx, onceGetNum);
    logging.info("getBlogUrl=%s", getBlogUrl);
    # get blogs
    blogsResp = crifanLib.getUrlRespHtml(getBlogUrl);
      
      
    #------------------------------------------------------------------------------
    # generate get blogs URL
    def genGetBlogsUrl(userId, startBlogIdx, onceGetNum):
        getBlogsUrl = '';
      
        try :
            # http://api.blog.163.com/againinput4/dwr/call/plaincall/BlogBeanNew.getBlogs.dwr
            # callCount=1
            # scriptSessionId=${scriptSessionId}187
            # c0-scriptName=BlogBeanNew
            # c0-methodName=getBlogs
            # c0-id=0
            # c0-param0=number:172799491
            # c0-param1=number:0
            # c0-param2=number:20
            # batchId=955290
      
            paraDict = {
                'callCount'     :   '1',
                'scriptSessionId':  '${scriptSessionId}187',
                'c0-scriptName' :   'BlogBeanNew',
                'c0-methodName' :   'getBlogs',
                'c0-id'         :   '0',
                'c0-param0'     :   '',
                'c0-param1'     :   '',
                'c0-param2'     :   '',
                'batchId'       :   '1',
            };
            paraDict['c0-param0'] = "number:" + str(userId);
            paraDict['c0-param1'] = "number:" + str(startBlogIdx);
            paraDict['c0-param2'] = "number:" + str(onceGetNum);
              
            mainUrl = gConst['blogApi163'] + '/' + gVal['blogUser'] + '/' + 'dwr/call/plaincall/BlogBeanNew.getBlogs.dwr';
            getBlogsUrl = crifanLib.genFullUrl(mainUrl, paraDict);
      
            logging.debug("Generated get blogs url %s", getBlogsUrl);
        except :
            logging.debug("Can not generate get blog url.");
      
        return getBlogsUrl;

    希望改为最新的，POST的代码。

    3。调试期间发现：

    其中的post data，还不是普通的以&分隔的，而是以换行分隔的：

    request body is special with seperate via crlf

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    callCount=1
    scriptSessionId=${scriptSessionId}187
    c0-scriptName=BlogBeanNew
    c0-methodName=getBlogs
    c0-id=0
    c0-param0=number:186541395
    c0-param1=number:0
    c0-param2=number:1
    batchId=123756

    所以， 还是比较特殊的。

    结果折腾期间，还出错了：

    LINE 959  : INFO     getBlogsDwrMainUrl=http://api.blog.163.com/ni_chen/dwr/call/plaincall/BlogBeanNew.getBlogs.dwr          
    LINE 971  : INFO     postData=c0-id=0          
    batchId=1          
    c0-param1=number:0          
    scriptSessionId=${scriptSessionId}187          
    c0-param2=number:400          
    c0-methodName=getBlogs          
    c0-param0=number:186541395          
    c0-scriptName=BlogBeanNew          
    callCount=1          
    LINE 973  : INFO     req=<urllib2.Request instance at 0x0000000002F45C88>          
    LINE 1011 : INFO     resp=<addinfourl at 49572616L whose fp = <socket._fileobject object at 0x0000000002F2B408>>          
    LINE 1016 : INFO     gVal[‘cj’]=<_LWPCookieJar.LWPCookieJar[<Cookie USERTRACK=221.224.111.74.1366017130803085 for .163.com/>, <Cookie NTESBLOGSI=C625520A2DCC43BB15C722EA6984CD64.app-64-8010 for .blog.163.com/>]>          
    LINE 1013 : INFO     blogsDwrRespHtml=//#DWR-REPLY          
    if (window.dwr) dwr.engine._remoteHandleBatchException({ name:’org.directwebremoting.extend.ServerException’, message:’The specified call count is not a number’ });          
    else if (window.parent.dwr) window.parent.dwr.engine._remoteHandleBatchException({ name:’org.directwebremoting.extend.ServerException’, message:’The specified call count is not a number’ });

    参考：

    httpclient 调用DWR应用时发生The specified call count is not a number 错误

    去把默认的：

    ?
    1
    req.add_header('Content-Type', "application/x-www-form-urlencoded");

    改为：

    ?
    1
    req.add_header('Content-Type', "text/plain");

    才最终获得对应的DWR返回的内容：

    //#DWR-INSERT

    //#DWR-REPLY

    var s0={};var s1={};var s2={};var s3=[];s0.abstractSysGen=1;s0.accessCount=20;s0.allowComment=-100;s0.allowView=-100;s0.blogAbstract="<div><br></div><div><div style=\"line-height: 22px;\"   ><font size=\"4\"   style=\"line-height: 28px;\"   ><br style=\"line-height: 28px;\"   ></font></div><div style=\"line-height: 22px;\"   ><font size=\"4\"   style=\"line-height: 28px;\"   ><br style=\"line-height: 28px;\"   ></font></div></div><font size=\"4\"   >\u535A\u5BA2\u5DF2\u7ECF\u5168\u90E8\u642C\u8D70\u4E86\uFF0C\u8BF7\u79FB\u9A7E\u81F3\uFF1A</font><a target=\"_blank\" rel=\"nofollow\" href=\"http://nichen.info/blogs\"   >http://nichen.info</a><br><div><font size=\"4\"   ><br></font></div><div><font size=\"4\"   ><br></font></div>";s0.blogAttachments=null;s0.blogCount=s1;s0.blogExt=s2;s0.circleCount=0;s0.circleIdList=s3;s0.circleIds=null;s0.classId="fks_084064092094082071087087087095085094087070080087085074081";s0.className="\u968F\u7B14";s0.commentCount=0;s0.comments=null;s0.content="<div><br></div><div><div style=\"line-height: 22px;\"   ><font size=\"4\"   style=\"line-height: 28px;\"   ><br style=\"line-height: 28px;\"   ></font></div><div style=\"line-height: 22px;\"   ><font size=\"4\"   style=\"line-height: 28px;\"   ><br style=\"line-height: 28px;\"   ></font></div></div><font size=\"4\"   >\u535A\u5BA2\u5DF2\u7ECF\u5168\u90E8\u642C\u8D70\u4E86\uFF0C\u8BF7\u79FB\u9A7E\u81F3\uFF1A</font><a target=\"_blank\" rel=\"nofollow\" href=\"http://nichen.info/blogs\"   >http://nichen.info</a><br><div><font size=\"4\"   ><br></font></div><div><font size=\"4\"   ><br></font></div>";s0.contentPlainText=null;s0.id="fks_087065081087083065081085085071072087089069081082087064093083";s0.ip="147.46.115.126";s0.isBlogAbstractComplete=false;s0.isPublished=1;s0.keyName="ID";s0.keyWordCheckedState=0;s0.lastAccessCountUpdateTime=1365142180844;s0.matchedKeyWord=false;s0.modifyTime=1365142180643;s0.moveFrom="NONE";s0.permaSerial="18654139520132782258253";s0.permalink="blog/static/18654139520132782258253";s0.photoIds=null;s0.photoStoreTypes=null;s0.publishTime=1362615778253;s0.publishTimeStr="8:22:58";s0.publisherId=0;s0.publisherNickname=null;s0.publisherUsername=null;s0.rank=5;s0.recomBlogHome=false;s0.ref=false;s0.shortPublishDateStr="2013-3-7";s0.synchMiniBlog=-1;s0.tag="";s0.title="\u642C\u8D70\u5566";s0.trackbackCount=0;s0.trackbackUrl="blog/18654139520132782258253.track";s0.userId=186541395;s0.userName="ni_chen";s0.userNickname="Neysa";s0.valid=0;s0.zipContent=null;

    s1.accessCount=20;s1.blogId=1251225334;s1.commentCount=0;s1.mainCommentCount=0;s1.permaSerial="18654139520132782258253";s1.recommendCount=0;s1.trackbackCount=0;s1.userId=186541395;

    s2.blogId=1251225334;s2.doubanResourceInfo=null;s2.miniBlogCard=0;s2.userId=186541395;s2.voteId=0;

    dwr.engine._remoteHandleCallback(‘1′,’0’,[s0]);

    接下来，就可以正常的去解析此内容了。

    4.最后，终于可以使用POST的代码，获得对应的内容了：

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    blogsDwrRespHtml = getBlogsDwrRespHtml(userId, startBlogIdx, onceGetNum);
    logging.debug("blogsDwrRespHtml=%s", blogsDwrRespHtml);
      
    def getBlogsDwrRespHtml(userId, startBlogIdx, onceGetNum):
        # getBlogUrl = genGetBlogsUrl(userId, startBlogIdx, onceGetNum);
        # logging.info("getBlogUrl=%s", getBlogUrl);
        # # get blogs
        # blogsRespHtml = crifanLib.getUrlRespHtml(getBlogUrl);
          
        #change GET to POST
      
        #http://api.blog.163.com/ni_chen/dwr/call/plaincall/BlogBeanNew.getBlogs.dwr
        #callCount=1
        #scriptSessionId=${scriptSessionId}187
        #c0-scriptName=BlogBeanNew
        #c0-methodName=getBlogs
        #c0-id=0
        #c0-param0=number:186541395
        #c0-param1=number:0
        #c0-param2=number:1
        #batchId=494302
          
        # http://api.blog.163.com/againinput4/dwr/call/plaincall/BlogBeanNew.getBlogs.dwr
        # callCount=1
        # scriptSessionId=${scriptSessionId}187
        # c0-scriptName=BlogBeanNew
        # c0-methodName=getBlogs
        # c0-id=0
        # c0-param0=number:172799491
        # c0-param1=number:0
        # c0-param2=number:20
        # batchId=955290
      
        postDict = {
            'callCount'     :   '1',
            'scriptSessionId':  '${scriptSessionId}187',
            'c0-scriptName' :   'BlogBeanNew',
            'c0-methodName' :   'getBlogs',
            'c0-id'         :   '0',
            'c0-param0'     :   '',
            'c0-param1'     :   '',
            'c0-param2'     :   '',
            'batchId'       :   '1',
        };
        postDict['c0-param0'] = "number:" + str(userId);
        postDict['c0-param1'] = "number:" + str(startBlogIdx);
        postDict['c0-param2'] = "number:" + str(onceGetNum);
        #http://api.blog.163.com/ni_chen/dwr/call/plaincall/BlogBeanNew.getBlogs.dwr
        getBlogsDwrMainUrl = gConst['blogApi163'] + '/' + gVal['blogUser'] + '/' + 'dwr/call/plaincall/BlogBeanNew.getBlogs.dwr';
        logging.debug("getBlogsDwrMainUrl=%s", getBlogsDwrMainUrl);
              
        #Referer    http://api.blog.163.com/crossdomain.html?t=20100205
        headerDict = {
            'Referer'       :   'http://api.blog.163.com/crossdomain.html?t=20100205',
            'Content-Type'  :   "text/plain",
        };
        blogsRespHtml = crifanLib.getUrlRespHtml(getBlogsDwrMainUrl, postDict=postDict, headerDict=headerDict, postDataDelimiter='\r\n');
      
        logging.debug("blogsRespHtml=%s", blogsRespHtml);
              
        return blogsRespHtml;

    5.等待后续再去添加 网易的 心情随笔。

    【后记 2013-09-22】

    1.后来，实现了对应的抓取心情随笔：

    【教程】以抓取网易博客帖子中的最近读者信息为例，手把手教你如何抓取动态网页中的内容

    以及：

    【记录】用Python解析网易163博客的心情随笔FeelingCard返回的DWR-REPLY数据

    转载请注明：在路上 » 【记录】给BlogsToWordPress添加支持导出网易的心情随笔

    Post Views: 1,826

    与本文相关的文章

    发表我的评论
    取消评论

    表情

    Hi，您需要填写昵称和邮箱！

    • 昵称 (必填)
    • 邮箱 (必填)
    • 网址

    网友最新评论 (3)

    1. 你好，我看过了你写的网易博客的采集评论，最近在学习网易博客用火车头采集博客内容，但是遇到很多困难，问了很多人都无法解决啊。。希望大神可以解决一下。。酬劳可以商量。。谢谢
      黄良11年前 (2014-06-11)回复
      • 加我的技术群：104028266 然后再讨论细节。
        crifan11年前 (2014-06-12)回复
    2. 终于导过去了，谢谢啊。 还有一点小小的建议，就是留言板好像不能导。
      Neysa11年前 (2013-12-26)回复
    87 queries in 0.480 seconds, using 22.31MB memory