【已解决】批量处理gitbook的脚本时忽略掉已处理的book

折腾：

【记录】给crifan的gitbook的books去批量make deploy部署

期间，之前弄的：

【已解决】批量编译和发布gitbook电子书 – 在路上

脚本可以批量的去

make deploy

了。但是遇到问题：

偶尔会报错：

然后如果重新

make deploy

又会重头开始，很麻烦。

所以希望：支持指定起始book，这样就可以跳过之前已处理过的book了。

去看看，希望可以指定起始的要处理的book

这样就可以跳过该book之前的book了。

不过也去看看此处的问题：

【已解决】git pull报错：Auto packing the repository fatal failed to run repack

此处脚本中：

SUB_BOOKS=$(shell ls ../books)
BOOKS_SRC_ROOT=$(shell cd ../books && pwd)


# Batch make for all gitbooks
help debug_dir init sync_content clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @echo "LS_OUTPUT="$(SUB_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(SUB_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            echo `pwd`; \
            if [ -f Makefile ]; then \
                make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;

核心就是for循环

如何在for循环中指定起始的，不好指定

突然想到，或许可以通过SUB_BOOKS，去掉某个book之前的？

makefile str list

linux – How to loop on array of string – Stack Overflow

Text Functions (GNU make)

没有合适的函数

Functions (GNU make)

Shell Function (GNU make)

shell split list

想到了：

或许可以在for循环中，加bool变量，在start之前都忽略，等于设置为true，后续发现true，再处理

bash bool value

bash – How can I declare and use Boolean variables in a shell script? – Stack Overflow

makefile bool value

gnu make – Variable in makefile conditional? – Stack Overflow

【已解决】Makefile中如何定义和判断布尔类型变量 – 在路上

makefile for loop

How to write loop in a Makefile? – Stack Overflow

bash – simple loop over files in some directory makefile – Stack Overflow

shell – How to use a for loop in make recipe – Stack Overflow

Makefile：Loop循环的使用 – 简书

bash if else fi

Bash if..else Statement | Linuxize

if [[ $VAR -gt 10 ]]
then
  echo "The variable is greater than 10."
else
  echo "The variable is equal or less than 10."
fi

4 Bash If Statement Examples ( If then fi, If then else fi, If elif else fi, Nested if )

bash inside if set value

if statement – BASH: Basic if then and variable assignment – Stack Overflow

Unix Bash – Assign if/else to Variable – Stack Overflow

bash set value

Variable Assignment

BASH Programming – Introduction HOW-TO: Variables

How to Work with Variables in Bash

折腾了半天：

START_BOOK = automobile_sales_summary


SUB_BOOKS=$(shell ls ../books)
BOOKS_SRC_ROOT=$(shell cd ../books && pwd)


IS_PROCESS = false


ifeq ($(START_BOOK), )
IS_PROCESS = true
$(info START_BOOK is empty, now IS_PROCESS=$(IS_PROCESS))
else
IS_PROCESS = false
$(info START_BOOK=$(START_BOOK), not empty, now IS_PROCESS=$(IS_PROCESS))
endif


# Batch make for all gitbooks
help debug_dir init sync_content clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @# echo "LS_OUTPUT="$(SUB_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(SUB_BOOKS); \
    do \
        if [ $IS_PROCESS = true ] ; then \
            echo IS_PROCESS is true, set to false; \
            $IS_PROCESS=false; \
        else \
            echo IS_PROCESS is false, set to true; \
            $IS_PROCESS=true; \
        fi; \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            # echo `pwd`; \
            if [ -f Makefile ]; then \
                echo $$each_item; \
                # make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;

还是没有实现要的效果。

放弃。

另外，也想到了，用忽略omit的方式，但是觉得太麻烦。还要挨个去拷贝book的name，麻烦。算了。

后来遇到：运行了一晚上，结果还是由于错误停在某个book

导致后续重新全部运行，还是太慢，无法接受，所以还是去加上吧

即使是omit list，也比没有强

去加上忽略列表，在列表中的，不处理

makfile findstring

Linux之Makefile（findstring）_zhoudengqing的专栏-CSDN博客

使用函数 — 跟我一起写Makefile 1.0 文档

用：

BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
$(info BATCH_PROCESS_IGNORE_FILE=$(BATCH_PROCESS_IGNORE_FILE))
$(info ----------------------------------------)


BATCH_PROCESS_IGNORE_BOOKS := $(shell cat $(BATCH_PROCESS_IGNORE_FILE))
$(info BATCH_PROCESS_IGNORE_BOOKS=$(BATCH_PROCESS_IGNORE_BOOKS))
$(info ----------------------------------------)


SUB_BOOKS=$(shell ls ../../books)
$(info SUB_BOOKS=$(SUB_BOOKS))
$(info ----------------------------------------)


BOOKS_SRC_ROOT=$(shell cd ../../books && pwd)
$(info BOOKS_SRC_ROOT=$(BOOKS_SRC_ROOT))
$(info ----------------------------------------)


TO_PROCESS_BOOKS := $(filter-out $(BATCH_PROCESS_IGNORE_BOOKS), $(SUB_BOOKS))
$(info TO_PROCESS_BOOKS=$(TO_PROCESS_BOOKS))
$(info ----------------------------------------)


help:
    echo help;

已经可以基本上输出我们要的了：

●  make help
BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
----------------------------------------
BATCH_PROCESS_IGNORE_BOOKS=5g_message_rcs_tech_summary 5g_tech_summary all_age_sports_badminton android_app_security_crack android_automation_uiautomator2 api_tool_postman app_capture_package_tool_charles apple_develop_summary automobile_sales_summary best_editor_vscode china_suitable_living_suzhou common_logic_hardware_embedded computer_tech_summary crack_assistant_xposed_framework crawl_your_data_spider_technology desktop_app_framework_electron doc_format_markdown ebook_system_gitbook editor_ide_summary gitbook_demo good_automation_tool_makefile html_parse_tool_beautifulsoup http_restful_api http_summary ic_chip_industry_chain_summary improve_work_efficiency industrial_automation_plc industrial_control_security_overview information_security_overview ios_automation_facebook_wda learn_tech_method_experience mobile_app_summary mobile_automation_overview mobile_network_evolution_history most_intelligent_python_ide_pycharm multimedia_core_system_ims popular_document_db_mongodb program_code_style program_common_logic
----------------------------------------
SUB_BOOKS=5g_message_rcs_tech_summary 5g_tech_summary all_age_sports_badminton android_app_security_crack android_automation_uiautomator2 api_tool_postman app_capture_package_tool_charles apple_develop_summary automobile_sales_summary best_editor_vscode china_suitable_living_suzhou common_logic_hardware_embedded computer_tech_summary convertToNewCommonTemplate.sh crack_assistant_xposed_framework crawl_your_data_spider_technology desktop_app_framework_electron doc_format_markdown ebook_system_gitbook editor_ide_summary gitbook_demo good_automation_tool_makefile html_parse_tool_beautifulsoup http_restful_api http_summary ic_chip_industry_chain_summary improve_work_efficiency industrial_automation_plc industrial_control_security_overview information_security_overview ios_automation_facebook_wda learn_tech_method_experience make_life_better_python mobile_app_summary mobile_automation_overview mobile_network_evolution_history most_intelligent_python_ide_pycharm multimedia_core_system_ims popular_document_db_mongodb program_code_style program_common_logic python_common_code_snippet python_experience_csv_excel python_html_parse_pyquery python_newbie_mistakes_questions python_regex_re_intro python_spider_pyspider python_spider_scrapy rcs_tech_dev_summary regex_usage_examples resident_life_experience_summary scientific_network_summary selenium_summary smart_speaker_disassemble_summary super_search_regex use_python_write_spider work_job_summary xpath_summary youdao_note_summary
----------------------------------------
BOOKS_SRC_ROOT=/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/books
----------------------------------------
TO_PROCESS_BOOKS=convertToNewCommonTemplate.sh make_life_better_python python_common_code_snippet python_experience_csv_excel python_html_parse_pyquery python_newbie_mistakes_questions python_regex_re_intro python_spider_pyspider python_spider_scrapy rcs_tech_dev_summary regex_usage_examples resident_life_experience_summary scientific_network_summary selenium_summary smart_speaker_disassemble_summary super_search_regex use_python_write_spider work_job_summary xpath_summary youdao_note_summary
----------------------------------------
echo help;
help

但是还要去除特殊的非book的：convertToNewCommonTemplate.sh

本来要加上special，去除掉的

但是不够智能

希望去加上更智能的：符合一定规则的

就像正则的\w+的，不符合的，都剔除

去找找 makefile 或者 shell中是否有正则的写法

dir

https://seisman.github.io/how-to-write-makefile/functions.html#dir

notdir

https://seisman.github.io/how-to-write-makefile/functions.html#notdir

发现可以用：判断是否是dir的方式去实现过滤

suffix

https://seisman.github.io/how-to-write-makefile/functions.html#suffix

或者判断有suffix，则过滤掉

还是判断是否是dir更合适

看错了，dir是返回path的dir部分。。。

不过可以去for循环中匹配出，非-d的部分

不过突然发现之前过来结果，好像不对

好像多出个，在program_common_logic之前的：make_life_better_python

不过再去看看，发现是：

的确之前漏了：make_life_better_python

generated中是没有make_life_better_python

算了，暂时就这样吧，通过-d判断：

if [ -d $(BOOKS_SRC_ROOT)/$$each_item ];

即：

help:
    @# echo help;
    @for each_item in $(TO_PROCESS_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            echo $$each_item; \
        fi; \
    done;

可以忽略掉。

makefile check is folder

bash – Test whether a directory exists inside a makefile – Stack Overflow

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS),)
$(info REAL_ALL_BOOKS=$(REAL_ALL_BOOKS))
$(info ----------------------------------------)

看看能否实现过滤掉非folder的book

Makefile:27: *** unterminated call to function 'foreach': missing ')'.  Stop.

报错。

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS))

结果：错误依旧

继续试试：

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(SUB_BOOKS)],$(SUB_BOOKS)))

写错成SUB_BOOKS，改为：

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook)))

可以运行，但没过滤掉convertToNewCommonTemplate.sh

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))

结果：问题依旧。

test if directory exists in a makefile · GitHub

Makefile: Test directories for existence and take first one found – Stack Overflow

试了试：

 crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls -d ../../books
../../books
 crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls -d ../../books -d
ls: -d: No such file or directory
../../books
 ✘ crifan@licrifandeMacBook-Pro  ~/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess   master ●  ls --help           
ls: illegal option -- -
usage: ls [-@ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1%] [file ...]

ls的-d，不可以？

ls only folder

directory – Listing only directories using ls in Bash? – Stack Overflow

●  ls -l ../../books | grep "^d"
drwxr-xr-x  11 crifan  staff  352  1 16 23:43 5g_message_rcs_tech_summary
drwxr-xr-x  11 crifan  staff  352  1 16 23:45 5g_tech_summary
...

但是前面多余了，不方便处理只保留book的name

  find ../../books -maxdepth 1 -type d
../../books
../../books/computer_tech_summary
../../books/mobile_app_summary
../../books/editor_ide_summary

也不是

shell – Listing only directories in UNIX – Stack Overflow

shell – list only folders (ls -d) relative to parent in bash – Stack Overflow

Mac ls only folder

utilities – Is there any option with ‘ls’ command that I see only the directories? – Unix & Linux Stack Exchange

 ls -ld ../../books           
drwxr-xr-x  62 crifan  staff  1984  1 16 11:58 ../../books

ls parameter

ls Command With All Its Parameter In Linux | Geekstrick

ls man page

       -d, --directory
              list directories themselves, not their contents


       -D, --dired
              generate output designed for Emacs' dired mode

ls command in Linux/Unix | list files/directories

ls -d
list directories - with ' */'

试试：

ls -d */

结果：不行。

makefile ifeq

makefile中的条件判断ifeq、ifneq、ifdef_nyist327的专栏-CSDN博客

Conditional Example (GNU make)

makefile inside for check folder

how to prevent “directory already exists error” in a makefile when using mkdir – Stack Overflow

REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [test -d $(BOOKS_SRC_ROOT)/$(singleBook)],,$(singleBook)))

How can I check if directory exists in a makefile

【总结】

最后代码是：

# Function: Batch make for all gitbooks
# Author: Crifan Li
# Update: 20210117
# Latest: https://github.com/crifan/gitbook_template/blob/master/common/Makefile
# 
# [Note]
# 1. this makefile should be located common folder:
# /Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/Makefile
# /Users/crifan/dev/dev_root/gitbook/gitbook_src_root/common


BATCH_PROCESS_IGNORE_FILE=batch_process_ignore_books.txt
$(info BATCH_PROCESS_IGNORE_FILE=$(BATCH_PROCESS_IGNORE_FILE))
$(info ----------------------------------------)


BATCH_PROCESS_IGNORE_BOOKS := $(shell cat $(BATCH_PROCESS_IGNORE_FILE))
$(info BATCH_PROCESS_IGNORE_BOOKS=$(BATCH_PROCESS_IGNORE_BOOKS))
$(info ----------------------------------------)


SUB_BOOKS=$(shell ls ../../books)
$(info SUB_BOOKS=$(SUB_BOOKS))
$(info ----------------------------------------)


BOOKS_SRC_ROOT=$(shell cd ../../books && pwd)
$(info BOOKS_SRC_ROOT=$(BOOKS_SRC_ROOT))
$(info ----------------------------------------)


# # REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))
# # REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [-f $(BOOKS_SRC_ROOT)/$(singleBook)],,$(singleBook)))
# REAL_ALL_BOOKS = $(foreach singleBook,$(SUB_BOOKS),$(if [test -d $(BOOKS_SRC_ROOT)/$(singleBook)],$(singleBook),))
# $(info REAL_ALL_BOOKS=$(REAL_ALL_BOOKS))
# $(info ----------------------------------------)


TO_PROCESS_BOOKS := $(filter-out $(BATCH_PROCESS_IGNORE_BOOKS), $(SUB_BOOKS))
$(info TO_PROCESS_BOOKS=$(TO_PROCESS_BOOKS))
$(info ----------------------------------------)


# help:
#     @# echo help;
#     @for each_item in $(TO_PROCESS_BOOKS); \
#     do \
#         if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
#             echo $$each_item; \
#         fi; \
#     done;


# Batch make for all gitbooks
help debug_dir init sync_content pull clean_all website pdf epub mobi all upload commit deploy:
    @echo "Current path="`pwd`;
    @echo "LS_OUTPUT="$(TO_PROCESS_BOOKS);
    @echo "BOOKS_SRC_ROOT="$(BOOKS_SRC_ROOT);
    @for each_item in $(TO_PROCESS_BOOKS); \
    do \
        if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \
            cd $(BOOKS_SRC_ROOT)/$$each_item; \
            # echo `pwd`; \
            if [ -f Makefile ]; then \
                # echo $$each_item; \
                make $@ || exit "$$?"; \
            fi; \
            cd ..; \
        fi; \
    done;

可以实现：

通过设置：

/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/common/batchProcess/batch_process_ignore_books.txt

5g_message_rcs_tech_summary
5g_tech_summary
all_age_sports_badminton
android_app_security_crack
android_automation_uiautomator2
api_tool_postman
app_capture_package_tool_charles
apple_develop_summary
automobile_sales_summary
best_editor_vscode
china_suitable_living_suzhou
common_logic_hardware_embedded
computer_tech_summary
crack_assistant_xposed_framework
crawl_your_data_spider_technology
desktop_app_framework_electron
doc_format_markdown
ebook_system_gitbook
editor_ide_summary
gitbook_demo
good_automation_tool_makefile
html_parse_tool_beautifulsoup
http_restful_api
http_summary
ic_chip_industry_chain_summary
improve_work_efficiency
industrial_automation_plc
industrial_control_security_overview
information_security_overview
ios_automation_facebook_wda
learn_tech_method_experience
mobile_app_summary
mobile_automation_overview
mobile_network_evolution_history
most_intelligent_python_ide_pycharm
multimedia_core_system_ims
popular_document_db_mongodb
program_code_style
program_common_logic

然后批量处理是，忽略掉这些book

另外其中的：

if [ -d $(BOOKS_SRC_ROOT)/$$each_item ]; then \

通过仅仅处理（正常book都是）目录

实现过滤掉：

/Users/crifan/dev/dev_root/gitbook/GitbookTemplate/gitbook_template/books/

下面的非book内容

比如此处的：

convertToNewCommonTemplate.sh

防止误处理非book

转载请注明：在路上 » 【已解决】批量处理gitbook的脚本时忽略掉已处理的book

Post Views: 1,107

【已解决】批量处理gitbook的脚本时忽略掉已处理的book

与本文相关的文章

Hi，您需要填写昵称和邮箱！