[研究]wget 1.21.1 網站下載軟體安裝、試用 (Rocky Linux 9.1)

[研究]wget 1.21.1 網站下載軟體安裝、試用 (Rocky Linux 9.1)

2023-03-21

Wget - 維基百科，自由的百科全書
https://zh.wikipedia.org/wiki/Wget

GNU Wget（常簡稱為Wget）是一個在網路上進行下載的簡單而強大的自由軟體，其本身也是GNU計劃的一部分。它的名字是「World Wide Web」和「Get」的結合，同時也隱含了軟體的主要功能。目前它支援通過HTTP、HTTPS，以及FTP這三個最常見的TCP/IP協定協定下載。

********************************************************************************

安裝

[user1@localhost ~]$ sudo yum -y install wget We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. [sudo] password for user1: Rocky Linux 9 - BaseOS 950 kB/s | 1.8 MB 00:01 Rocky Linux 9 - AppStream 2.8 MB/s | 6.6 MB 00:02 Rocky Linux 9 - Extras 7.7 kB/s | 8.5 kB 00:01 Package wget-1.21.1-7.el9.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! [user1@localhost ~]$

執行

[user1@localhost ~]$ wget wget: missing URL Usage: wget [OPTION]... [URL]... Try `wget --help' for more options. [user1@localhost ~]$

說明

[user1@localhost ~]$ wget --help GNU Wget 1.21.1, a non-interactive network retriever. Usage: wget [OPTION]... [URL]... Mandatory arguments to long options are mandatory for short options too. Startup: -V, --version display the version of Wget and exit -h, --help print this help -b, --background go to background after startup -e, --execute=COMMAND execute a `.wgetrc'-style command Logging and input file: -o, --output-file=FILE log messages to FILE -a, --append-output=FILE append messages to FILE -d, --debug print lots of debugging information -q, --quiet quiet (no output) -v, --verbose be verbose (this is the default) -nv, --no-verbose turn off verboseness, without being quiet --report-speed=TYPE output bandwidth as TYPE. TYPE can be bits -i, --input-file=FILE download URLs found in local or external FILE -F, --force-html treat input file as HTML -B, --base=URL resolves HTML input-file links (-i -F) relative to URL --config=FILE specify config file to use --no-config do not read any config file --rejected-log=FILE log reasons for URL rejection to FILE Download: -t, --tries=NUMBER set number of retries to NUMBER (0 unlimits) --retry-connrefused retry even if connection is refused --retry-on-http-error=ERRORS comma-separated list of HTTP errors to retry -O, --output-document=FILE write documents to FILE -nc, --no-clobber skip downloads that would download to existing files (overwriting them) --no-netrc don't try to obtain credentials from .netrc -c, --continue resume getting a partially-downloaded file --start-pos=OFFSET start downloading from zero-based position OFFSET --progress=TYPE select progress gauge type --show-progress display the progress bar in any verbosity mode -N, --timestamping don't re-retrieve files unless newer than local --no-if-modified-since don't use conditional if-modified-since get requests in timestamping mode --no-use-server-timestamps don't set the local file's timestamp by the one on the server -S, --server-response print server response --spider don't download anything -T, --timeout=SECONDS set all timeout values to SECONDS --dns-timeout=SECS set the DNS lookup timeout to SECS --connect-timeout=SECS set the connect timeout to SECS --read-timeout=SECS set the read timeout to SECS -w, --wait=SECONDS wait SECONDS between retrievals (applies if more then 1 URL is to be retrieved) --waitretry=SECONDS wait 1..SECONDS between retries of a retrieval (applies if more then 1 URL is to be retrieved) --random-wait wait from 0.5*WAIT...1.5*WAIT secs between retrievals (applies if more then 1 URL is to be retrieved) --no-proxy explicitly turn off proxy -Q, --quota=NUMBER set retrieval quota to NUMBER --bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host --limit-rate=RATE limit download rate to RATE --no-dns-cache disable caching DNS lookups --restrict-file-names=OS restrict chars in file names to ones OS allows --ignore-case ignore case when matching files/directories -4, --inet4-only connect only to IPv4 addresses -6, --inet6-only connect only to IPv6 addresses --prefer-family=FAMILY connect first to addresses of specified family, one of IPv6, IPv4, or none --user=USER set both ftp and http user to USER --password=PASS set both ftp and http password to PASS --ask-password prompt for passwords --use-askpass=COMMAND specify credential handler for requesting username and password. If no COMMAND is specified the WGET_ASKPASS or the SSH_ASKPASS environment variable is used. --no-iri turn off IRI support --local-encoding=ENC use ENC as the local encoding for IRIs --remote-encoding=ENC use ENC as the default remote encoding --unlink remove file before clobber --xattr turn on storage of metadata in extended file attributes Directories: -nd, --no-directories don't create directories -x, --force-directories force creation of directories -nH, --no-host-directories don't create host directories --protocol-directories use protocol name in directories -P, --directory-prefix=PREFIX save files to PREFIX/.. --cut-dirs=NUMBER ignore NUMBER remote directory components HTTP options: --http-user=USER set http user to USER --http-password=PASS set http password to PASS --no-cache disallow server-cached data --default-page=NAME change the default page name (normally this is 'index.html'.) -E, --adjust-extension save HTML/CSS documents with proper extensions --ignore-length ignore 'Content-Length' header field --header=STRING insert STRING among the headers --compression=TYPE choose compression, one of auto, gzip and none. (default: none) --max-redirect maximum redirections allowed per page --proxy-user=USER set USER as proxy username --proxy-password=PASS set PASS as proxy password --referer=URL include 'Referer: URL' header in HTTP request --save-headers save the HTTP headers to file -U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION --no-http-keep-alive disable HTTP keep-alive (persistent connections) --no-cookies don't use cookies --load-cookies=FILE load cookies from FILE before session --save-cookies=FILE save cookies to FILE after session --keep-session-cookies load and save session (non-permanent) cookies --post-data=STRING use the POST method; send STRING as the data --post-file=FILE use the POST method; send contents of FILE --method=HTTPMethod use method "HTTPMethod" in the request --body-data=STRING send STRING as data. --method MUST be set --body-file=FILE send contents of FILE. --method MUST be set --content-disposition honor the Content-Disposition header when choosing local file names (EXPERIMENTAL) --content-on-error output the received content on server errors --auth-no-challenge send Basic HTTP authentication information without first waiting for the server's challenge HTTPS (SSL/TLS) options: --secure-protocol=PR choose secure protocol, one of auto, SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 and PFS --https-only only follow secure HTTPS links --no-check-certificate don't validate the server's certificate --certificate=FILE client certificate file --certificate-type=TYPE client certificate type, PEM or DER --private-key=FILE private key file --private-key-type=TYPE private key type, PEM or DER --ca-certificate=FILE file with the bundle of CAs --ca-directory=DIR directory where hash list of CAs is stored --crl-file=FILE file with bundle of CRLs --pinnedpubkey=FILE/HASHES Public key (PEM/DER) file, or any number of base64 encoded sha256 hashes preceded by 'sha256//' and separated by ';', to verify peer against --ciphers=STR Set the priority string (GnuTLS) or cipher list string (OpenSSL) directly. Use with care. This option overrides --secure-protocol. The format and syntax of this string depend on the specific SSL/TLS engine. HSTS options: --no-hsts disable HSTS --hsts-file path of HSTS database (will override default) FTP options: --ftp-user=USER set ftp user to USER --ftp-password=PASS set ftp password to PASS --no-remove-listing don't remove '.listing' files --no-glob turn off FTP file name globbing --no-passive-ftp disable the "passive" transfer mode --preserve-permissions preserve remote file permissions --retr-symlinks when recursing, get linked-to files (not dir) FTPS options: --ftps-implicit use implicit FTPS (default port is 990) --ftps-resume-ssl resume the SSL/TLS session started in the control connection when opening a data connection --ftps-clear-data-connection cipher the control channel only; all the data will be in plaintext --ftps-fallback-to-ftp fall back to FTP if FTPS is not supported in the target server WARC options: --warc-file=FILENAME save request/response data to a .warc.gz file --warc-header=STRING insert STRING into the warcinfo record --warc-max-size=NUMBER set maximum size of WARC files to NUMBER --warc-cdx write CDX index files --warc-dedup=FILENAME do not store records listed in this CDX file --no-warc-compression do not compress WARC files with GZIP --no-warc-digests do not calculate SHA1 digests --no-warc-keep-log do not store the log file in a WARC record --warc-tempdir=DIRECTORY location for temporary files created by the WARC writer Recursive download: -r, --recursive specify recursive download -l, --level=NUMBER maximum recursion depth (inf or 0 for infinite) --delete-after delete files locally after downloading them -k, --convert-links make links in downloaded HTML or CSS point to local files --convert-file-only convert the file part of the URLs only (usually known as the basename) --backups=N before writing file X, rotate up to N backup files -K, --backup-converted before converting file X, back up as X.orig -m, --mirror shortcut for -N -r -l inf --no-remove-listing -p, --page-requisites get all images, etc. needed to display HTML page --strict-comments turn on strict (SGML) handling of HTML comments Recursive accept/reject: -A, --accept=LIST comma-separated list of accepted extensions -R, --reject=LIST comma-separated list of rejected extensions --accept-regex=REGEX regex matching accepted URLs --reject-regex=REGEX regex matching rejected URLs --regex-type=TYPE regex type (posix|pcre) -D, --domains=LIST comma-separated list of accepted domains --exclude-domains=LIST comma-separated list of rejected domains --follow-ftp follow FTP links from HTML documents --follow-tags=LIST comma-separated list of followed HTML tags --ignore-tags=LIST comma-separated list of ignored HTML tags -H, --span-hosts go to foreign hosts when recursive -L, --relative follow relative links only -I, --include-directories=LIST list of allowed directories --trust-server-names use the name specified by the redirection URL's last component -X, --exclude-directories=LIST list of excluded directories -np, --no-parent don't ascend to the parent directory Email bug reports, questions, discussions to <bug-wget@gnu.org> and/or open issues at https://savannah.gnu.org/bugs/?func=additem&group=wget. [user1@localhost ~]$

測試 (網站名稱敝人換了，不是真實測試的)

[user1@localhost ~]$ mkdir WWW [user1@localhost ~]$ cd WWW [user1@localhost WWW]$ wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains www.xxx.idv.tw --no-parent https://www.xxx.idv.tw/ Both --no-clobber and --convert-links were specified, only --convert-links will be used. --2023-03-24 09:12:18-- https://www.xxx.idv.tw/ Resolving www.xxx.idv.tw (www.xxx.idv.tw)... xxx.xxx.xxx.xxx Connecting to www.xxx.idv.tw (www.xxx.idv.tw)|xxx.xxx.xxx.xxx|:443... connected. ERROR: The certificate of ‘www.xxx.idv.tw’ is not trusted. ERROR: The certificate of ‘www.xxx.idv.tw’ doesn't have a known issuer. Converted links in 0 files in 0 seconds. [user1@localhost WWW]$

之前 Wget for Windows 1.21.3 測試相同網站時，不會有無法信任問題。

增加參數 --no-check-certificate 

測試 (網站名稱敝人換了，不是真實測試的)，AM 09:19 開始測試，AM 09:42完成，約23分鐘，傳輸速度3.8 MB/s (頻寬多人共用 100 Mbps)。

Converting links in www.xxx.idv.tw/Content/Custom.css... nothing to do. Converted links in 780 files in 0.6 seconds. [user1@localhost WWW]$ date Fri Mar 24 09:42:03 AM CST 2023 [user1@localhost WWW]$

(下圖)wget 1.21.1在 Rocky Linux 9.1上支援繁體中文檔案名稱；

之前 wget 1.21.3 在MS-Windows上測試會有下面錯誤

Incomplete or invalid multibyte sequence encountered

********************************************************************************

推薦參數

wget  --no-check-certificate --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains www.xxx.idv.tw --no-parent https://www.xxx.idv.tw/

常用解釋：

--mirror ：使 wget 下載整個網站，包括子目錄和子網頁。

--convert-links ：將下載的文件中的鏈接轉換為本地文件的鏈接。

--adjust-extension ：將下載的文件的擴展名更改為與其在網站上的 URL 對應的擴展名。

--page-requisites ：下載頁面的所有必要元素，例如圖像和樣式表。

--no-parent ：不遵循父級鏈接，即不下載超出指定網站的連接。

********************************************************************************

(完)

相關

[研究]wget 1.21.1 網站下載軟體安裝、測試 (Rocky Linux 9.1)
https://shaurong.blogspot.com/2023/03/wget-1211-rocky-linux-91.html

[研究]Wget for Windows 1.21.3試用
https://shaurong.blogspot.com/2023/03/wget-for-windows-1213.html