Linux系列之学会使用CURL命令
Linux系列之学会使用CURL命令
curl命令是一个利用URL规则在shell终端命令行下工作的文件传输工具;curl命令作为一款强力工具,curl支持包括HTTP、HTTPS、ftp等众多协议,还支持POST、cookies、认证、从指定偏移处下载部分文件、用户代理字符串、限速、文件大小、进度条等特征;做网页处理流程和数据检索自动化。
语法:
语法格式:curl [参数] [网址]
一. 查看帮助
1.1 查看命令参数
curl的参数很多,可以用命令
curl --help查看:
[root@localhost ~]# curl --help Usage: curl [options...] <url> Options: (H) means HTTP/HTTPS only, (F) means FTP only --anyauth Pick "any" authentication method (H) -a, --append Append to target file when uploading (F/SFTP) --basic Use HTTP Basic Authentication (H) --cacert FILE CA certificate to verify peer against (SSL) --capath DIR CA directory to verify peer against (SSL) -E, --cert CERT[:PASSWD] Client certificate file and password (SSL) --cert-type TYPE Certificate file type (DER/PEM/ENG) (SSL) --ciphers LIST SSL ciphers to use (SSL) --compressed Request compressed response (using deflate or gzip) -K, --config FILE Specify which config file to read --connect-timeout SECONDS Maximum time allowed for connection -C, --continue-at OFFSET Resumed transfer offset -b, --cookie STRING/FILE String or file to read cookies from (H) -c, --cookie-jar FILE Write cookies to this file after operation (H) --create-dirs Create necessary local directory hierarchy --crlf Convert LF to CRLF in upload --crlfile FILE Get a CRL list in PEM format from the given file -d, --data DATA HTTP POST data (H) --data-ascii DATA HTTP POST ASCII data (H) --data-binary DATA HTTP POST binary data (H) --data-urlencode DATA HTTP POST data url encoded (H) --delegation STRING GSS-API delegation permission --digest Use HTTP Digest Authentication (H) --disable-eprt Inhibit using EPRT or LPRT (F) --disable-epsv Inhibit using EPSV (F) -D, --dump-header FILE Write the headers to this file --egd-file FILE EGD socket path for random data (SSL) --engine ENGINGE Crypto engine (SSL). "--engine list" for list -f, --fail Fail silently (no output at all) on HTTP errors (H) -F, --form CONTENT Specify HTTP multipart POST data (H) --form-string STRING Specify HTTP multipart POST data (H) --ftp-account DATA Account data string (F) --ftp-alternative-to-user COMMAND String to replace "USER [name]" (F) --ftp-create-dirs Create the remote dirs if not present (F) --ftp-method [MULTICWD/NOCWD/SINGLECWD] Control CWD usage (F) --ftp-pasv Use PASV/EPSV instead of PORT (F) -P, --ftp-port ADR Use PORT with given address instead of PASV (F) --ftp-skip-pasv-ip Skip the IP address for PASV (F) --ftp-pret Send PRET before PASV (for drftpd) (F) --ftp-ssl-ccc Send CCC after authenticating (F) --ftp-ssl-ccc-mode ACTIVE/PASSIVE Set CCC mode (F) --ftp-ssl-control Require SSL/TLS for ftp login, clear for transfer (F) -G, --get Send the -d data with a HTTP GET (H) -g, --globoff Disable URL sequences and ranges using {} and [] -H, --header LINE Custom header to pass to server (H) -I, --head Show document info only -h, --help This help text --hostpubmd5 MD5 Hex encoded MD5 string of the host public key. (SSH) -0, --http1.0 Use HTTP 1.0 (H) --ignore-content-length Ignore the HTTP Content-Length header -i, --include Include protocol headers in the output (H/F) -k, --insecure Allow connections to SSL sites without certs (H) --interface INTERFACE Specify network interface/address to use -4, --ipv4 Resolve name to IPv4 address -6, --ipv6 Resolve name to IPv6 address -j, --junk-session-cookies Ignore session cookies read from file (H) --keepalive-time SECONDS Interval between keepalive probes --key KEY Private key file name (SSL/SSH) --key-type TYPE Private key file type (DER/PEM/ENG) (SSL) --krb LEVEL Enable Kerberos with specified security level (F) --libcurl FILE Dump libcurl equivalent code of this command line --limit-rate RATE Limit transfer speed to this rate -l, --list-only List only names of an FTP directory (F) --local-port RANGE Force use of these local port numbers -L, --location Follow redirects (H) --location-trusted like --location and send auth to other hosts (H) -M, --manual Display the full manual --mail-from FROM Mail from this address --mail-rcpt TO Mail to this receiver(s) --mail-auth AUTH Originator address of the original email --max-filesize BYTES Maximum file size to download (H/F) --max-redirs NUM Maximum number of redirects allowed (H) -m, --max-time SECONDS Maximum time allowed for the transfer --metalink Process given URLs as metalink XML file --negotiate Use HTTP Negotiate Authentication (H) -n, --netrc Must read .netrc for user name and password --netrc-optional Use either .netrc or URL; overrides -n --netrc-file FILE Set up the netrc filename to use -N, --no-buffer Disable buffering of the output stream --no-keepalive Disable keepalive use on the connection --no-sessionid Disable SSL session-ID reusing (SSL) --noproxy List of hosts which do not use proxy --ntlm Use HTTP NTLM authentication (H) -o, --output FILE Write output to <file> instead of stdout --pass PASS Pass phrase for the private key (SSL/SSH) --post301 Do not switch to GET after following a 301 redirect (H) --post302 Do not switch to GET after following a 302 redirect (H) --post303 Do not switch to GET after following a 303 redirect (H) -#, --progress-bar Display transfer progress as a progress bar --proto PROTOCOLS Enable/disable specified protocols --proto-redir PROTOCOLS Enable/disable specified protocols on redirect -x, --proxy [PROTOCOL://]HOST[:PORT] Use proxy on given port --proxy-anyauth Pick "any" proxy authentication method (H) --proxy-basic Use Basic authentication on the proxy (H) --proxy-digest Use Digest authentication on the proxy (H) --proxy-negotiate Use Negotiate authentication on the proxy (H) --proxy-ntlm Use NTLM authentication on the proxy (H) -U, --proxy-user USER[:PASSWORD] Proxy user and password --proxy1.0 HOST[:PORT] Use HTTP/1.0 proxy on given port -p, --proxytunnel Operate through a HTTP proxy tunnel (using CONNECT) --pubkey KEY Public key file name (SSH) -Q, --quote CMD Send command(s) to server before transfer (F/SFTP) --random-file FILE File for reading random data from (SSL) -r, --range RANGE Retrieve only the bytes within a range --raw Do HTTP "raw", without any transfer decoding (H) -e, --referer Referer URL (H) -J, --remote-header-name Use the header-provided filename (H) -O, --remote-name Write output to a file named as the remote file --remote-name-all Use the remote file name for all URLs -R, --remote-time Set the remote file's time on the local output -X, --request COMMAND Specify request command to use --resolve HOST:PORT:ADDRESS Force resolve of HOST:PORT to ADDRESS --retry NUM Retry request NUM times if transient problems occur --retry-delay SECONDS When retrying, wait this many seconds between each --retry-max-time SECONDS Retry only within this period -S, --show-error Show error. With -s, make curl show errors when they occur -s, --silent Silent mode. Don't output anything --socks4 HOST[:PORT] SOCKS4 proxy on given host + port --socks4a HOST[:PORT] SOCKS4a proxy on given host + port --socks5 HOST[:PORT] SOCKS5 proxy on given host + port --socks5-hostname HOST[:PORT] SOCKS5 proxy, pass host name to proxy --socks5-gssapi-service NAME SOCKS5 proxy service name for gssapi --socks5-gssapi-nec Compatibility with NEC SOCKS5 server -Y, --speed-limit RATE Stop transfers below speed-limit for 'speed-time' secs -y, --speed-time SECONDS Time for trig speed-limit abort. Defaults to 30 --ssl Try SSL/TLS (FTP, IMAP, POP3, SMTP) --ssl-reqd Require SSL/TLS (FTP, IMAP, POP3, SMTP) -2, --sslv2 Use SSLv2 (SSL) -3, --sslv3 Use SSLv3 (SSL) --ssl-allow-beast Allow security flaw to improve interop (SSL) --stderr FILE Where to redirect stderr. - means stdout --tcp-nodelay Use the TCP_NODELAY option -t, --telnet-option OPT=VAL Set telnet option --tftp-blksize VALUE Set TFTP BLKSIZE option (must be >512) -z, --time-cond TIME Transfer based on a time condition -1, --tlsv1 Use => TLSv1 (SSL) --tlsv1.0 Use TLSv1.0 (SSL) --tlsv1.1 Use TLSv1.1 (SSL) --tlsv1.2 Use TLSv1.2 (SSL) --trace FILE Write a debug trace to the given file --trace-ascii FILE Like --trace but without the hex output --trace-time Add time stamps to trace/verbose output --tr-encoding Request compressed transfer encoding (H) -T, --upload-file FILE Transfer FILE to destination --url URL URL to work with -B, --use-ascii Use ASCII/text transfer -u, --user USER[:PASSWORD] Server user and password --tlsuser USER TLS username --tlspassword STRING TLS password --tlsauthtype STRING TLS authentication type (default SRP) -A, --user-agent STRING User-Agent to send to server (H) -v, --verbose Make the operation more talkative -V, --version Show version number and quit -w, --write-out FORMAT What to output after completion --xattr Store metadata in extended file attributes -q If used as the first parameter disables .curlrc
国内翻译过来的curl文档:curl参数介绍
二. 查询使用手册
2.1 查看详细使用
也可以使用
curl --manual查看基本用法例子:
三. 提取网页
3.1 查看网页源码
语法:
curl uri
[root@localhost ~]# curl www.baidu.com <!DOCTYPE html> <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> & 1ae22 lt;/body> </html>
3.2 显示头信息
curl -i https://www.baidu.com
打印信息:
[root@localhost ~]# curl -i https://www.baiud.com curl: (7) Failed connect to www.baiud.com:443; Connection refused [root@localhost ~]# curl -i https://www.baidu.comHTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform Connection: keep-alive Content-Length: 2443 Content-Type: text/html Date: Mon, 13 Jul 2020 07:22:02 GMT Etag: "588603eb-98b" Last-Modified: Mon, 23 Jan 2017 13:23:55 GMT Pragma: no-cache Server: bfe/1.0.8.18 Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ ...
3.3 显示通信过程
curl -v www.baidu.com
curl -v www.baidu.com* About to connect() to www.baidu.com port 80 (#0) * Trying 182.61.200.7... * Connected to www.baidu.com (182.61.200.7) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: www.baidu.com > Accept: */* > < HTTP/1.1 200 OK < Accept-Ranges: bytes < Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform < Connection: keep-alive < Content-Length: 2381 < Content-Type: text/html < Date: Mon, 13 Jul 2020 07:24:46 GMT < Etag: "588604c8-94d" < Last-Modified: Mon, 23 Jan 2017 13:27:36 GMT < Pragma: no-cache < Server: bfe/1.0.8.18 < Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ < <!DOCTYPE html> ...
3.4 查看议定书
[root@localhost ~]# curl --trace-ascii output.txt https://www.baidu.com
3.5 查看时间
curl --trace-ascii output.txt --trace-time https://www.baidu.com
四. 用curl进行认证
4.1 referer设置
referer参数描述了一个网页的来源,可以按F12调试模式,查看:
访问百度时候,加上referer为csdn:
[root@localhost ~]# curl --referer https://blog.csdn.net https://www.baidu.com
4.2 用户代理设置
用户代理常被用于兼容ie,有时,您会看到获取带有curl的页面不会返回与使用浏览器获取页面时看到的页面相同的页面,这时候可以设置用户代理:
curl www.baidu.com --user-agent "Mozilla/5.0" curl www.baidu.com -A "Mozilla/5.0"
4.3 用户信息校验
比如登录百度,可以用-u参数,接着会提示输入密码
[root@localhost ~]# curl -u admin https://www.baidu.net/ Enter host password for user 'admin':
五. 文件操作
5.1 保存文件
语法:
curl -o filename uri
[root@localhost ~]# curl -o baidu.html www.baidu.com % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2381 100 2381 0 0 22761 0 --:--:-- --:--:-- --:--:-- 22894
5.2 文件下载
语法:
curl uri filename --silent|progress
- –silent:不显示进度条
- –progress:显示进度条
[root@localhost ~]# curl https://avatar.csdnimg.cn/3/5/E/1_meituantech_1564971608.jpg -omeituan.jpg --progress ######################################################################## 100.0%
5.3 上传文件
语法:
$ curl --form upload=@localfilename --form press=OK
6.3 指定cookie文件
也可以用–cookie-jar指定cookie文件
[root@localhost ~]# curl www.baidu.com --cookie-jar cookie_file
指定可下载的最大文件大小,可以用命令–limit-rate[root@localhost ~]# curl www.baidu.com --limit-rate 50k七. curl RESTFul
7.1 RESTFul API
curl支持RESTFul API格式的,所以先补充一些RESTFul API知识:
GET 请求:获取服务器中的对象
相当于SQL的Select命令- GET /blogs:列出所有博客信息
POST 请求:在服务器上新增对象
相当于SQL的Update命令
POST /blogs/ID:新增指定的博客信息 PUT 请求:在服务器上更新对象
相当于SQL的Create命令
PUT /blogs/ID:更新一个博客信息 DELETE 请求:删除服务器中的对象
相当于SQL的Delete命令
DELETE /blogs/ID:删除指定的博客 HEAD 请求:仅仅用于获取对象的基础信息
7.2 POST请求
ps:-H指定请求header,-d指定参数
- CURL POST:
[root@localhost ~]# curl -H "Content-Type:application/json" -X POST localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"123"}'7.3 GET请求
- CURL GET:
[root@localhost ~]# curl -X GET localhost:9200/company/employees/1?pretty=true7.4 DELETE请求
- CURL DELETE:
[root@localhost ~]# curl -X DELETE localhost:9200/company/employees/17.5 PUT请求
- CURL PUT:
curl -H "Content-Type:application/json" -X PUT localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"111"}'curl默认是GET方式请求url的,要调POST等接口,必须加上-X参数指定
八. CURL参考资料
CURL 官网教程:https://curl.haxx.se/docs/httpscripting.html" target=_blank>[/code]
[root@localhost ~]# curl --form upload=@baidu.html --form press=OK url六. Cookie设置
6.1 设置cookie
设置cookie,可以用命令
--cookie6.2 cookie -b -c
-c cookie-file可以保存服务器返回的cookie到文件,-b cookie-file可以使用这个文件作为cookie信息,进行后续的请求。$ curl -c cookies http://example.com $ curl -b cookies http://example.com[root@localhost ~]# curl -c "name=cookiename" www.baidu.com
6.3 指定cookie文件
也可以用–cookie-jar指定cookie文件
[root@localhost ~]# curl www.baidu.com --cookie-jar cookie_file
指定可下载的最大文件大小,可以用命令–limit-rate[root@localhost ~]# curl www.baidu.com --limit-rate 50k七. curl RESTFul
7.1 RESTFul API
curl支持RESTFul API格式的,所以先补充一些RESTFul API知识:
GET 请求:获取服务器中的对象
相当于SQL的Select命令- GET /blogs:列出所有博客信息
POST 请求:在服务器上新增对象
相当于SQL的Update命令
POST /blogs/ID:新增指定的博客信息 PUT 请求:在服务器上更新对象
相当于SQL的Create命令
PUT /blogs/ID:更新一个博客信息 DELETE 请求:删除服务器中的对象
相当于SQL的Delete命令
DELETE /blogs/ID:删除指定的博客 HEAD 请求:仅仅用于获取对象的基础信息
7.2 POST请求
ps:-H指定请求header,-d指定参数
- CURL POST:
[root@localhost ~]# curl -H "Content-Type:application/json" -X POST localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"123"}'7.3 GET请求
- CURL GET:
[root@localhost ~]# curl -X GET localhost:9200/company/employees/1?pretty=true7.4 DELETE请求
- CURL DELETE:
[root@localhost ~]# curl -X DELETE localhost:9200/company/employees/17.5 PUT请求
- CURL PUT:
curl -H "Content-Type:application/json" -X PUT localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"111"}'curl默认是GET方式请求url的,要调POST等接口,必须加上-X参数指定
八. CURL参考资料
CURL 官网教程:[url=https://curl.haxx.se/docs/httpscripting.html]https://curl.haxx.se/docs/httpscripting.html
阮一峰老师教程:[url=https://www.geek-share.com/detail/2687421922.html]https://www.geek-share.com/detail/2687421922.html
- 我使用过的Linux命令之curl - 强大的网络传输工具
- Linux中使用curl命令访问https站点4种常见错误和解决方法
- linux命令行下使用curl命令查看自己机器的外网ip
- Linux下curl命令的使用
- Linux下使用curl命令发送GET请求时多个参数截断的问题
- linux中curl命令使用小结
- linux下curl命令的使用
- Linux 的 curl命令的使用
- Linux系统中的curl命令使用教程
- 使用ps命令输出进程列表--用Enki学Linux系列(17)
- Linux中curl命令和wget命令的使用介绍与比较
- lftp命令使用 操作系统(Windows、Linux) PHP学会网 php培训网 PHP暑期培训 PHP寒假培训 PHP假期培训 - powered by phpwhy.com
- Linux中使用curl命令访问https站点4种常见错误和解决方法
- grep 命令系列:如何在 Linux/UNIX 中使用 grep 命令
- Linux Curl常用命令使用【转】
- grep 命令系列:如何在 Linux/UNIX 中使用 grep 命令
- 使用touch命令改变文件修改时间--用Enki学Linux系列(11)
- Linux curl命令使用
- 7z介绍、7z命令在linux的安装和使用、7z移植到嵌入式linux的系列专题汇总
- 在Windows系列中的Linux系统下“Which”命令的使用方法介绍