How to use curl to get a GET request exactly same as using Chrome?
CurlBrowserHttprequestUser AgentSimulateCurl Problem Overview
I have a web api http://something.com/api and I want to use GET to get the response body.
This is my command:
curl "http://something.com/api"
Of course, it fails and gives an error message.
When I use Chrome and input the above url, everythings correct. However I do the same things with Firefox, the url gives me the same error message. I try to repeat the action with Chrome extension DHC, the request gives correct response again. After some searching, I believe that the curl option --user-agent makes a difference. What is the correct way to set the user agent to Chrome? Or this is not the point, the problem comes from other fields? Thank you very much.
Curl Solutions
Solution 1 - Curl
If you need to set the user header string in the curl request, you can use the -H option to set user agent like:
curl -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36" http://stackoverflow.com/questions/28760694/how-to-use-curl-to-get-a-get-request-exactly-same-as-using-chrome
Updated user-agent form newest Chrome at 02-22-2021
Using a proxy tool like Charles Proxy really helps make short work of something like what you are asking. Here is what I do, using this SO page as an example (as of July 2015 using Charles version 3.10):
- Get Charles Proxy running
- Make web request using browser
- Find desired request in Charles Proxy
- Right click on request in Charles Proxy
- Select 'Copy cURL Request'
You now have a cURL request you can run in a terminal that will mirror the request your browser made. Here is what my request to this page looked like (with the cookie header removed):
curl -H "Host: stackoverflow.com" -H "Cache-Control: max-age=0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36" -H "HTTPS: 1" -H "DNT: 1" -H "Referer: https://www.google.com/" -H "Accept-Language: en-US,en;q=0.8,en-GB;q=0.6,es;q=0.4" -H "If-Modified-Since: Thu, 23 Jul 2015 20:31:28 GMT" --compressed http://stackoverflow.com/questions/28760694/how-to-use-curl-to-get-a-get-request-exactly-same-as-using-chrome
Solution 2 - Curl
Open Chrome Developer Tools, go to Network tab, make your request (you may need to check "Preserve Log" if the page refreshes). Find the request on the left, right-click, "Copy as cURL".
Solution 3 - Curl
Check the HTTP headers that chrome is sending with the request (Using browser extension or proxy) then try sending the same headers with CURL - Possibly one at a time till you figure out which header(s) makes the request work.
curl -A [user-agent] -H [headers] "http://something.com/api"