cURL is a tool to connect to a remote server and load data from it while schemes like HTTP, HTTPS, FTP, gopher, telnet, DICT, FILE, LDAP and more are supported for the request URI.
PHP has built-in support by providing its users a layer upon the underlying libcurl library. Here is an example how cURL is used in PHP:
$ch = curl_init("http://example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
The option CURLOPT_RETURNTRANSFER advices PHP to return the result after executing curl_exec() on success and FALSE else. If this option is ommitted it would return TRUE instead of the result.
At some point you need to follow a location. This is the case if a server you are connecting to is replying with a location redirect. In a HTTP response the server would reply with a 301 or 302 status code and the HTTP header Location pointing to the new URI. In the code the option CURLOPT_FOLLOWLOCATION needs to be set to allow libcurl to follow the redirect.
$ch = curl_init("http://example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
curl_close($ch);
However if your web server has safe_mode activated or open_basedir set then CURLOPT_FOLLOWLOCATION won’t have any effect. The below warning will appear and libcurl won’t follow the new location.
Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION
cannot be activated when in safe_mode or an open_basedir is set in ...
This blog post shows you a workaround by changing the server configuration and putting the loader script to a directory defined in open_basedir. Unfortunately many websites are hosted on a shared host. Hence most people can’t just alter something in a configuration file but rather need a user-space solution.
One solution is to follow redirects manually by examining the server response and send the request to the new location again. The next sample code does exactly this. The function curl_exec_follow is passed two arguments, one is the cURL handler and the second the maximum amount of allowed redirects. If a server response contains a redirect location the script also checks if it’s a URL or a relative path to the resource.
function curl_exec_follow($ch, &$maxredirect = null) {
// we emulate a browser here since some websites detect
// us as a bot and don't let us do our job
$user_agent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)".
" Gecko/20041107 Firefox/1.0";
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent );
$mr = $maxredirect === null ? 5 : intval($maxredirect);
if (ini_get('open_basedir') == '' && ini_get('safe_mode' == 'Off')) {
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $mr > 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, $mr);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
} else {
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
if ($mr > 0)
{
$original_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$newurl = $original_url;
$rch = curl_copy_handle($ch);
curl_setopt($rch, CURLOPT_HEADER, true);
curl_setopt($rch, CURLOPT_NOBODY, true);
curl_setopt($rch, CURLOPT_FORBID_REUSE, false);
do
{
curl_setopt($rch, CURLOPT_URL, $newurl);
$header = curl_exec($rch);
if (curl_errno($rch)) {
$code = 0;
} else {
$code = curl_getinfo($rch, CURLINFO_HTTP_CODE);
if ($code == 301 || $code == 302) {
preg_match('/Location:(.*?)\n/', $header, $matches);
$newurl = trim(array_pop($matches));
// if no scheme is present then the new url is a
// relative path and thus needs some extra care
if(!preg_match("/^https?:/i", $newurl)){
$newurl = $original_url . $newurl;
}
} else {
$code = 0;
}
}
} while ($code && --$mr);
curl_close($rch);
if (!$mr)
{
if ($maxredirect === null)
trigger_error('Too many redirects.', E_USER_WARNING);
else
$maxredirect = 0;
return false;
}
curl_setopt($ch, CURLOPT_URL, $newurl);
}
}
return curl_exec($ch);
}
This function is used in place of curl_exec() and no extra user privileges are required compared to Olaf’s workaround in the above linked blog post. Here’s how curl_exec_follow() is used:
$ch = curl_init("http://example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec_follow($ch);
curl_close($ch);
Hopefully this helped you.
Syntax error in
“curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $mr > 0);”
“if ($mr > 0)”
Thanks. It looks like something went wrong when I published the code. ‘>’ turned into >
Thank you! This helped a ton.
I can’t make this work with on a server that has safe_mode on and register_globals off. I’m trying to make a curl login script that would compare if the login attempt was successful or not with preg_match. Seems that PHP and CURL are making things very hard. Eventually all the hosts will upgrade to php 5.3.19 and the lastest libcurl 7.28.1.
Please allow PHP to report error messages as follows
You have to put that before the curl_exec_follow(…) usage. If no error shows up add some debugging messages and track down the place things go wrong. Be also aware of the cURL options which maybe need to be adapted in your case.
I found it useful to mimic a browser by adding before the first IF
curl_setopt($ch, CURLOPT_USERAGENT, ‘Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0′);
By doing this, I could access a website that was otherwise seeing me as malware and was therefore denying me.
Yes, I also have the impression, that emulating a browser makes life easier and avoids additional debugging. I’m adding that line. Thanks for your report.
The code window on this page is too narrow for the whole CURLOPT_USERAGENT line to show…
I’m aware of that. I didn’t split up the string because I copied while not testing it. I just wanted to avoid any stupid error being introduced. I’ve done it now but it’s still untested. It looks error-free but could you test it to be sure that everything’s fine?
It seems to be working well, although in the original string, there is a space between “…rv:1.7.5)” and “Gecko…”
I don’t know if that really matters. I tested it with Firefox.
I’m getting 502 error and events in one request event published many times even if I set second param of curl_exec_follow function. I’m trying to use this with Google Calendar API.
Please try the modified version below and tell me if it worked. I wrapped the curl_exec() function by an output buffer.
Little update. I made a tiny mistake in the modified script. I’ve corrected it so use the snippet from the previous comment again.
Many thanks. Unfortunatelly I’m still getting 502. Its because of host. On another server your function works almost fine – task is added to google calendar but often many times in one request.
Try to compare the configuration of both hosts you were trying it (with the phpinfo() function). If you wish to get it run on the one server but aren’t able to do so, I can spend some spare time but it will be beyond of free support though.
Thanks, but open_base dir is set. Cant use ini_set to change this cause it is PHP_INI_SYSTEM value.
The above script in fact is meant to be used on hosts where open_basedir is set which I am pretty sure has nothing to do with your issue. You would get this kind of message instead:
502 means that cURL has received an invalid response from the upstream server. This can have various reasons e.g. a too big file being sent when using the CURLOPT_POST option. I’d say your issue is more cURL-related than php-related.
Hello,
I encountered the same problem and I would like to use your solution. However, I have no idea where to put the code. Can you give me a hint?
This highly depends on your cms/theme or what else system you use.
The function
can be put anywhere in your php code, even after the actual usage (at least in the same source file).
Then put the following four lines where you want to curl data from the internet:
If you put the function in one source file and the four lines into another be sure you include the source file with the function in the source file where you call it. To include a source file use one of these php functions:
Ok…
I use Joomla. This is way beyond my knowledge. I guess I must try to persuade the web hoster to adjust the settings on his side…
Thanks for your time!
I’m trying to login to a web page using cURL but after searching solutions i keep getting
Anybody knows why this keep happening? I’m using
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata) to login, so maybe thats the problem.
Thanks!