I tried the functions file_get_contents and curl - it does not work with the adidas.ru site.

Previously, everything worked fine through cURL, but now it does not work. Did they block my IP?

function curl($url, $postdata) { $uagent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1"; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERAGENT, $uagent); curl_setopt($ch, CURLOPT_TIMEOUT, 60); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata); curl_setopt($ch, CURLOPT_REFERER, ""); curl_setopt($ch, CURLOPT_HEADER, 1); $content = curl_exec( $ch ); curl_close( $ch ); return $content; } echo curl('http://www.adidas.ru/search','q=');

Before this, the code worked for half a year without problems. If instead of adidas write reebok, everything works.

Indeed HTTP / 1.0 403 Forbidden Error But why? If you go through the browser, it opens without problems. How can I get the page code?

  • Not clear ... Do you want us to offer you a ready-made solution? This is not how it works. Give your code and describe what you think is not working. - GrayHoax
  • Write what specifically does not work for you? No one will write for you! But psychics tell me that it is possible that the goods on the page are loaded with ajax request - LANSELOT
  • HTTP/1.0 403 Forbidden doesn't say anything? - Roman Grinyov

1 answer 1

The site needs some cookies. for example, after adding something similar to your code (I took the value from firebug in the browser)

 curl_setopt($ch, CURLOPT_HTTPHEADER, array( 'Cookie:dwac_bfgL2iaagIFBaaaadT2cF99ObL=Q_oa325yGUdRU2S0mnm4GKEv5CWwAYwro3k%3D|demandaaglru|||RUB|false|Etc%2FGMT%2D3|true; cqcid=acjUV4UWLDS4F6B9VO1aKEWWnp; sid=Q_oa325yGUdRU2S0mnm4GKEv5CWwAYwro3k; pagecontext_geo_country=UA; dwanonymous_cf44e4f9b635e8943650a805c7ab4242=acjUV4UWLDS4F6B9VO1aKEWWnp; pagecontext_logged_in=""; pagecontext_customer_id=acjUV4UWLDS4F6B9VO1aKEWWnp; lastVisitedProducts=""; dwsid=11SWv08zZ1_iD5_vXC3QljWBaEWRQPIzECWPaO-djQKhtX4nhs_Xg1s5aMMYJr90F7NV2uklpztal2y8YsWsjQ==; LastPageUpdateTimestamp=1464124623709; optimizelyEndUserId=oeu1464124623859r0.30954379713645497; optimizelySegments=%7B%22299577001%22%3A%22direct%22%2C%22299649707%22%3A%22none%22%2C%22299754438%22%3A%22false%22%2C%22299765888%22%3A%22opera%22%2C%22655644525%22%3A%22true%22%7D; optimizelyBuckets=%7B%225885940669%22%3A%225872951618%22%7D; _ym_uid=146412462540490344; _ym_isad=1; _ym_visorc_7857334=w; __cq_uuid=da803170-21f4-11e6-aa9c-fd8d2eed77ab; PrevPageURL=http%3A%2F%2Fwww.adidas.ru%2Fsearch; UserSignUpAndSave=2; ak_bmsc=5D8D5ADA817A1830D669EF612B47DD67170E5ECBF7140000C9C4445710CC116A~plOv26rMgrLkyJGa1jeI+acE7yC/zZ5DcnEFrz/AB0FIwJYdSZhJuYAEBnm1Z3W98XqL+ZwOhMskLVVzf5YaC4mk/t2OXyTQ0ezU6stQZz0L4EsEs3XBzWNZ2QdZzcCnaNNYsW/7SuFd7VslgkmgAmh+bspaIMYSpWq7k5oQvau5o=; onesite_language=ru; onesite_country=RU; onesite_market=RU; geolocatedRegionOrCity=Moscow; us_criteo_sociomantic_split=criteo; __adi_rt_DkpyPh8=CTRLH2H; _ga=GA1.2.520746757.1464124634; RES_TRACKINGID=866314134161171; ResonanceSegment=1; RES_SESSIONID=347364828772007; _gat_tealium_0=1; AMCV_7ADA401053CCF9130A490D4C%40AdobeOrg=283337926%7CMCMID%7C02420446016253062430521925596725595527%7CMCAAMLH-1464729434%7C6%7CMCAAMB-1464729434%7CNRX38WO0n5BH8Th-nqAG_A%7CMCAID%7CNONE; s_cc=true; optimizelyPendingLogEvents=%5B%5D; s_sq=ag-adi-global-prod%252Cag-adi-ru-prod%3D%2526c.%2526a.%2526activitymap.%2526page%253DPLP%25257CSEARCH%2526link%253Dclose%2526region%253DBODY%2526pageIDType%253D1%2526.activitymap%2526.a%2526.c%2526pid%253DPLP%25257CSEARCH%2526pidt%253D1%2526oid%253Dhttp%25253A%25252F%25252Fwww.adidas.ru%25252Fsearch%252523%2526ot%253DA; s_tps=23; s_pvs=33; s_pers=%20s_vnum%3D1464728400393%2526vn%253D1%7C1464728400393%3B%20pn%3D1%7C1466716634407%3B%20c4%3DPLP%257CSEARCH%7C1464126457318%3B%20s_visit%3D1%7C1464126457321%3B%20s_invisit%3Dtrue%7C1464126457323%3B; _gali=simpleSearchFormRedesign; customer_hashed_email_pers=d41d8cd98f00b204e9800998ecf8427e; utag_main=v_id:0154e4a0ef8f00206d14dbe5f35204076001b06e00bd0$_sn:1$_ss:1$_pn:1%3Bexp-session$_st:1464126458567$ses_id:1464124632975%3Bexp-session$search_data:SEARCH-QUERY%7CA%7CA%3Bexp-1466716658550; searchQuery=a; RT="nu=http%3A%2F%2Fwww.adidas.ru%2Fsearch&dm=adidas.ru&si=1fb6c943-f0b1-4668-9e59-b8465ab274c5&ss=1464124622069&sl=1&tt=11023&obo=0&sh=1464124634779%3D1%3A0%3A11023&cl=1464124650728&bcn=%2F%2F364bf52c.mpstat.us%2F&r=http%3A%2F%2Fwww.adidas.ru%2Fsearch&ul=1464124658585"' ));

Returned the page. further understand what cookies, how to screw them.

also for the future - if such problems occur, which usually should be done first

1) see what curl_error returns

2) open firebug , Network tab, on the site enter something into the search bar, press Enter . in the list of pages to which the requests went, you are looking for search?q=a you need (in your case, search?q=a ), in the Headers tab, you are looking for Request headers . In 90% of cases, your problem is that you do not send any HTTP headers that are sent when you visit through the browser. And then just substitute all the headers from the browser into your request (note the curl CURLOPT_HTTPHEADER ) and explore.