Suppose there is a site on which there is a field.
In this field, you need to send some value, click on the button, then get a new page, where to go on the first link.
Do I need to refer to the WebBrowser class for these?
If possible, show an example (for example, you can take the target google site, but without using its API.)
Let me show you by the example of a program that allows you to google directly in the console (it extracts the first title links of the search on google.com ):
Use CefSharp - a wrapper library based on Chromium. Described it in great detail in this answer . It is installed simply through the Nuget package.
Install-Package CefSharp.OffScreen -Version 57.0.0 CefSharpWrapper two classes from my answer ( CefSharpWrapper and ConvertHelper ), you have a skeleton program ready with which you can execute any JavaScript directly from the console application.
Also install x64 or x86 as a platform. Any CPU platform is supported, but requires additional code .
Also for this task, add the Address property to CefSharpWrapper :
public string Address => _browser.Address; and the WaitTillAddressChanges method:
public void WaitTillAddressChanges() { // wait till address changes AutoResetEvent waitHandle = new AutoResetEvent(false); EventHandler<AddressChangedEventArgs> onAddressChanged = null; onAddressChanged = (sender, e) => { _browser.AddressChanged -= onAddressChanged; waitHandle.Set(); }; _browser.AddressChanged += onAddressChanged; waitHandle.WaitOne(); } Here is an example of the program itself ( Program class, Main method):
public class Program { private static void Main() { MainAsync().Wait(); } private static async Task MainAsync() { CefSharpWrapper wrapper = new CefSharpWrapper(); wrapper.InitializeBrowser(); Console.Write("Введите поисковой запрос: "); string searchText = Console.ReadLine(); string[] urls = await wrapper.GetResultAfterPageLoad("https://google.com", async () => { await wrapper.EvaluateJavascript( // заполняем тесковое поле $@"document.getElementById('lst-ib').value = '{searchText}'; // выполняем submit поисковой формы document.getElementById('tsf').submit()"); // Ждём когда перейдёт на результаты поиска wrapper.WaitTillAddressChanges(); // Когда страница результатов поиска полностью подгрузится, излекаем результаты return await wrapper.GetResultAfterPageLoad(wrapper.Address, async () => await wrapper.EvaluateJavascript<string[]>( // получаем результаты "Array.prototype.map.call(document.querySelectorAll('h3.r > a'), (a) => a.href);")); }); Console.WriteLine("Первые ссылки поиска:"); foreach (string url in urls) { Console.WriteLine(url); } wrapper.ShutdownBrowser(); } } For example, if I enter "parse html C #":
It is also quite convenient to work with AJAX using this library .
If you only need to code some specific, known parts of a website, you can simply pull out the necessary requests that are sent by the browser (you can isolate them in the browser developer console, usually on the Network tab) and send this request in the program, for example via HttpClient .
If you cannot know the initial requests themselves (it is necessary to receive dynamically in the program itself), then there can be 2 variants of events:
<form> and build on it. It has an action attribute that contains a path and an encoded attribute, which specifies the encoding format (if not, the default is " application/x-www-form-urlencoded ", it seems, although it’s definitely better to look through the developer console). All this is collected, including the parameters inside the <form> , sent via the HttpClient and get an answerBut in most cases, there is enough the usual sending of pre-prepared (subtracted) requests with their own parameters.
You can also bind to HttpClient 'from HttpClientHandler for storing cookies and other buns.
For example, you need to add a post on the wall in VK for a certain time - the algorithm will be like this:
HttpClient with HttpClient with itHttpResponseMessage to come so that we have a cookie
I found an interesting lib of Selenium with the help of which, through the driver, you can interact with a real browser. Pretty easy to manage:
using OpenQA.Selenium; using OpenQA.Selenium.Firefox; // Requires reference to WebDriver.Support.dll using OpenQA.Selenium.Support.UI; class GoogleSuggest { static void Main(string[] args) { // Create a new instance of the Firefox driver. // Note that it is wrapped in a using clause so that the browser is closed // and the webdriver is disposed (even in the face of exceptions). // Also note that the remainder of the code relies on the interface, // not the implementation. // Further note that other drivers (InternetExplorerDriver, // ChromeDriver, etc.) will require further configuration // before this example will work. See the wiki pages for the // individual drivers at http://code.google.com/p/selenium/wiki // for further information. using (IWebDriver driver = new FirefoxDriver()) { //Notice navigation is slightly different than the Java version //This is because 'get' is a keyword in C# driver.Navigate().GoToUrl("http://www.google.com/"); // Find the text input element by its name IWebElement query = driver.FindElement(By.Name("q")); // Enter something to search for query.SendKeys("Cheese"); // Now submit the form. WebDriver will find the form for us from the element query.Submit(); // Google's search is rendered dynamically with JavaScript. // Wait for the page to load, timeout after 10 seconds var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10)); wait.Until(d => d.Title.StartsWith("cheese", StringComparison.OrdinalIgnoreCase)); // Should see: "Cheese - Google Search" (for an English locale) Console.WriteLine("Page title is: " + driver.Title); } } } However, I still wanted to know how to do this using standard .NET tools.
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10)); wait.Until(d => d.Title.StartsWith("cheese", StringComparison.OrdinalIgnoreCase)); var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10)); wait.Until(d => d.Title.StartsWith("cheese", StringComparison.OrdinalIgnoreCase)); . I understand he waits 10 seconds until the page loads. Is there a more elegant way to not guess the interval? So that it was a method that waits for as long as necessary without specifying an interval, or just some kind of event like AddressChanged ? - Vadim OvchinnikovSpecify how the processing of the entered text in the field on the site? Does the site send it GET / POST or process it differently?
I will give the course of my thoughts on the example of googl'a and GET .. Why should we emulate user actions on the site if we can directly work with sending a request, obtaining a result and further parsing
string query_user = Console.ReadLine(); string url = "https://www.google.ru/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#newwindow=1&q=" + query_user; HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url); HttpWebResponse response = (HttpWebResponse)request.GetResponse(); And then we extract from the response variable all that is needed.
curl .. | grep ... curl .. | grep ... juzat from the console and that's it. But this is a crutch - i. will break down if the site they are trying to manage changes a little. And it will not work at all if human-like emulation is required - for example, if it is required that the site cannot distinguish a robot from a human. Also this is not applicable if the emulator is needed for auto-testing the site UI. - Goncharov AlexanderIt all depends on your needs and on the site:
In principle, if we summarize all the information submitted below and lead to the most universal answer:
but. if you need the fastest "one-time" solution and the speed of work is not important - it means Selenium
b. If this is a testing task - Selenium
at. If you need a universality of the solution and the most simple support for long periods of time and no matter how much coding will take:
More about each of the ways ....
Webbrowser
Minus
Plus
+? in theory, cross-platform, but not sure.
Direct work through HttpWebRequest Post / Get requests with the subsequent parsing. The required sequence can be found in the browser developer console, on the Network tab. It is advisable to use NOT low-level requests and some libraries for REST requests. It will be faster much the code.
Minus
Plus
Selenium - in general, it is designed specifically for automated testing of websites. Including with ajaks-technology. That is, it can do almost all the actions that people can do in the browser: finding the interface elements of the site, sending them button presses, scrolling, taking screenshots, checking for visibility / accessibility, etc. Because specified C # tag, as well as it was said about the absence of unnecessary windows, you need to use a bunch: Selenium Web-Driver and PhantomJS. Phantom is a bezayayny browser on the engine like a chrome. It can all the same as a simple browser, only it does not show visually and does not spend resources on rendering. Separately, I want to say that the selenium itself is rather inconvenient and far from the PLO. The problem is that there is no particular choice ...
Minus
Plus
Not sure
? I have no idea about cross-platform
The shell around the finished engine like Chromium. For example - CefSharp.
Minus
Plus
Not sure:
? I have no idea about cross-platform. Most likely there
In the title of the question is a console application. But, I can assume the text
In this field, you need to send some value, click on the button, then get a new page, where to go on the first link.
Perhaps you will suit the emulation of user actions in the browser. About browser emulation:
Very accurate emulation of user actions is done by writing your own browser extension (Firefox / GoogleChrome for example) - not to distinguish from this user. A plugin can completely take over control - for example, open a tab with a specific site, and "grab" it.
Inside the plugin, a certain entity is developed that can emulate movement and mouse clicks from object to object: objects themselves from another site are selected, for example, by simple jQuery - just like in the code of any site. Entities also need to add the ability to emulate text input in any field.
Emulation can occur by sending a trusted DOM event (it can do the plug-in code), or you can directly use WinApi and its PostMessageW (at least firefox add-ons can connect any dll and then use it from JS) —the reinforced concrete method. In general, technically, such an emulator cannot be distinguished from the user.
Under similar activity the known Greasemonkey addon is ground . But if the scale of the emulation is planned large (the policy of adding certain scripts to certain pages is not enough), I would not recommend it, because its logic is not enough.
If you want to add emulation and browser control from the console, use selenium. It does not work if you want it to work from a standard server - that is, without a GUI. But in my practice I will say - they used emulators often, for them we had separate servers with Windows, with GUI. Without a GUI, with a bare console, there is an opinion that it will not work to emulate quality ..
The classic use of such emulators is bots-parsers, bots-spammers, bots-gaming, auto-testing your own site.
In my opinion, the best solution is WatiN .
WatiN Features
- Automate all major HTML elements with ease
- Find elements by multiple attributes
- Native support for Page and Control model
- Supports AJAX website testing
- Supports creating screenshots of webpages
- Supports frames (cross domain) and iframes
- Handles popup dialogs like alert, confirm, login etc ..
- Supports HTML dialogs (modal and modeless)
- Easy to integrate with your favorit (unit) test tool
- Works with Internet Explorer 6, 7, 8, 9 and FireFox 2 and 3
- Can be used with any .Net Language
- Licensed under the Apache License 2.0
- Downloaded more than 120,000 times
- New features yourself
Example:
[Test] public void SearchForWatiNOnGoogle() { using (var browser = new IE("http://www.google.com")) { browser.TextField(Find.ByName("q")).TypeText("WatiN"); browser.Button(Find.ByName("btnG")).Click(); Assert.IsTrue(browser.ContainsText("WatiN")); } } I myself use for grabbing.
Source: https://ru.stackoverflow.com/questions/606754/
All Articles
headless browsers- Herrgott