Use the CefSharp library for such tasks.

Why should this approach be used?

• You have a much simpler development process due to the fact that instead of writing XPath, conditions and / or cycles in C #, you simply develop everything you need in the browser console (preferably based on Chromium), then when a small core of the class is written (I'll show you it below), you just paste the javascript code that you need.
• Reliability. You are not trying to parse HTML and do not reinvent the wheel, which is almost always a very bad idea. The project is based on Chromium, so you do not have to trust any new / unfamiliar product. Actively supported for synchronization with the new version.

For Javascript calls, jQuery is used for simplicity and demonstration, assuming that it also exists on the target site. But it can also be pure JavaScript or another library provided that this library is used on the site.

If you scroll down, you will notice that in addition to writing a small layer of code and initialization, the solution takes one or two lines:

 string[] urls = await wrapper.GetResultAfterPageLoad("https://yandex.ru", async () => await wrapper.EvaluateJavascript<string[]>( "$('a[href]').map((index, element) => $(element).prop('href')).toArray()")); 

What it is?

This is a managed shell over CEF ( Chromium Embedded Framework ). That is, you get the power of Chromium, which is controlled programmatically.

Why choose CEF / CefSharp?

• You should not bother with parsing pages (and this is a difficult and ungrateful task, which I highly recommend not to do).
• You can work with an already loaded page (after running scripts).
• It is possible to execute arbitrary javascript with the latest features.
• It makes it possible to invoke AJAX with the help of JavaScript, and then with success (success), to pull events in the C # code with the result of AJAX. Detailed and with an example reviewed here .

CefSharp Varieties

• CefSharp.WinForms
• CefSharp.Wpf
• CefSharp.OffScreen
  

The first two are used if you need to give users a Browser control. Conceptually similar to WebBrowser in Windows Forms, which is a wrapper for managing IE, not Chromium, as in our case.

Therefore, we will use the CefSharp.OffScreen (offscreen) version.

Code writing

Suppose we have a console application, but it already depends on you.

Install the CefSharp.OffScreen Nuget package of the 57th version:
Install-Package CefSharp.OffScreen -Version 57.0.0

The fact is that C # all arrays map to List<object> , the result of JavaScript is wrapped in object , which already contains List<object> , string , bool , int depending on the result. In order to make the results strongly typed, create a small ConvertHelper :

 public static class ConvertHelper { public static T[] GetArrayFromObjectList<T>(object obj) { return ((IEnumerable<object>)obj) .Cast<T>() .ToArray(); } public static List<T> GetListFromObjectList<T>(object obj) { return ((IEnumerable<object>)obj) .Cast<T>() .ToList(); } public static T ToTypedVariable<T>(object obj) { if (obj == null) { dynamic dynamicResult = null; return dynamicResult; } Type type = typeof(T); if (type.IsArray) { dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetArrayFromObjectList)) .MakeGenericMethod(type.GetElementType()) .Invoke(null, new[] { obj }); return dynamicResult; } if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof(List<>)) { dynamic dynamicResult = typeof(ConvertHelper).GetMethod(nameof(GetListFromObjectList)) .MakeGenericMethod(type.GetGenericArguments().Single()) .Invoke(null, new[] { obj }); return dynamicResult; } return (T)obj; } } 

To handle javascript errors, create a JavascriptException class.

 public class JavascriptException : Exception { public JavascriptException(string message) : base(message) { } } 

You may have your own way of handling errors.

Create a CefSharpWrapper class:

 public sealed class CefSharpWrapper { private ChromiumWebBrowser _browser; public void InitializeBrowser() { Cef.EnableHighDPISupport(); // Perform dependency check to make sure all relevant resources are in our output directory. Cef.Initialize(new CefSettings(), performDependencyCheck: false, browserProcessHandler: null); _browser = new ChromiumWebBrowser(); // wait till browser initialised AutoResetEvent waitHandle = new AutoResetEvent(false); EventHandler onBrowserInitialized = null; onBrowserInitialized = (sender, e) => { _browser.BrowserInitialized -= onBrowserInitialized; waitHandle.Set(); }; _browser.BrowserInitialized += onBrowserInitialized; waitHandle.WaitOne(); } public void ShutdownBrowser() { // Clean up Chromium objects. You need to call this in your application otherwise // you will get a crash when closing. Cef.Shutdown(); } public Task<T> GetResultAfterPageLoad<T>(string pageUrl, Func<Task<T>> onLoadCallback) { TaskCompletionSource<T> tcs = new TaskCompletionSource<T>(); EventHandler<LoadingStateChangedEventArgs> onPageLoaded = null; T t = default(T); // An event that is fired when the first page is finished loading. // This returns to us from another thread. onPageLoaded = async (sender, e) => { // Check to see if loading is complete - this event is called twice, one when loading starts // second time when it's finished // (rather than an iframe within the main frame). if (!e.IsLoading) { // Remove the load event handler, because we only want one snapshot of the initial page. _browser.LoadingStateChanged -= onPageLoaded; t = await onLoadCallback(); tcs.SetResult(t); } }; _browser.LoadingStateChanged += onPageLoaded; _browser.Load(pageUrl); return tcs.Task; } public async Task EvaluateJavascript(string script) { JavascriptResponse javascriptResponse = await _browser.GetMainFrame().EvaluateScriptAsync(script); if (!javascriptResponse.Success) { throw new JavascriptException(javascriptResponse.Message); } } public async Task<T> EvaluateJavascript<T>(string script) { JavascriptResponse javascriptResponse = await _browser.GetMainFrame().EvaluateScriptAsync(script); if (javascriptResponse.Success) { object scriptResult = javascriptResponse.Result; return ConvertHelper.ToTypedVariable<T>(scriptResult); } throw new JavascriptException(javascriptResponse.Message); } } 

Next we call our CefSharpWrapper class from the Main method.

 public class Program { private static void Main() { MainAsync().Wait(); } private static async Task MainAsync() { CefSharpWrapper wrapper = new CefSharpWrapper(); wrapper.InitializeBrowser(); string[] urls = await wrapper.GetResultAfterPageLoad("https://yandex.ru", async () => await wrapper.EvaluateJavascript<string[]>("$('a[href]').map((index, element) => $(element).prop('href')).toArray()")); wrapper.ShutdownBrowser(); } } 

Also: in this library there is a feature that an empty JavaScript array is cast to null . Therefore, it may be worthwhile to add the appropriate code to ConvertHelper (it depends on your code and needs), or to write something like in the calling code

 if (urls == null) urls = new string[0] 

Also install x64 or x86 as a platform. Any CPU platform is supported, but requires additional code .

If the performance requirements are not very high, you can use the Internet Explorer COM object (add a link to the Microsoft HTML Object Library):

 public static List<string> ParseLinks(string html) { List<string> res = new List<string>(); mshtml.HTMLDocument doc = null; mshtml.IHTMLDocument2 d2 = null; mshtml.IHTMLDocument3 d = null; try { doc = new mshtml.HTMLDocument();//инициализация IE d2 = (mshtml.IHTMLDocument2)doc; d2.write(html); d = (mshtml.IHTMLDocument3)doc; var coll = d.getElementsByTagName("a");//получить коллекцию элементов по имени тега object val; foreach (mshtml.IHTMLElement el in coll)//извлечь атрибут href из всех элементов { val=el.getAttribute("href"); if (val == null) continue; res.Add(val.ToString()); } } finally { //освобождение ресурсов if (doc != null) Marshal.ReleaseComObject(doc); if (d2 != null) Marshal.ReleaseComObject(d2); if (d != null) Marshal.ReleaseComObject(d); } return res; } 

I’ll insert my five kopecks, if you don’t want to mess around with COM objects mshtml, you can create a WebBrowser () object from Windows.Forms, and if you don’t need all the scripts to work, then I understand that the page can be loaded not by the browser itself, but than simpler, like WebClient.DownloadString (), and then load the resulting page text for parsing in WebBrowser:

 var itemPageText = _webClient.DownloadString(url); using (var pageHtml = new WebBrowser()) { pageHtml.DocumentText = itemPageText; var elem = pageHtml.Document.GetElementById("imainImgHldr"); } 

well, etc., the main thing is that methods like GetElementById () are also somewhat more digestible wrappers, unlike mshtml.

F #

Search page for all references to books on F #:

  let fsys = "https://www.google.com/search?tbm=bks&q=F%23" let doc2 = HtmlDocument.Load(fsys) let books = doc2.CssSelect("div.g h3.ra") |> List.map(fun a -> a.InnerText().Trim(), a.AttributeValue("href")) |> List.filter(fun (title, href) -> title.Contains("F#")) 

F # Data
F # Data HTML Parser
F # Data HTML CSS selectors