How to make asynchronous IEnumerable?

See, it's not that hard.

You install the Ix, System.Interactive and System.Interactive.Async nuget packages. You get the IAsyncEnumerable interface and helper classes.

You can use it like this:

 static class FileEx { public static IAsyncEnumerable<string> ReadLinesAsync(string path) { // IAsyncEnumerable<T> обладает только одной функцией - создать энумератор return AsyncEnumerable.CreateEnumerable(() => { var stream = File.OpenText(path); string current = null; // создаём энумератор при помощи готовой фабрики return AsyncEnumerable.CreateEnumerator( // у StreamReader.ReadLineAsync нет перегрузки с ct, пичалько moveNext: async ct => (current = await stream.ReadLineAsync()) != null, current: () => current, dispose: stream.Dispose); }); } } 

See what happens here. To begin with, the same sequence can be run over by different pieces of code alternately, so we keep the state of the current round in the enumerator. In principle, we would need to have a separate class for the enumerator, and keep properties in it. But we will go more fashionable way, and we will keep the data in the circuit. We open StreamReader , we get a variable for the current line.

The asynchronous function MoveNext iterator gets the next line from StreamReader 's, and checks the result for null ( null means the end of the file). The Current function simply returns the current line. And Dispose closes the stream at the end.

Now you can work with this:

 class Program { static async Task Main(string[] args) { var lines = FileEx.ReadLinesAsync("text.txt"); using (var en = lines.GetEnumerator()) { while (await en.MoveNext()) Console.WriteLine(en.Current); } } } 

Or simply

 await FileEx.ReadLinesAsync("text.txt") .ForEachAsync(s => Console.WriteLine(s)); 

In the next version of C #, support for asynchronous enumerators is planned right in the language. With her, our example is written as:

 static class FileEx { public static async IAsyncEnumerable<string> ReadLinesAsync(string path) { using (var stream = File.OpenText(path)) { string current; while ((current = await stream.ReadLineAsync()) != null) yield return current; }; } } class Program { static async Task Main(string[] args) { foreach await (var s in FileEx.ReadLinesAsync("text.txt")) Console.WriteLine(s); } } 

See, for your particular case ( this is the code ) you should parse the task into its component parts, since there are a lot of asynchronous pieces in it. Then you can link them instead.

Let's start by getting the pages and parsing them. It’s easy to get a list of hosts, asynchrony is not needed here:

 var hosts = ConfigStorage.Plugins .Where(p => p.GetParser().GetType() == typeof(Parser)) .Select(p => p.GetSettings().MainUri); 

Now, we need to get a list of HtmlNodeCollection by host. This is a “long” task; we take it to task:

 async Task<HtmlNodeCollection> GetHostMangasAsync(string name, Uri host, CookieClient client) { var searchHost = new Uri(host, "search?q=" + WebUtility.UrlEncode(name)); var page = await Task.Run(() => Page.GetPage(searchHost, client)); if (!page.HasContent) return null; return await Task.Run(() => { var document = new HtmlDocument(); document.LoadHtml(page.Content); return document.DocumentNode.SelectNodes("//div[@class='tile col-sm-6']"); }); } 

Now, we need from the non-asynchronous sequence of hosts and Task , which receives from each host HtmlNodeCollection , to get an asynchronous sequence. I did not find such a method in Ix out of the box, but it is easy to put together it myself. Let us make it generalized, suddenly it will be needed. The code is almost the same as the File.ReadLinesAsync example.

 static class AsyncEnumerableExtensions { public static IAsyncEnumerable<R> SelectAsync<T, R>( this IEnumerable<T> seq, Func<T, Task<R>> selector) { return AsyncEnumerable.CreateEnumerable(() => { IEnumerator<T> seqEnum = seq.GetEnumerator(); R current = default; return AsyncEnumerable.CreateEnumerator( moveNext: async ct => { if (!seqEnum.MoveNext()) return false; current = await selector(seqEnum.Current); return true; }, current: () => current, dispose: seqEnum.Dispose); }); } } 

Armed with this, we can write this:

 IAsyncEnumerable<HtmlNodeCollection> GetSearchPages(string name) { var hosts = ConfigStorage.Plugins .Where(p => p.GetParser().GetType() == typeof(Parser)) .Select(p => p.GetSettings().MainUri); var client = new CookieClient(); return hosts.SelectAsync(host => GetHostMangasAsync(name, host, client))) .Where(nc => nc != null); } 

Checking for null needed because GetHostMangasAsync can return null .

Great, move on. So, we again have a non-asynchronous HtmlNodeCollection collection, from each element of which we can pull an instance of IManga using an asynchronous function (because we have access to the network). Write the code:

 async Task<IManga> GetMangaFromNode(Uri host, CookieClient client, HtmlNode manga) { // Это переводчик, идем дальше. if (manga.SelectSingleNode(".//i[@class='fa fa-user text-info']") != null) return null; var image = manga.SelectSingleNode(".//div[@class='img']//a//img"); var imageUri = image?.Attributes.Single(a => a.Name == "data-original").Value; var mangaNode = manga.SelectSingleNode(".//h3//a"); var mangaUri = mangaNode.Attributes.Single(a => a.Name == "href").Value; var mangaName = mangaNode.Attributes.Single(a => a.Name == "title").Value; if (!Uri.TryCreate(mangaUri, UriKind.Relative, out Uri test)) return null; var result = Mangas.Create(new Uri(host, mangaUri)); result.Name = WebUtility.HtmlDecode(mangaName); if (imageUri != null) result.Cover = await client.DownloadDataAsync(imageUri); return result; } 

We need to connect them now. It is not difficult. The only problem is that GetMangaFromNode also needs a CookieClient , and it is hidden from us inside GetSearchPages . OK, we will pass it outside. Then, we only HtmlNodeCollection , and we also need a host . Modify GetSearchPages : we will return pairs from the host and the HtmlNode collection, and take as input the CookieClient :

 IAsyncEnumerable<(Uri host, HtmlNodeCollection nodes)> GetSearchPages( string name, CookieClient client) { var hosts = ConfigStorage.Plugins .Where(p => p.GetParser().GetType() == typeof(Parser)) .Select(p => p.GetSettings().MainUri); return hosts.SelectAsync( async host => (host, nodes: await GetHostMangasAsync(name, host, client))) .Where(pair => pair.nodes != null); } 

Well, combine. With each asynchronous function, each synchronous collection of HtmlNode gives us a collection of instances of IManga . This is done again with our SelectAsync :

 IAsyncEnumerable<IManga> GetFromHostAndNodes( Uri host, HtmlNodeCollection nodes, CookieClient client) => nodes.SelectAsync(node => GetMangaFromNode(host, client, node)); 

Now you can add a puzzle:

 public IAsyncEnumerable<IManga> Search(string name) { var client = new CookieClient(); return GetSearchPages(name, client) .SelectMany(pair => GetFromHostAndNodes(pair.host, pair.nodes, client)) .Where(m => m != null); } 

Everything!