Image search in image

Using OpenCV should of course be more successful. I experimented with AForge

it turned out well too

Here is the class of work with the library, in fact the use itself takes 2 lines, I confess, I took this part from the Internet, the rest was written by myself

 public class AforgeService { //найденные совпадения private TemplateMatch[] _matchings; /// <summary> /// Количество найденных совпадений /// </summary> public int CountMatchings { get => _matchings != null ? _matchings.Length : 0; } //ctor public AforgeService() { } /// <summary> /// Содержит ли исходное изображение представленый образец /// </summary> /// <param name="pathOriginalImage">путь к файлу исходного изображения</param> /// <param name="pathSampleImage">путь к файлу образца</param> /// <returns>true если содержит</returns> public async Task<bool> IsContains(string pathOriginalImage, string pathSampleImage) { if (String.IsNullOrEmpty(pathOriginalImage)) throw new ArgumentNullException(nameof(pathOriginalImage)); if (String.IsNullOrEmpty(pathSampleImage)) throw new ArgumentNullException(nameof(pathSampleImage)); var sample = new Bitmap(pathSampleImage); var orig = new Bitmap(pathOriginalImage); //пользуемся библиотекой ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f); _matchings = await Task.Run(() => tm.ProcessImage(orig, sample)); return _matchings.Any(); } /// <summary> /// Получение коллекции найденных мест где находится образец /// </summary> /// <returns>коллекция найденных мест</returns> public List<FoundPlace> GetPlaces() { List<FoundPlace> result = new List<FoundPlace>(); if (CountMatchings == 0) return result; int id = 0; foreach (var match in _matchings) { FoundPlace place = new FoundPlace { Id = ++id, Similarity = match.Similarity, Top = match.Rectangle.Top, Left = match.Rectangle.Left, Height = match.Rectangle.Height, Width = match.Rectangle.Width }; result.Add(place); } return result; } } 

This class is for saving the found place.

 public class FoundPlace { public int Id { get; set; } public double Left { get; set; } public double Top { get; set; } public double Width { get; set; } public double Height { get; set; } public double Similarity { get; set; } } 

This is a view model class.

 public class MainViewModel : INotifyPropertyChanged, IDisposable { public event PropertyChangedEventHandler PropertyChanged; private readonly IMainWindow _mainWindow; private string _pathOriginalImage; //ctor public MainViewModel(IMainWindow mainWindow) { _mainWindow = mainWindow; } /// <summary> /// Флаг запуска поиска, для выкл./вкл. кнопок /// </summary> private bool _IsSearching; public bool IsSearching { get => _IsSearching; set { _IsSearching = value; SelectSampleCommand.RaiseCanExecuteChanged(); SearchCommand.RaiseCanExecuteChanged(); } } /// <summary> /// Исходное изображение (поле игры) /// </summary> public string OriginalImage { get => @"~\..\Assets\Original.jpg"; } /// <summary> /// Образец для поиска /// </summary> private string _SampleImage; public string SampleImage { get => _SampleImage; set { _SampleImage = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(SampleImage))); SearchCommand.RaiseCanExecuteChanged(); } } /// <summary> /// Текстовое сообщение о процессе /// </summary> private string _Message; public string Message { get => _Message; set { _Message = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(Message))); } } /// <summary> /// Список найденных мест для ListBox /// </summary> private List<FoundPlace> _Places; public List<FoundPlace> Places { get { return _Places; } set { _Places = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(Places))); } } private FoundPlace _SelectedPlace; public FoundPlace SelectedPlace { get => _SelectedPlace; set { _SelectedPlace = value; PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof(SelectedPlace))); _mainWindow.DrawPlace(_SelectedPlace); } } /// <summary> /// Кнопка Выбрать /// </summary> private RelayCommand _SelectSampleCommand; public RelayCommand SelectSampleCommand { get => _SelectSampleCommand = _SelectSampleCommand ?? new RelayCommand(OnSelectSample, CanSelectSample); } private bool CanSelectSample() { if (IsSearching) { return false; } return true; } private void OnSelectSample() { string file = _mainWindow.SelectSample(); if (String.IsNullOrEmpty(file)) return; SampleImage = file; } /// <summary> /// Кнопка Искать /// </summary> private RelayCommand _SearchCommand; public RelayCommand SearchCommand { get => _SearchCommand = _SearchCommand ?? new RelayCommand(OnSearch, CanSearch); } private bool CanSearch() { if (String.IsNullOrEmpty(SampleImage) || IsSearching) { return false; } return true; } private async void OnSearch() { Message = "Ждите..."; IsSearching = true; AforgeService service = new AforgeService(); try { using (OverrideCursor cursor = OverrideCursor.GetWaitOverrideCursor()) { string pathOrigin = GetOriginalImage(); bool isContains = await service.IsContains(pathOrigin, SampleImage); if (isContains) { Places = service.GetPlaces(); } else { Places = new List<FoundPlace>(); } } } catch (Exception ex) { var message = $"Возникла ошибка: {ex.Message}"; _mainWindow.ShowMessage(message, "Ошибка"); } finally { Message = $"Найдено мест: {service.CountMatchings}"; IsSearching = false; } } /// <summary> /// Получение пути к оригинальному изображению (полю игры) /// </summary> /// <returns></returns> private string GetOriginalImage() { if (!String.IsNullOrEmpty(_pathOriginalImage) && File.Exists(_pathOriginalImage)) { return _pathOriginalImage; } _pathOriginalImage = Path.Combine(Path.GetTempPath(), "Original.jpg"); Uri imgUri = new Uri("pack://application:,,,/Assets/Original.jpg"); StreamResourceInfo imgStream = Application.GetResourceStream(imgUri); using (Stream imgs = imgStream.Stream) using (FileStream fs = File.Create(_pathOriginalImage)) { byte[] ar = new byte[imgs.Length]; imgs.Read(ar, 0, ar.Length); fs.Write(ar, 0, ar.Length); } return _pathOriginalImage; } /// <summary> /// IDisposable /// </summary> public void Dispose() { if (!String.IsNullOrEmpty(_pathOriginalImage) && File.Exists(_pathOriginalImage)) { File.Delete(_pathOriginalImage); } } } 

The whole example can be downloaded here.

I think you have enough functionality implemented in OpenCV

OpenCV (English Open Source Computer Vision Library, open source computer vision library) is a library of computer vision, image processing and general-purpose numerical algorithms with open source.

(c) Wikipedia.

Here are some good lessons about it, but what you can come in handy in theory.

There is a large, complex and popular area of ​​technology called image recognition. For the most part deep neural networks are used there, but not only. And it is unlikely that such complexity is necessary in your case (although it can inspire the correct vector of further actions).

A simple way is to break the playing field into parts (it seems that this game is especially important in this game) and check each part for similarities with known objects. If items can be in random places, and not just on a given grid, then a more advanced analysis is needed: you can first check on the grid, then on the same grid with a half shift (or even a third and two thirds). You can also scale it, take a certain hash from a part of the image ...

By the way, if you can’t achieve 99% accuracy (which is possible in principle, because all objects will be static and not rotated under different lighting, as in the real world), then you can compensate for the accuracy by the reasonableness of clicks: to save positions that have been unsuccessfully clicked in the next times do not blunt, do not waste time Then the main thing is that the search algorithm should produce as few false positives as possible, and false positives will be quickly cut off in practice.

Useful links:

• Closed question on ruSO , but still with some interesting ideas.

• Article about the recognition of images (Habr)

• Viola-Jones method for searching and recognizing faces (Habr)

If you need to search for an image of a fragment of this image, and not something remotely resembling a fragment, there is nothing better than the correlation method. Fast and reliable. And no recognition is needed.