Skip to main content

Screen API

Please read the Material Usage Rules on this site.

Description​

IDroidScreenAPI provides methods for analyzing the device screen: searching for images and pixels, taking screenshots, and managing screen capture.

Methods​

FindImage​

  • Rectangle FindImage(string imageHash, Rectangle[] rectangles, int confidenceInterval)
    Searches for an image by hash within the specified screen areas.

    Parameters:

    • imageHash — hash of the image to compare against;
    • rectangles — array of Rectangle areas to search in;
    • confidenceInterval — match accuracy (0–100, higher means stricter match).

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImage(string imageHash, Rectangle[] rectangles, int confidenceInterval, int searchTime)
    Searches for an image with a time limit.

    Parameters:

    • imageHash — image hash;
    • rectangles — search areas;
    • confidenceInterval — match accuracy;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the found area.

  • Rectangle FindImage(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval)
    Searches for one of several images.

    Parameters:

    • imageHashes — array of image hashes;
    • rectangles — search areas;
    • confidenceInterval — match accuracy.

    Returns:
    Rectangle — the area of the first found image.

  • Rectangle FindImage(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval, int searchTime)
    Searches for one of several images with a time limit.

    Parameters:

    • imageHashes — array of hashes;
    • rectangles — search areas;
    • confidenceInterval — match accuracy;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the found area.

Example​

var screen = instance.DroidInstance.Screen;

var hash = project.Variables["imageHash"].Value; // Hash of the reference image
var areas = new[] { new Rectangle(0, 0, 720, 1280) }; // Where to search - the whole screen here

var found = screen.FindImage(hash, areas, 90); // Match accuracy of 90
if (found.IsEmpty)
	throw new Exception("Image not found");

var waited = screen.FindImage(hash, areas, 90, 10); // Wait up to 10 seconds for it to appear

FindImageBlackWhite​

  • Rectangle FindImageBlackWhite(string imageHash, Rectangle[] rectangles, int confidenceInterval, int threshold)
    Searches for an image in black-and-white mode.

    Parameters:

    • imageHash — image hash;
    • rectangles — search areas;
    • confidenceInterval — match accuracy;
    • threshold — binarization threshold.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageBlackWhite(string imageHash, Rectangle[] rectangles, int confidenceInterval, int threshold, int searchTime)
    Black-and-white search with a timeout.

    Parameters:

    • imageHash — hash;
    • rectangles — areas;
    • confidenceInterval — accuracy;
    • threshold — threshold;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageBlackWhite(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval, int threshold)
    Searches for multiple images in black-and-white mode.

    Parameters:

    • imageHashes — array of hashes;
    • rectangles — areas;
    • confidenceInterval — accuracy;
    • threshold — threshold.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageBlackWhite(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval, int threshold, int searchTime)
    Black-and-white search for multiple images with a timeout.

    Parameters:

    • imageHashes — array;
    • rectangles — areas;
    • confidenceInterval — accuracy;
    • threshold — threshold;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the area where the image was found.

Example​

var screen = instance.DroidInstance.Screen;

var hash = project.Variables["imageHash"].Value; // Hash of the reference image
var areas = new[] { new Rectangle(0, 0, 720, 1280) }; // Search area

var found = screen.FindImageBlackWhite(hash, areas, 90, 128); // Binarization threshold of 128
if (found.IsEmpty)
	throw new Exception("Image not found");

FindImageGrayscale​

  • Rectangle FindImageGrayscale(string imageHash, Rectangle[] rectangles, int confidenceInterval)
    Searches for an image in grayscale mode.

    Parameters:

    • imageHash — hash;
    • rectangles — areas;
    • confidenceInterval — accuracy.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageGrayscale(string imageHash, Rectangle[] rectangles, int confidenceInterval, int searchTime)
    Grayscale search with a timeout.

    Parameters:

    • imageHash — hash;
    • rectangles — areas;
    • confidenceInterval — accuracy;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageGrayscale(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval)
    Searches for multiple images in grayscale.

    Parameters:

    • imageHashes — array;
    • rectangles — areas;
    • confidenceInterval — accuracy.

    Returns:
    Rectangle — the area where the image was found.

  • Rectangle FindImageGrayscale(string[] imageHashes, Rectangle[] rectangles, int confidenceInterval, int searchTime)
    Grayscale search with a timeout.

    Parameters:

    • imageHashes — array;
    • rectangles — areas;
    • confidenceInterval — accuracy;
    • searchTime — maximum search time in seconds.

    Returns:
    Rectangle — the area where the image was found.

Example​

var screen = instance.DroidInstance.Screen;

var hash = project.Variables["imageHash"].Value; // Hash of the reference image
var areas = new[] { new Rectangle(0, 0, 720, 1280) }; // Search area

var found = screen.FindImageGrayscale(hash, areas, 90); // Compared in grayscale
if (found.IsEmpty)
	throw new Exception("Image not found");

FindPixel​

  • Point FindPixel(Color color, Rectangle[] rectangles, int variation)
    Searches for a pixel of the specified color.

    Parameters:

    • color — the target color (Color);
    • rectangles — search areas;
    • variation — allowed color deviation.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixel(Color color, Rectangle[] rectangles, int variation, int searchTime)
    Searches for a pixel with a time limit.

    Parameters:

    • color — color;
    • rectangles — search areas;
    • variation — color deviation;
    • searchTime — maximum search time in seconds.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixel(string htmlColor, Rectangle[] rectangles, int variation)
    Searches for a pixel by HTML-format color.

    Parameters:

    • htmlColor — color as a string (e.g. #FF0000);
    • rectangles — search areas;
    • variation — allowed deviation.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixel(string htmlColor, Rectangle[] rectangles, int variation, int searchTime)
    Searches for an HTML color with a timeout.

    Parameters:

    • htmlColor — color;
    • rectangles — search areas;
    • variation — deviation;
    • searchTime — maximum search time in seconds.

    Returns:
    Point — coordinates of the found pixel.

Example​

var screen = instance.DroidInstance.Screen;

var areas = new[] { new Rectangle(0, 0, 720, 1280) }; // Search area

var point = screen.FindPixel("#FF0000", areas, 10); // A red pixel, deviation of 10
if (point.IsEmpty)
	throw new Exception("Pixel not found");

var waited = screen.FindPixel(Color.Red, areas, 10, 10); // The same, but waiting up to 10 seconds

FindPixels​

  • Point FindPixels(Color[] colors, Rectangle[] rectangles, int variation)
    Searches for a pixel from a set of colors.

    Parameters:

    • colors — array of colors;
    • rectangles — search areas;
    • variation — allowed deviation.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixels(Color[] colors, Rectangle[] rectangles, int variation, int searchTime)
    Searches for one of the colors with a time limit.

    Parameters:

    • colors — array of colors;
    • rectangles — search areas;
    • variation — deviation;
    • searchTime — maximum search time in seconds.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixels(string[] htmlColors, Rectangle[] rectangles, int variation)
    Searches for a pixel from a set of HTML colors.

    Parameters:

    • htmlColors — array of colors in string format;
    • rectangles — search areas;
    • variation — deviation.

    Returns:
    Point — coordinates of the found pixel.

  • Point FindPixels(string[] htmlColors, Rectangle[] rectangles, int variation, int searchTime)
    Searches for HTML colors with a timeout.

    Parameters:

    • htmlColors — array of colors;
    • rectangles — search areas;
    • variation — deviation;
    • searchTime — maximum search time in seconds.

    Returns:
    Point — coordinates of the found pixel.

Example​

var screen = instance.DroidInstance.Screen;

var areas = new[] { new Rectangle(0, 0, 720, 1280) }; // Search area
var colors = new[] { "#FF0000", "#00FF00" }; // Any one of the colours will do

var point = screen.FindPixels(colors, areas, 10);
if (point.IsEmpty)
	throw new Exception("None of the colours was found");

GetScreenSize​

  • Size GetScreenSize()
    Returns the real screen resolution of the device in pixels. This is the physical resolution the device reports, not the size of the captured image, so it is the coordinate space used by touch, swipe and pixel operations.

    Returns:
    Size — screen width and height in pixels.

Example​

var size = instance.DroidInstance.Screen.GetScreenSize();
project.SendInfoToLog($"Screen resolution: {size.Width}x{size.Height}");
note

The value is cached per device. After a resolution change (e.g. with wm size WidthxHeight), call ResetScreen() so it is read again.

RefreshScreen​

  • void RefreshScreen()
    Refreshes the screen image. Used when a screen capture type is selected that requires manual refresh (captureScreenMethod = "off").

Example​

var screen = instance.DroidInstance.Screen;

instance.DroidInstance.Action.Start("off"); // Start the device without continuous capture
screen.RefreshScreen(); // Update the image manually

ResetScreen​

  • void ResetScreen()
    Resets screen capture parameters and starts a new capture process. Used to restart capture when conditions change (e.g. after changing the resolution with wm size WidthxHeight).

Example​

var screen = instance.DroidInstance.Screen;

instance.DroidInstance.Input.RootShell("wm size 720x1280"); // Change the screen resolution
screen.ResetScreen(); // Restart the capture for the new resolution

ScreenshotAsArray​

  • byte[] ScreenshotAsArray()
    Takes a screenshot of the entire screen.

    Returns:
    Byte array of the image.

Example​

var data = instance.DroidInstance.Screen.ScreenshotAsArray();
using (var ms = new MemoryStream(data))
using (var bmp = new Bitmap(ms))
	bmp.Save(project.Path + "myScreen.png");

  • byte[] ScreenshotAsArray(Rectangle area)
    Takes a screenshot of the specified area.

    Parameters:

    • area — screen area.

    Returns:
    Byte array.

ScreenshotAsBase64String​

  • string ScreenshotAsBase64String()
    Takes a screenshot of the screen.

    Returns:
    A Base64 string.

  • string ScreenshotAsBase64String(Rectangle area)
    Takes a screenshot of an area.

    Parameters:

    • area — screen area.

    Returns:
    A Base64 string.

Example​

var screen = instance.DroidInstance.Screen;

var base64 = screen.ScreenshotAsBase64String(); // The whole screen
var area = screen.ScreenshotAsBase64String(new Rectangle(0, 0, 300, 300)); // One area only

var data = Convert.FromBase64String(base64);
using (var ms = new MemoryStream(data))
using (var bmp = new Bitmap(ms))
	bmp.Save(project.Path + "myScreen.png");

SetCaptureScreenQuality​

  • void SetCaptureScreenQuality(int quality)
    Sets the capture quality.

    Parameters:

    • quality — quality level (affects speed and accuracy). Allowed values are 0 to 10.

Example​

var screen = instance.DroidInstance.Screen;

screen.SetCaptureScreenQuality(10); // Highest quality - more accurate search, but slower
screen.SetCaptureScreenQuality(0); // Lowest quality - faster