4. AXUI built-in drivers¶
Page Status: | Development |
---|---|
Last Reviewed: |
AXUI has implement some drivers for common used platforms, this chapter will have a introduce about this three drivers
- windows desktop, based on windows native UIAutomation Client API
- web, based on selenium project
- mobile android/ios, based on appium project
4.1. driver for windows UIAutomation API¶
We use comtypes to access this windows COM API, thus to use AXUI to automate windows UI, you need install comtypes first.
Windows UIAutomation API separates operations for different kinds of UI into a set of control patterns, it’s recommended to use these patterns to operate target UI, AXUI expose these patterns to end users, anyway end users should need to have a check of these patterns.
Note
I have tested windows driver on win8.1 and win10,
- it’s works well with windows UI framework like win32, winform, WPF, windows store app
- not works well with Qt framework, UIA recognize all kinds of Qt controls as frames
- not support for directX, custom UI controls
4.1.1. UIA identifiers¶
UIA supports different ways to find UI elements, AXUI only uses FindFisrt and FindAll to find UI element.
For search scope, AXUI uses TreeScope_Descendants as default, only uses TreeScope_Children for find element under root element.
Basically, AXUI supports most of UIA search conditions, in a different flavour using AppMap. The search condition in AppMap has a structure like:
identifier="<key=value [AND key=value] [OR key=value]>"
the identifier key is from UIA property identifiers, we strip the repeat part and get our AXUI identifier key, like:
UIA_NamePropertyId -> [UIA_]Name[PropertyId] -> Name
- single property condition, like:
"Name='element_name'"
- simple and/or condition, like:
"Name='element_name' AND IsEnabled=True"
,"Name='element_name_1' OR Name='element_name_1'"
- multiple and/or condition, like:
"Name='element_name_1' OR Name='element_name_1' AND IsEnabled=True"
Note
Suggest using inspect tool to retrieve UI identifier values
4.1.2. UIA element properties¶
UIA element has properties that we can retrieve and check their values, to get these properties value, we just need to append the property name after the element, just like normal python properties:
appmap.<element_1>.[element_2]...[element_n].property_name
property_name is from UIA property identifiers, just like AXUI identifier key:
UIA_NamePropertyId -> [UIA_]Name[PropertyId] -> Name
4.1.3. UIA element patterns¶
As said before, UIA split interfaces for different UI element into different control patterns, AXUI porting these pattern interfaces untouched, you can use these pattern interfaces directly:
appmap.<element_1>.[element_2]...[element_n].pattern_name.[pattern_method][pattern_property]
Notice pattern_name is from control patterns, with “IUIAutomation” prefix stripped:
IUIAutomationValuePattern -> [IUIAutomation]ValuePattern -> ValuePattern
4.2. driver for WebDriver compatible projects¶
selenium webdriver
and appium
all use a C/S structure to support multiple languages, the client side and server side use WebDriver protocol to communicate with each other.
since selenium webdriver
and appium
already have python clients, we don’t reinvent the wheel, but use these python clients to implement our drivers for AXUI
Note
Since I only little experience for web UI and mobile UI automation, these driver could be not good to use
Welcome if somebody to write better drivers to replace my reference drivers.
4.2.1. selenium webdriver¶
4.2.2. selenium identifiers¶
All selenium identifiers to search elements is in selenium/webdriver/common/by.py
, we can use these identifiers to search elements in AXUI, like:
identifier="key=value"
It’s similar with windows driver, but not supports and/or search condition.
The identifier key is property names of By
class. Like “ID”, “XPATH”, “TAG_NAME”...
4.2.3. selenium properties¶
We can retrieve the selenium element’s properties just like normal python properties:
appmap.<element_1>.[element_2]...[element_n].property_name
property_name
is same with selenium element property name
4.2.4. selenium patterns/interfaces¶
I have restructured the selenium element methods to different pattern class, so you cannot access selenium element methods directly. Currently there are 4 pattern interfaces:
4.2.4.1. Keyboard
interface¶
This interface has one method “input”, to replace selenium “send_keys” method. For input normal keys like [0~9][a~z][A~Z], input directly:
appmap.<element_1>.[element_2]...[element_n].Keyboard.input("123asdfADSD")
For special charactors like “space”, “tab”, “newline”, “F1~F12”, You use {key_name} to replace them, all support keys in “selenium/webdriver/common/keys”.
appmap.<element_1>.[element_2]...[element_n].Keyboard.input(“{space}”, “{tab}”, “{F1}”)
4.2.4.2. Mouse
interface¶
This interface has one method “left_click”, to replace selenium “click” method:
appmap.<element_1>.[element_2]...[element_n].Mouse.left_click()
4.2.4.3. WebUIElementPattern
interface¶
This interface wrap original selenium methods for normal web element:
interfaces = [
"submit",
"clear",
"is_selected",
"is_enabled",
"is_displayed",
"value_of_css_property",
]
Use like:
appmap.<element_1>.[element_2]...[element_n].WebUIElementPattern.is_enabled()
4.2.4.4. BrowserPattern
interface¶
This interface wrap original selenium methods for browser element:
interfaces = [
"get",
"close",
"maximize_window",
"execute_script",
"execute_async_script",
"set_script_timeout",
"back",
"forward",
"refresh",
"get_cookies",
"get_cookie",
"delete_cookie",
"delete_all_cookies",
"add_cookie",
"implicitly_wait",
"set_page_load_timeout",
"set_window_size",
"get_window_size",
"set_window_position",
"get_window_position",
"get_log",
]
Use like:
appmap.<element_1>.[element_2]...[element_n].BrowserPattern.get("http:://www.bing.com")
Note
I have tested selenium driver with some browsers on windows, seems selenium webdriver has some problems with IE 11.
Suggest use window driver to test IE’s UI, windows UIA supports IE pretty well.
4.2.5. appium¶
Note
since I don’t have an apple/android environment, the appium driver is not tested
I will be very glad someone can have a test for it :)