UI automation is a programmatic interface to the user interface of your application to external applications that may be interested in programmatic communication with your UI. In fact UI automation is the prime facilitator for accessibility features in Windows where external applications such as screen readers that have no clue about your application, can still interact with your application easily.
I was acquainted with UI automation during my tenure at Microsoft in 2005 when I started working as software developer in test within Visual Studio Team Architects team on a project called Whitehorse. For those who need some background, Whitehorse consisted of SOA diagrams within Visual Studio (you can learn more about this project in the MSDN magazine here). Testing of visual designers in the product test team was done entirely using an internal UI test automation framework built on top of Microsoft Active Accessibility framework.
The UI automation framework is relatively new to Windows platform and successor to Windows Active Accessibility. The framework provides a unified way to discover UI controls in an application built using any of the following desktop technologies: Win32, Windows Forms or WPF.
UI automation has several benefits when used in applications:
-
Accessibility features – UI automation provides accessibility support in your application. Accessibility support becomes even more important when there are matter of legal compliance in the domain your application is used in (e.g. Government regulations).
-
Automated UI testing – UI automation can automate testing for application UI saving time and costs associated with manual and regression testing. Moreover given a set of use cases, UI automation can be used to verify application behavior via scenario tests.
-
Custom controls – If you are authoring custom controls, UI automation support enables end clients of your control to automate your control in their application UI.
How does UI automation work?
UI automation provides a common protocol for exchanging information between your application and an external entity such as screen reader or automated UI tests. UI automation provides an API using which an external application can discover controls, navigate the visual tree, access the state of the controls and perform control specific actions.
In WPF, automation object model is provided via System.Windows.Automation.AutomationElement instance associated with a UIElement (said to be “automation peer” of the control). If you are authoring a custom control, you may want to implement one or more interfaces defined in the UIAutomationProvider.dll under the System.Windows.Automation.Provider namespace to support UI automation.
To support UI automation, a control author needs to implement an abstract class AutomationPeer from UIElement class’ virtual method OnCreateAutomationPeer. AutomationPeer is then used at runtime to extract AutomationElement for the UIElement. It is important to note that the standard controls in WPF have standard implementation of AutomationPeer associated with the controls out of the box. In authoring a derivative of AutomationPeer you may want to subclass a standard implementation rather than deriving directly from AutomationPeer and authoring a class from scratch.
Following image shows out of the box derivatives of AutomationPeer:
Each AutomationPeer must implement one or more “standard control patterns” to expose capabilities that can be used by automation clients such as screen readers. A control may expose one or more control patterns defined by the members of PatternIntern enumeration:
A control can be queried for a specific PatternInferface via AutomationPeer’s GetPattern method that takes PatternInterface member as a parameter and returns the supported pattern – a derivative of BasePattern or null if the specified PatternInterface value does not correspond to a supported pattern.
BasePattern has several out of the box derivatives that target standard control patterns:
For example Button, Hyperlink and MenuItem controls support InvokePattern and can therefore indicate to the automation clients that control is capable of invoking a command. Similarly an expander control supports ExpandCollapsePattern to indicate that control is capable of expanding or collapsing content within it.
Control patterns are strictly limited and custom control patterns are not supported. The reason is that automation clients must work on a standardized protocol and cannot interpret custom patterns to figure out the functionality of the controls.
Since UI automation is optional in WPF, you may tag on automation properties on a control via attached properties specified in the AutomationProperties static class:
For example if we want to specify automation id on a text box we could specify it as following in XAML:
<TextBox Text=”{Binding Telephone}” AutomationProperties.Id=”ID_TXT_TELEPHONE”/>
Navigating the automation tree in WPF
Similar to visual tree navigation, one can navigate UI automation tree in WPF using the AutomationElement instance. However there is an important distinction – unlike the visual tree, an automation tree is not constructed upfront and is only constructed during automation tree navigation. This is so because unlike visual tree, UI automation is optional in applications. There are more than one way to navigate the automation tree depending on the view of automation tree you would want:
-
Raw view – Raw view refers to non-filtered complete automation tree view. Using TreeWalker class’ RawViewWalker property raw view can be navigated.
-
Control view – A control view is a subset of raw view that only contains AutomationElements that correspond to controls (or in other words have their AutomationElement.IsControlElementProperty property set). Using TreeWalker class’ ControlViewWalker property control view can be navigated.
-
Content view – A content view is a subset of control view that only contains AutomationElements that correspond to elements that represent information in terms of content to the end user(or in other words have their AutomationElement.IsContentElementProperty property set). Using TreeWalker class’ ContentViewWalker property content view can be navigated.
-
Custom view – Using a AutomationElement and a condition (which can be one of PropertyCondition, AndCondition, OrCondition or NotCondition) one can navigate automation tree in a custom manner. To navigate a custom view a call to the FirstFind or FindAll methods of AutomationElement is made with a tree scope (TreeScope enumeration specifying the levels in terms of Element, Children, Descendants, Subtree, Parent and Ancestors or any combination of these flags) and a condition.
The root of the automation tree is represented by static property AutomationElement.RootElement refers to the user’s desktop. One can either navigate a automation tree via RootElement or quite often more efficiently from the AutomationElement corresponding to the application window which can be obtained via AutomationElement.FromHandle method by passing in the handle to the application window.
A real world example – automating Windows calculator
As a real world example, let’s author test cases that uses UI automation to test Windows calculator. The aim of the tests is to test the following aspects of calculator:
-
Data entry verification – Validates that a number being typed appears correctly in the calculator
-
Editing options – Validates that copy and paste works as expected in the calculator
-
Calculation validation – Validates given an expression tree, calculator calculates the expression correctly.
You may download the completed example from below:
Calculator class
We will start the example by modeling Calculator class which will be our interface to Windows calculator. The calculator class shall fire up an instance of Windows calculator upon construction and shall provide methods to manipulate the calculator. Also this class shall implement IDisposable interface and shall dispose the calculator process upon test conclusion. Clean calculator instance for every test ensures no d
Following is the constructor of Calculator class:
public Calculator()
{
_calculatorProcess = Process.Start("Calc.exe");
int ct = 0;
do
{
_calculatorAutomationElement = AutomationElement.RootElement.FindFirst(TreeScope.Children,
new PropertyCondition(AutomationElement.NameProperty, "Calculator"));
++ct;
Thread.Sleep(100);
}
while (_calculatorAutomationElement == null && ct < 50);
if (_calculatorAutomationElement == null)
{
throw new InvalidOperationException("Calculator must be running");
}
_resultTextBoxAutomationElement = _calculatorAutomationElement.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.AutomationIdProperty, "150"));
if (_resultTextBoxAutomationElement == null)
{
throw new InvalidOperationException("Could not find result box");
}
GetInvokePattern(GetFunctionButton(Functions.Clear)).Invoke();
}
The code fires up calculator process and waits for automation element to become available. Then it proceeds to discovering and initializing automation elements that will be used to interact with the calculator. For example to obtain access to calculator result text box we do a lookup of automation id of 150 within the main calculator automation element. Automation Ids of elements were obtained using Spy++ utility that ships with Visual Studio by following the steps below:
-
Fire up a calculator instance and then fire up Spy++.
-
Type any number in calculator (I typed 1234) to identify it within Spy++.
-
Press Ctrl + F3 in Spy++ and from the search window drag the cross bow over to the calculator instance. This will highlight the calculator process in Spy ++.
-
Find the window instance corresponding to the pressed number and click to select it.
-
Right click on the selected window and select Properties menu item to fire up the properties window
-
Look up the control Id in the Property Inspector window. This is the automation id for the control in Hex. Convert it into decimal before using it in code. In my case a hex value of 96 corresponds to 150 in decimal. This is how I got 150 !
Dispose method simply terminate the process:
public void Dispose()
{
_calculatorProcess.CloseMainWindow();
_calculatorProcess.Dispose();
}
To obtain InvokePattern to invoke buttons we use a utility method GetInvokePattern for a specified AutomationElement:
public InvokePattern GetInvokePattern(AutomationElement element)
{
return element.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
}
AutomationElement for a function button can be retrieved via GetFunctionButton method that takes in string for the function (e.g. “Clear”) as its input:
public AutomationElement GetFunctionButton(string functionName)
{
AutomationElement functionButton = _calculatorAutomationElement.FindFirst(TreeScope.Descendants,
newPropertyCondition(AutomationElement.NameProperty, functionName));
if (functionButton == null)
{
throw new InvalidOperationException("No function button found with name: " + functionName);
}
return functionButton;
}
All the function names are defined within the Functions static class:
public class Functions
{
// Functions
public const string MemoryClear = "Memory clear";
public const string Backspace = "Backspace";
public const string MemoryRecall = "Memory recall";
public const string ClearEntry = "Clear entry";
public const string MemoryStore = "Memory store";
public const string Clear = "Clear";
public const string DecimalSeparator = "Decimal separator";
public const string MemoryAdd = "Memory add";
public const string MemoryRemove = "Memory subtract";
public const string Equals = "Equals";
}
Similarly we query for a digit button using GetDigitButton method that takes digit as an input and returns the associated AutomationElement:
public AutomationElement GetDigitButton(int number)
{
if ((number < 0) || (number > 9))
{
throw new InvalidOperationException("mumber must be a digit 0-9");
}
AutomationElement buttonElement = _calculatorAutomationElement.FindFirst(TreeScope.Descendants,
new PropertyCondition(AutomationElement.NameProperty, number.ToString()));
if (buttonElement == null)
{
throw new InvalidOperationException("Could not find button corresponding to digit" + number);
}
return buttonElement;
}
One thing to note is the use of AutomationElement.NameProperty in querying for the element. This property cannot be seen via Spy++ and I had to open the inspect the object in debugger to find it out (I used AutomationId to load an element and queried it in the debugger to find the name property).
The result of calculator can be retrieved using Result property. The setter of this property parses the character string from left to right, locates the AutomationElement for the button on the calculator corresponding to the character and then invokes it using InvokePattern’s Invoke method. This is really mimicking user typing numbers in the calculator.
public object Result
{
get
{
return _resultTextBoxAutomationElement.GetCurrentPropertyValue(AutomationElement.NameProperty);
}
set
{
string stringRep = value.ToString();
for (int index = 0; index < stringRep.Length; index++)
{
int leftDigit = int.Parse(stringRep[index].ToString());
GetInvokePattern(GetDigitButton(leftDigit)).Invoke();
}
}
}
The Evaluate, Clear and InvokeFunction methods simply evaluates (mimicking pressing of = button), clears (mimicking pressing of C button) and invokes a function respectively:
public void Evaluate()
{
InvokeFunction(Functions.Equals);
}
public void Clear()
{
InvokeFunction(Functions.Clear);
}
public void InvokeFunction(string functionName)
{
GetInvokePattern(GetFunctionButton(functionName)).Invoke();
}
FindMenu method locates the specified calculator menu and returns its ExpandCollapsePattern:
private ExpandCollapsePattern FindMenu(CalculatorMenu menu)
{
AutomationElement menuElement = _calculatorAutomationElement.FindFirst(TreeScope.Descendants,
new PropertyCondition(AutomationElement.NameProperty, menu.ToString()));
ExpandCollapsePattern expPattern = menuElement.GetCurrentPattern(ExpandCollapsePattern.Pattern) as
ExpandCollapsePattern;
return expPattern;
}
OpenMenu and CloseMenu uses FindMenu to obtain ExpandCollapsePattern for menus and then expand and collapse menus respectively. ExecuteByMenuName looks for a menu item within expanded menu and invokes the command using the InvokePattern of the menu.
public void OpenMenu(CalculatorMenu menu)
{
ExpandCollapsePattern expPattern = FindMenu(menu);
expPattern.Expand();
}
public void CloseMenu(CalculatorMenu menu)
{
ExpandCollapsePattern expPattern = FindMenu(menu);
expPattern.Collapse();
}
public void ExecuteMenuByName(string menuName)
{
AutomationElement menuElement = _calculatorAutomationElement.FindFirst(TreeScope.Descendants,
new PropertyCondition(AutomationElement.NameProperty, menuName));
if (menuElement == null)
{
return;
}
InvokePattern invokePattern = menuElement.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
if (invokePattern != null)
{
invokePattern.Invoke();
}
}
Now that we have exposed functionality of calculator that we would interact with, we can proceed and model expression tree so we can evaluate expressions in the calculator. For the purpose of testing expression evaluation we need two modes of expression computation: expression evaluation through calculator UI (calculated expression) and evaluation through code (expected evaluation). If calculated expression equals expected expression we can assert that calculator computes expression in the correct manner. This evaluation option is captured via EvaluateOption enumeration:
internal enum EvaluateOption
{
UIEvaluate,
ActualEvaluate
}
At the end level, we need to model the operands that have sub-expressions to evaluate. Operands are modeled using IOperand interface:
internal interface IOperand
{
int Evaluate(EvaluateOption option, Calculator calculator);
}
At the fundamental level, an operator is an operand:
internal abstract class Operator : IOperand
{
internal Operator(string automationName)
{
AutomationName = automationName;
}
public string AutomationName { private set; get; }
public abstract int Evaluate(EvaluateOption option, Calculator calculator);
protected virtual void InvokeOperator(Calculator calculator)
{
calculator.InvokeFunction(AutomationName);
}
}
Automation name refers to constants within Operators class which correspond to function names in calculator (i.e. AutomationElement.NameProperty values):
internal class Operators
{
public const string Negate = "Negate";
public const string Divide = "Divide";
public const string Multiply = "Multiply";
public const string Subtract = "Subtract";
public const string Add = "Add";
public const string Sqrt = "Square root";
public const string Percentage = "Percentage";
public const string Reciprocal = "Reciprocal";
}
A binary operator is represented using BinaryOperator abstract class:
internal abstract class BinaryOperator : Operator
{
public BinaryOperator(string automationName)
: base(automationName)
{
}
public IOperand Left { get; set; }
public IOperand Right { get; set; }
public override int Evaluate(EvaluateOption option, Calculator calculator)
{
VerifyEvaluationState();
int result = 0;
switch (option)
{
case EvaluateOption.UIEvaluate:
calculator.Clear();
int leftValue = Left.Evaluate(option, calculator);
calculator.Clear();
int rightValue = Right.Evaluate(option, calculator);
calculator.Clear();
calculator.Result = leftValue;
InvokeOperator(calculator);
calculator.Result = rightValue;
calculator.Evaluate();
result = int.Parse(calculator.Result.ToString());
break;
case EvaluateOption.ActualEvaluate:
result = Evaluate(Left.Evaluate(option, calculator), Right.Evaluate(option, calculator));
break;
}
return result;
}
protected void VerifyEvaluationState()
{
if ((Left == null) || (Right == null))
{
throw new InvalidOperationException();
}
}
protected abstract int Evaluate(int left, int right);
}
As can be seen if EvaluateOption.UIEvaluate is used within Evaluate function then UI automation is used to mimic the user’s input in calculator; otherwise evaluation of expression is done in code. The other Evaluate overload is implemented in derivative classes such as AddOperator, SubtractOperator, MultiplyOperator and DivideOperator. AddOperator class is as follows (rest of the operators are very similar differing only in actual computation and automation name in constructor):
internal class AddOperator : BinaryOperator
{
public AddOperator()
: base(Operators.Add)
{
}
protected override int Evaluate(int left, int right)
{
return left + right;
}
}
NumberOperator is just a number constant in expression tree:
internal class NumberOperator : Operator
{
public NumberOperator(int number)
: base (null)
{
_number = number;
}
public override int Evaluate(EvaluateOption option, Calculator calculator)
{
return _number;
}
private readonly int _number;
}
ExpressionTree class creates an expression tree from a XML stream via CreateTree method:
internal static class ExpressionTree
{
internal static IOperand CreateTree(Stream stream)
{
XDocument doc = XDocument.Load(stream);
return CreateOperend(doc.Root);
}
private static IOperand CreateOperend(XElement operandElement)
{
XAttribute type = operandElement.Attribute("Type");
IOperand operand = null;
switch (type.Value)
{
case "NumberOperator":
operand = new NumberOperator(int.Parse(operandElement.Attribute("Value").Value));
break;
default:
string qualifyingName = "CalculatorTests." + type.Value;
operand = Activator.CreateInstance(Type.GetType(qualifyingName)) as IOperand;
List<XNode> childNodes = new List<XNode>(operandElement.Nodes());
if (operand is BinaryOperator)
{
BinaryOperator binaryOperator = operand as BinaryOperator;
binaryOperator.Left = CreateOperend(childNodes[0] as XElement);
binaryOperator.Right = CreateOperend(childNodes[1] as XElement);
}
else if (operand is UnaryOperator)
{
UnaryOperator unaryOperator = operand as UnaryOperator;
unaryOperator.Operand = CreateOperend(childNodes[0] as XElement);
}
break;
}
return operand;
}
}
CreateTree calls CreateOperend method that parses the XElement’s Type attribute to determine type of operand and depending on whether type is NumberOperator (in which case it looks for Value attribute) or otherwise (unary or binary operand in which case it looks for child Operand xml elements), it creates and returns an IOperand recursively (if needed). Finally the root operand is returned. As an example the expression (6 - 1) + (7 * 9) is represented using the xml:
<?xml version="1.0" encoding="utf-8" ?>
<Operand Type="AddOperator">
<Operand Type="SubtractOperator">
<Operand Type="NumberOperator" Value="6"/>
<Operand Type="NumberOperator" Value="1"/>
</Operand>
<Operand Type="MultiplyOperator">
<Operand Type="NumberOperator" Value="7"/>
<Operand Type="NumberOperator" Value="9"/>
</Operand>
</Operand>
Now that we have successfully created the infrastructure for interacting with the calculator and evaluating the expression tree, we can start authoring test cases:
- Verify data entry of a random number in calculator and verify result shows up correctly in the UI
[TestMethod]
public void TypeRandomNumber()
{
using (Calculator calc = new Calculator())
{
int number = new Random().Next(100, 10000);
string stringRep = number.ToString();
calc.Result = stringRep;
Assert.AreEqual(stringRep, calc.Result);
}
}
- Verify cut and paste functionality:
[TestMethod]
public void VerifyCopyPaste()
{
using (Calculator calc = new Calculator())
{
string stringRep = new Random().Next(100, 10000).ToString();
calc.Result = stringRep; calc.OpenMenu(Calculator.CalculatorMenu.Edit);
calc.ExecuteMenuByName("Copy");
calc.Clear();
calc.OpenMenu(Calculator.CalculatorMenu.Edit);
calc.ExecuteMenuByName("Paste");
Assert.AreEqual(stringRep, calc.Result);
}
}
- Verification of expression tree computation from embedded xml resource files:
[TestMethod]
public void VerifyExpressionTrees()
{
string[] files = new[]
{
"CalculatorTests.Resources.SimpleNumberOperator.xml",
"CalculatorTests.Resources.SimpleAdditionOperator.xml",
"CalculatorTests.Resources.MixedOperators.xml"
}; using (Calculator calc = new Calculator())
{
foreach (string file in files)
{
calc.Clear();
IOperand expression = LoadExpressionTreeFromFile(file);
Assert.AreEqual(expression.Evaluate(EvaluateOption.ActualEvaluate, calc), expression.Evaluate(EvaluateOption.UIEvaluate, calc));
}
}
}
private IOperand LoadExpressionTreeFromFile(string resourceFileName)
{
return ExpressionTree.CreateTree(this.GetType().Assembly.GetManifestResourceStream(resourceFileName));
}
You may run the tests using Visual Studio test manager and will see calculator instances popping up and controls being manipulated as per the test cases followed by termination of calculator process (remember the IDisposable implementation in Calculator class that does the teardown).
So that’s it, you have just seen an automation client in action! Stay tuned for my next blog where I will demonstrate how to implement automation support during the development of custom controls.
You may also want to look at project White on codeplex which is based on UI automation framework and aims to simplify UI automation programming model.
