Alerting systems are being given new technological capabilities that raise novel issues in their interaction with the operator, requiring new methods of testing in order to accurately assess and predict their ultimate impact on operational performance. This paper reviews operational problems with alerting systems, and outlines difficulties in testing them. A range of effects which should be examined during the testing and implementation of alerting systems are detailed, with an emphasis on understanding the total performance of the joint human-alerting system combination. Methods for testing alerting systems beyond the most-common tightly-controller laboratory experiments are detailed, using recent studies as examples.