Overview

This panel allows users to train and classify recordings, and see the results of the training and classifications. The recordings in the Recordings Panel are classified based on the feature settings in the Features Panel and the preferences in the Preferences Panel into the categories in the Taxonomy Panel. The panel is shown in Figure 1 below.

The main display area displays reports on training history, classification results and feature and classifier selections and weightings. These can either be overall reports or reports for individual classifiers. The contents of these reports and their formatting is controlled by the preferences in the Preferences Panel.

This panel allows users to assign different functions to some of its buttons with combo boxes. Each button that has a combo box below it is influenced by the contents of that combo box.

Buttons

• Train: Causes the system to train the classifiers using the currently loaded taxonomy, preferences, feature settings and recordings. In order for training to work, features must have already been extracted from recordings and at least one recording must be available for training with a model leaf category corresponding to all leaf categories in the taxonomy in the Taxonomy Panel. The features used depend on the settings in the Features Panel. Which particular recordings are used for training depends on the the contents of the combo box below the Train button. The Randomly Partitioned Recs option randomly selects a portion of the recordings for training (based on the value of the Fraction of Recordings to use for Training preference in the Preferences Panel) and reserves the rest for testing. The Recs Marked For Training option only uses those recordings marked with a Role of training in the Recordings Panel. The All Recs option trains on all currently loaded recordings. All types of training auto-generate the training progress and selection / weightings reports (see below). A multi-level progress bar (see Figure 2) is shown during training which gives an estimate of the training time remaining for each task and sub-task. Training can be cancelled at any time on this progress bar. The way in which training is performed depends greatly on the preferences in the Preferences Panel. Note that pressing this button causes any previously trained classifiers in memory to be entirely forgotten (existing classifiers cannot receive supplemental training after the initial classification is complete).
• Classify: Causes the system to classify a certain set of recordings based on the contents of the combo box below the Classify button. The Reserved Recs option classifies any recordings that have been reserved for testing, usually as a result of Randomly Partitioned Recs training (see above). The Recs Marked For Classification option classifies the recordings with a role of classification in the Recordings Panel. The Recs Marked For Testing option classifies the recordings with a role of testing in the Recordings Panel. The All Recs option classifies all currently loaded recordings. Classification only works if the classifiers have already been trained and the same features that were used in training have already been extracted from all recordings to be classified. Classification results reports are automatically generated upon completion of training (see below). An overall report is also generated and displayed upon completion of classification. Note that classification success rates are only shown for the Randomly Partitioned Recs option (although they can still be manually calculated otherwise). Depending on the number of recordings and the complexity of the classifiers, the classification task may take a few moments. The way in which classification is performed depends greatly on the preferences in the Preferences Panel. The user must ensure that these settings match those that were used during training.
• Cross-Validation Test: Automatically divides the loaded recordings into different folds. Each fold consists of training and testing partitions, such that each recording serves as a testing recording exactly once across all folds and as a training recording in the other folds. The number of recordings used for training in each fold is determined by the Fraction of Recordings to use for Training preference in the Preferences Panel. The Role of each recording in the Recordings Panel is ignored. Each fold is then trained and tested, and the resulting reports for each fold are saved to disk in a folder selected by the user with a file find dialog box. A combined results report is then displayed at the end. A set of reports on average normalized feature weightings is also saved to disk. The actual classifiers are not saved to disk, as the goal of cross-validation is simply to test the overall performance of the classification methodology.
• Report Training Progress: Displays reports generated during training. A report is available for each classifier ensemble, and the particular reports displayed depends on the selection in the combo box below this button. The designation "RR" at the beginning of a report name means round robin classifier ensemble, and the "DD" designation means hierarchal classifier ensemble. The contents of these reports and their formatting depend on the preferences in the Preferences Panel. These reports can include the names of all recordings used for training with their model leaf categories and the training progress (e.g. fitness, sum of squares error) for each classifier or selector/weighter in the appropriate classifier ensemble from iteration to iteration.
• Report Classification Results: Displays reports generated during classification. A report is available for each classifier ensemble, and the particular reports displayed depends on the selection in the combo box below this button. The designation "RR" at the beginning of a report name means round robin classifier ensemble, the "DD" designation means hierarchal classifier ensemble and the "Combined Classifier" refers to the results when the results from all classifier ensembles are combined. The contents of these reports and their formatting depend on the preferences in the Preferences Panel. These reports can include success rates and confusion matrices and their associated standard errors (these are only available with reserved recordings), a list of all eligible leaf categories, the training and classification options used, the individual classifier and combined ensemble classification results for each recording (including score, winning categories, second choices and associated model categories)), the complete listing of classification scores for each category for each recording for all individual classifiers as well as the entire classifier ensemble and the model results for each recording.
• Report Selections / Weightings: Displays feature/classifier selection/weighting reports generated during training. A report is available for each classifier ensemble, and the particular reports displayed depends on the selection in the combo box below this button. The designation "RR" at the beginning of a report name means round robin classifier ensemble, and the "DD" designation means hierarchal classifier ensemble. The contents of these reports and their formatting depend on the preferences in the Preferences Panel. These reports can include the particular one-dimensional features used and their associated weightings and the particular classifiers used and their associated weightings.
• Save Reports: Saves all available reports to separate text files in a directory created using a file chooser dialog box.
• Load Classifiers: Loads a set of trained classifiers, reserved testing recordings (if any) and selection/weighting reports contained in files pointed to by a classifiers_file XML file. Allows the user to choose the file using a file chooser dialog box. Note that the user must manually ensure that the training preferences in the Preferences Panel match those that were used during training of the loaded classifiers. It is not, however, necessary that reserved training recordings still be in the recordings list in the Recordings Panel. Please note that loading can take a few moments for complex classifiers.
• Save Classifiers: Saves the currently available trained classifiers, reserved testing recordings (if any) and selection/weighting reports into files pointed to by a classifiers_file XML file on disk. If a path is given in the File holding trained classifiers field of the Configure File Locations Dialog Box, then automatically saves the file to that path, overwriting any existing file. If this field is blank, then allows the user to choose the file location using a file chooser dialog box. Please note that saving can take a few moments for complex classifiers.
• Save Classifiers As: As: As above for the Save Classifiers button, but bypasses the contents of the Configure File Locations Dialog Box and allows the user to choose the file location using a file chooser dialog bo x..

Status Bar

Displays whether or not trained classifiers are available.

Warnings

• The way in which training and classification is performed depends greatly on the preferences in the Preferences Panel. The user must ensure that settings during classification match those that were used during training. Exceptions are the Classify Into Root Categories, Classify Into Direct Descendants of Parents, Classify Into All Leaf Categories and Classify Using Round Robin Ensembles, which can differ if fewer classifiers are to be used in classification than were used during training.
• No changes should be made to the taxonomy in the Taxonomy Panel once classifiers have been trained.

Screen Shots

Figure 1: Classification panel showing a classification report.

Figure 2: Progress bar showing progress of classifier training.