ace
Class Experimenter

java.lang.Object
  extended by ace.Experimenter

public class Experimenter
extends java.lang.Object

Finds the best classification method for the given instances.

Finds the best classification approach by making repeated calls to the CrossValidator class. Different types of Classifiers are tested with multiple types of dimensionality reduction. Dimensionality reduction is performed first. Experimenter calls DimensionalityReducer to get an array of Instances objects, each cell containing a different dimensionality reduced version of the original instances. Each type of Classifier is cross validated with each set of dimensionality reduced instances. Results are saved in multiple files. One file will be created for each set of dimensionality reduced instances. A summary of the results for each cross validation for each dimensionality reduction will be written in the file for the corresponding set of instances. The results for the best found Classifier for each dimensionality reduction are printed at the beginning of each file. The best classification overall is chosen by comparing the best results for each dimensionality reduction. A copy of the results summary for the best found classification approach will be written in a separate file called "experimentation_results_best_results_overall.txt" After the best classification has been chosen, validation is performed using a publication set that was set aside at the beginning of the experiment. A new Classifier of the chosen type is created and trained on the chosen type of dimensionality reduced instances (all instances are included except for the publication set). The newly trained Classifier is tested on the publication set and results are saved to the "experimentation_results_best_results_overall.txt" file.


Constructor Summary
Experimenter(java.io.OutputStream out)
          Constructs an instance if an Experimenter.
 
Method Summary
 java.lang.String crossValidateMultiApproaches(int folds, weka.core.Instances instances, java.lang.String results_base_file_name, boolean save_intermediate_arffs, boolean verbose, int max_attribute, java.lang.String[] instance_names)
          Apply a variety of dimensionality reduction and classification techniques to the given instances.
static void resetClassifier(TrainedModel trained)
          Instantiates a new instance of the Weka Classifier contained in the given TrainedModel.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Experimenter

public Experimenter(java.io.OutputStream out)
Constructs an instance if an Experimenter.

Parameters:
out - Progress reports are sent here as the experimentation runs.
Method Detail

crossValidateMultiApproaches

public java.lang.String crossValidateMultiApproaches(int folds,
                                                     weka.core.Instances instances,
                                                     java.lang.String results_base_file_name,
                                                     boolean save_intermediate_arffs,
                                                     boolean verbose,
                                                     int max_attribute,
                                                     java.lang.String[] instance_names)
                                              throws java.lang.Exception
Apply a variety of dimensionality reduction and classification techniques to the given instances. Results are calculated using cross-validation. A report of the details of the best performing approach are returned. A file issaved for every type of dimensionality reduction performed. Each file gives the results for a variety of Classifiers.

Apply a variety of dimensionality reduction techniques in order to get several sets of instances, each with different features selected. Acquire information about the names of the feature selectors, the features they selected and how long each took.

Note that progress is printed to given OutputStream. Results are saved to files with a default base file name of "experimentation_results" unless another file name was specified using the -sres flag at the command line.

Parameters:
folds - The number of folds to perform cross- validation over.
instances - The instances to use for performing cross- validation.
results_base_file_name - The basic file path to use for saving results files. No files are saved if this is null.
save_intermediate_arffs - Whether or not to save training data to an arff file at various intermediate stages. Useful for testing.
verbose - Whether or not to include detailed information in the experimentation results.
max_attribute - If the number of attributes for a data set is larger than this number, an exhaustive search will be not be performed.
instance_names - The unique identifiers for the given set of Instances. This is used when outputting the detailed results of the validation when the verbose option is specified.
Returns:
A report detailing the best approach found.
Throws:
java.lang.Exception - If an error occurs.

resetClassifier

public static void resetClassifier(TrainedModel trained)
                            throws java.lang.Exception
Instantiates a new instance of the Weka Classifier contained in the given TrainedModel. This method is called when validating the best found classification approach by testing the publication data set. The best found Classifier needs to be reset and retrained to test the publication set.

Parameters:
trained - The TrainedModel with reference to the best found classification approach for this experiment.
Throws:
java.lang.Exception - If an error occurs.