My commercial research is not listed here, as I am limited by non-disclosure agreements.
SIMSSA: The Single Interface for Music Score Searching and Analysis Project
SIMSSA (Single Interface for Music Score Searching and Analysis) is a major long-term international research project involving a large number of institutions and millions of dollars of grant funding. The primary goal of the project is to teach computers to recognize and understand the symbols in musical manuscripts archived at libraries and museums around the world. The resultant data is then to be assembled on a single website, making it possible to easily search and analyze the online scores. SIMSSA will thus create an architecture for processing music documents, transforming vast music collections into symbolic representations that can be searched, studied, analysed, and performed anywhere in the world. This involves two main research axes:
- Content: Addresses the process of creating optical music recognition (OMR) systems for transforming digital images of scores into searchable symbolic notation.
- Analysis: Addresses the creation of tools and techniques for large-scale search and analysis of the scores after they have been converted into symbolic representations.
My own primary role in this research is in the Analysis Axis, where I will focus on applying music information retrieval (MIR) techniques to the symbolic content in order to arrive at meaningful statistics that can be used to characterize and organize music. In particular, this will involve expanding and adapting the jSymbolic and ACE components of the jMIR suite in order to take advantage of and integrate their feature extraction and machine learning functionality. This work will also be combined with work done as part of the Music Information, Research, and Infrastructure (MIRAI) program.
jMIR: General-purpose standardized software for music information retrieval research
jMIR is an open-source software suite for use in music information retrieval (MIR) research. It can be used to study music in both audio and symbolic formats as well as mine cultural information from the web and manage music collections. jMIR also includes software for extracting features, applying machine learning algorithms and analyzing metadata.
The primary emphasis of jMIR is on providing software for general research in automatic music classification and similarity analysis. The main goals of the project are as follows:
- Make sophisticated pattern recognition technologies accessible to music researchers with both technical and non-technical backgrounds.
- Eliminate redundant duplication of effort.
- Increase cooperation and communication between research groups.
- Facilitate iterative development and sharing of new MIR technologies.
- Facilitate objective comparisons of algorithms.
- Facilitate research combining high-level, low-level and cultural musical features (i.e. symbolic, audio and web-mined features).
More information on jMIR is available on the jMIR SourceForge page. There are also many publications on the jMIR components available in the publications section of this web site.
Data Mining and Machine Learning
- ACE: Pattern recognition software that utilizes meta-learning. Evaluates, trains and uses a variety of classifiers, classifier ensembles and dimensionality reduction algorithms based on the needs of each particular research problem.
- ACE XML: Standardized file formats for communicating information between the jMIR components or with external applications.
- jAudio: Software for extracting low and high-level features from audio recordings.
- jSymbolic: Software for extracting high-level features from MIDI recordings.
- jWebMiner: Software for extracting cultural features from web text.
- jLyrics: Software for extracting features from transcriptions of lyrics.
Education and Audio Production
- jProductionCritic: Educational software for automatically finding technical redording and production errors in audio files.
Data and Metadata
- jSongMiner: Software for automatically identifying songs and extracting metadata about them from various sources on the web and elsewhere.
- lyricFetcher: Software for mining transcriptions of lyrics from the Internet.
- jMIRUtilities: Software for performing miscellaneous tasks relating to preparing musical data for processing by the jMIR components.
- jMusicMetaManager: Software for profiling music collections and detecting metadata errors and redundancies.
- Codaich, Bodhidharma MIDI and SLAC: Labeled MP3 and MIDI data sets for training, testing and evaluating MIR systems.
- Bodhidharma: MIREX 2005-winning software for classifying MIDI recordings by genre. The ancestor of ACE and jSymbolic.
Musical similarity analysis
This research examines the notion of musical similarity from both an applied and theoretical sense. Software is being developed that can automatically cluster and segment musical recordings based on similarity. This will be integrated with the jMIR project. Experiments will be performed with both supervised and unsupervised learning, and both audio and symbolic data are being considered.
There are four primary tasks involved in this research:
- The development of large libraries of low-level signal processing oriented features, high-level musical features and web-based cultural features. These task are respectively related to the jAudio, jSymbolic and jWebMiner software systems.
- The implementation of sophisticated pattern recognition, classification and clustering algorithms that can be used to group segments of music in arbitrary ways based on their features. These techniques must be able to successfully deal with the inconsistencies and irregularities that characterize the ways in which many humans organize music. This is related to the ACE software system.
- The development of a statistical analysis system that seeks to find meaning in the particular organizations produced by the classification and clustering systems.
- The development of an easily usable interface that allows users with needs as diverse as musicologists, psychologists, musical database administrators, librarians and average listeners to automatically organize music in whatever arbitrary ways they wish, as well as study what musical characteristics lead to particular organizations.
Such research could, for example, be used to classify or identify music based on compositional or performance style, to search for unknown music that a user might like based on examples of what he or she is known to like, to group music based on when a user might want to listen to it (e.g. while driving, while eating dinner, etc.), to perform similarity analysis for copyright purposes and to perform content-based searches of on-line databases. The following course paper outlines some of the exploratory research underlying this project:
McKay, C. 2005. Automatic music classification and similarity analysis. Course Paper. Université de Montreal, Canada.
Real-time audio transcription for interactive performance systems
This research involves developing a system for extracting control information regarding pitch, rhythm, dynamics and timbre from polyphonic audio signals in real time. This project is being worked on in the context of developing a standardized approach for parameter acquisition that addresses problems relating to the longevity, distribution and robustness of interactive accompaniment systems. The research is initially concentrating on electric guitar music, although it is hoped to eventually generalize the system to any monophonic or polyphonic instrument. A PowerPoint presentation is available from the MGSS 2005 Symposium describing some of the key ideas of the project, and this initial paper has been published outlining the priorities of such a standardized parameter extraction system:
McKay, C. 2005. Approaches to overcoming problems in interactive musical performance systems. Presented at the McGill Graduate Students Society Symposium.
The following course paper describes several transcription techniques that are being considered:
McKay, C and W. Hatch. 2003. Transcriber: A system for automatically transcribing musical duets. Course Paper. McGill University, Canada.
Optical recognition of medieval musical manuscripts
This research involves processing digital scans of medieval scores in order to convert them to symbolic musical data. Some basic work has been done on bleed through removal and on horizontal line detection, but the project is still in the initial stages of development and there is much to be done. This is a project of Prof. Ichiro Fujinaga's, and is currently being carried out by Laurent Pugin and John Ashley Burgoyne using Aruspix and Gamera. More information can be found on the DDMAL home page.
Modeling electric, acoustic and classical guitars, including slide guitar
This research involves using physical modeling techniques to synthesize electric, acoustic and classical guitars. A particular area of emphasis is modeling the effect of playing electric and acoustic guitars with a slide. This research also involves the development of a mapping system to enable humans to effectively manipulate the parameters of the physical model with a sensor-based controller. This is linked to the slide guitar inspired hyper-instrument described below. Some initial research on physical modeling of guitars is described in the following course paper:
McKay, C. 2003. A survey of physical modeling techniques for synthesizing the classical guitar. Course Paper. McGill University, Canada.
Slide guitar inspired hyper-instrument
This project involves building a hyper-instrument based on the paradigm of a slide guitar. Each "string" is simulated using band sensors able to perceive both position and pressure, therefore adding another dimension of control that is not available on traditional guitars. The right hand manipulates a series of pressure sensitive touch sensors. Only one string is implemented in the current prototype, but extensions to this are planned in the future. A number of software mappings in Max/MSP have been implemented in order to enhance learnability and flexibility. It is the eventual goal of this project to link this instrument to a physical modeling synthesis system, as described above. The motivation and details of the system are described in the following course paper:
McKay, C. 2002. The eDobro: An implementation of design strategies intended to diversify and expand the use of computer music controllers. Course Paper. McGill University, Canada.
Emotion and music
This research involves studying the ways in which humans perceive emotion in music. A cross-cultural empirical approach is being emphasized. Some initial research is described in the following course paper:
McKay, C. 2002. Emotion and music: Inherent responses and the importance of empirical cross-cultural research. Course Paper. McGill University, Canada.
-top of page-