Proposer: Sebastian Riedel, +44 131 6681457, s.r.riedel@sms.ed.ac.uk)
Self-Proposed: Yes
Supervisor: Ewan Klein (ewan@inf.ed.ac.uk)
Other Suggested Supervisors: Dave Robertson, Miles Osborne
Subject Areas: Distributed Systems, Knowledge Based Systems, Machine Learning/Neural Networks/Connectionist Computing, Natural Language Processing,
Suitable for the following degrees: MSc in Informatics,
Principal goal of the project: Design a Multi-Agent-System in which NLP classifiers are modelled as collaborating agents, autonomously doing feature selection, training and tweaking.
Description of the project:
Introduction:
Today there is a strong tendency within NLP to improve classification results by having classifiers collaborating in various forms. For instance, we see classifiers working in voting ensembles, teaching each other in co-learning approaches or in stacks using outputs of other classifers as features. Even the general use of features can be seen as a hierarchical collaboration of a classifiers, with feature extraction algorithms being generalized classifiers.
In fact, last year's best performing classifier in the CoNLL shared NER task was explicitely trying to exploit classifier combination and also used another already trained NER classifier as input for their system. Furthermore subsequent to the task an even better classifier was created by combining the five best systems in terms of accuracy and precision[1].
Given this observations, the design space for optimal classifiers seems to be vast but worthwhile exploring. In addition, it is also expanding in terms of new algorithms, collaboration schemes and annotated datasets to train on. Searching this space is a tedious process, requiring the researcher to prepare data, select features, boost, train, set up ensembles, evaluate performances and most likely redo the whole process with different configurations.
In addition, the search process is also computationally very demanding as it usually requires iterative retraining of potentially very large models.
A Multi-Agent world of classifiers as collaborating agents could be a means to search this space semi-automatically in a distributed manner. Researchers could autonomously deploy new classification agents (and maybe collaboration schemes in form of matchmaking agents) to see how they evolve and perform within a 'world of classifiers'. Once deployed a classifier will learn and improve itself autonomously. In this setting, a researcher won't have to be concerned about setting up the data, selecting features or trying different collaboration schemes to boost performances.
An example scenario could look like this: a newly deployed NER classifier in Edinburgh would ask a matchmaking agent in San Diego for a suitable feature to start with and might be referred to a POS Tagger in Oxford. Together they might literally move to Manchester, where, according to another broker, cpu resources might be available. Before training begins they have to stream suitable NER data to their location, which is provided by, say, researchers in Saarbruecken. Agents architectures seem to be suitable for this task as they account for the autonomous, distributed and open character of the presented scenario.
Proposed Research:
This research will focus on realising the basic processes in creating and teaching a NLP classifier by the means of communicating agents in a Multi-Agent enviroment. For a start, this will include selecting features as well as training, tweaking and testing on data. Other schemes might be included later if time allows this.
The project could be conducted in the following timeline:
1. Identify requirements of the system to design. This could include a closer look on general NLP workflows.
2. Design an initial system including agent types, means and rules of communication according to the identified requirements. My initial guess is a sort of brokerage/publish and subscribe approach within the agent system, with classifiers publishing their capabilities as well as their requirements to some matchmaker agent which also might learn to make matches by experience. Another option might be the use of genetic algorithms to improve the population of classifiers.
In addition a suitable technical platform has to be found. One possibility could be the Jade (http://sharon.cselt.it/projects/jade/) agent framework and middleware.
One should also have a look on previous approaches to NLP using Agent Systems[2], although different in nature.
3. Implementation of feature selection, training, tweaking and testing on data as processes within the system designed in 2. Task 2 and 3 are most likely to be very correlated and boundaries might blur. 4. Populate system with data, classifiers and feature extractors in order to solve the 2003 CoNLL shared NER task. Classifiers might be implemented by wrapping around Weka algorithms.
5. Evaluate and analyse the performance of the system, compare it to results achieved by other NER systems during the shared task.
6. Write up the disseration.
Resources Required: CoNLL 2003 NER Datasets (already available), Classifer implementations (Weka framework might be suitable, although a bit slow), Dice machine(s)
Degree of Difficulty: difficult
Background Needed: Data Intensive Linguistics, Learning From Data 1, some software engineering skills
References: