Creating and Managing a large annotated parallel corpora of Indian languages

TL;DR

This paper discusses the development of a web-based annotation tool for creating and managing large parallel corpora of Indian languages, addressing challenges of consistency, scalability, and distributed collaboration.

Contribution

It introduces ILCIANN, a novel annotation tool designed for POS tagging and corpus management across multiple Indian languages and institutions.

Findings

Successful creation of large parallel corpora for 12 Indian languages

Effective management of annotation process across diverse users and locations

Enhanced consistency and standards in corpus annotation

Abstract

This paper presents the challenges in creating and managing large parallel corpora of 12 major Indian languages (which is soon to be extended to 23 languages) as part of a major consortium project funded by the Department of Information Technology (DIT), Govt. of India, and running parallel in 10 different universities of India. In order to efficiently manage the process of creation and dissemination of these huge corpora, the web-based (with a reduced stand-alone version also) annotation tool ILCIANN (Indian Languages Corpora Initiative Annotation Tool) has been developed. It was primarily developed for the POS annotation as well as the management of the corpus annotation by people with differing amount of competence and at locations physically situated far apart. In order to maintain consistency and standards in the creation of the corpora, it was necessary that everyone works on a common platform which was provided by this tool.