A Groupware for Pigment Identification in Cultural Heritage

The preservation of cultural heritage is a multidisciplinary field of study that is aimed at analyzing and preserving ancient artifacts. Nowadays, cultural heritage investigators can make use of available non-invasive solutions able to provide useful information about materials, techniques, and retouches. Yet, investigations on artworks are manually performed with very little automation. This is due to the need for the unique expertise of human operators but also to the lack of adequate digital support. To this aim, we have designed a groupware for supporting collaborative work among cultural heritage investigators even when located far from each other. As a practical case study we have chosen the pigment identification problem in artworks and included in the groupware an automated procedure for selecting investigation areas and for matching unknown pigments on the considered artwork with known standards in a database.

be drawn from the study are not that different from what an investigator would have done tens of years ago.The computational power of current computers, the artificial intelligence or neural network capabilities of state-of-the-art algorithms, and the interaction capabilities offered by today's social network technologies are rarely exploited in this field.
Clearly, we are not aiming at substituting the unique expertise of a human operator, yet, a digital support could be of great benefit to help investigators in their diagnosis, or as a second opinion, or just as a powerful and scalable means to organize information and to foster interaction among researchers.
Therefore, we have designed a groupware (i.e., a collaborative software [6]) that enables computer supported collaborative work (CSCW) among cultural heritage investigators even when located far from each other [7].As a practical case study, we have chosen an important problem in cultural heritage preservation, i.e., pigment identification, and in this work, we present the design and technical analysis of a new integrated system that we have devised.In particular, our solution provides artworks' investigators with: i) possibility to work both offline on a PC or online through a web applet, ii) support for the digital acquisition of data from the image, iii) objective interpretation of gathered data, and iv) integration of a standards' database with a groupware designed to facilitate investigators' discussion.
The rest of the paper is organized as follows.In Section II we review background information on the scientific analysis of paintings that we are considering as case study.Section III provides a high level design of the proposed system, whereas Section IV discusses important technical details about it.Finally, Section V concludes this paper.

II. FALSE-COLOR IR IMAGING: BACKGROUND
The IR reflectography is a non-invasive technique suitable for the analysis of paintings.In essence, when hit by UV and IR radiations, painted surfaces reveal chemical properties of the pigments unveiled when hit by regular light (i.e., visible radiations).
The false-color IR method is a specific technique that tries to unveil hidden information related to the chemical property of a pigment and hence also to its nature.To achieve this result, it combines RGB color images and IR reflectograms to create a false-color image (or pseudo-color image) where each pigment is represented by a specific false-color that depends on its type of interaction with the IR light; this interaction is strictly related to the chemical composition of the pigment [2], [3].
This method needs as input both the RGB (red, green, blue) color image and the grey scale IR reflectogram, and combines them to generate a new grey scale image.In this new grey scale image, information about the blue color in the RGB is discarded; only the red and green components are overlapped with the greyscale IR component.Furthermore, the red component in the RGB image is substituted with the IR, the green with the red, and the blue with the green (Fig. 1).
As said, resulting false-color images present pseudo-colors closely related to the chemical composition of the pigments, thus allowing the characterization of pigments with similar hue under visible light; furthermore, the different degree of IR radiation reflected by each pigment may also depend on the painting technique used by the artist (i.e., color, composition, and layers on the painted surface).
In simple words, the darker areas in the painting in Fig. 1 represent different pigments utilized on the artwork.For instance, a very similar light blue color can be made by both the rare and expensive lapis lazuli blue pigment and by a much less expensive azurite.The former was used by the famous artist that created the artwork, whereas the latter was used by a restorer years later.Therefore, through the analysis of a false-color image, an investigator can determine which areas of a painting are original and which ones have received retouches.
Unfortunately, the interpretation of the false-color IR image is currently carried out just by direct examination of an investigator that visually identifies all the areas of interest in the painting and compares them with the outcomes of standard pigments of known composition (standards).Even if this is the procedure generally used, results could be uncertain or even contradictory due to individual limits and experience.For instance, an operator could miss some little area in the painting that deserved to be investigated, or false-color outcomes of two pigments could seem similar thus leading to an ambiguous association of the painting area under examination to a pigment.
Instead, computer science technology could be put to good use to support cultural heritage investigators.For instance, algorithms could be designed to identify artworks' area of interests for investigators and to augment the accuracy of pigments' analysis.Moreover, Internet communications could be exploited to generate a groupware, i.e., a computer-based system that provides a shared environment to a group of remote operators engaged in the task of analyzing a painting [2].

III. A GROUPWARE FOR PIGMENT IDENTIFICATION
We present the design and technical analysis of a new integrated system aimed at supporting investigators in pigment identification in artworks through false-color data.In particular, our solution intend to provide artworks' investigators with: i) possibility to work both offline on a PC or online through a web applet, ii) support to the digital acquisition of data from the image, iii) objective interpretation of gathered data, and iv) integration of a standards' database with a groupware designed to facilitate investigators' discussion.First of all, we describe the general functioning of the system with the help of Fig. 2. As the figure shows, the system is composed by both hardware and software instruments employed for the digital acquisition of false-color IR images.It includes the camera and filters utilized to take digital images, both in color and in infrared (VIS and IR in Fig. 2, respectively), of the considered painting (step A).Since the short distance between the painting and the camera when this operation takes place, each of the digital images covers only a limited section of the painting.Several single images have hence to be taken to have the whole artwork in a digital form.
All of these single images have then to be merged to generate complete digital copies, both for VIS and IR modes, of the whole original painting (step B).This task can be done automatically by the means of existing software and algorithms that exploit the overlapping area of each picture to match them as pieces in a puzzle [8]- [11].
Having the two digital copies, VIS and IR images, another software tool is then responsible to combine the two images into a single one through the false-color technique (step C).The false-color IR image (FC) is the subject of the investigators' analysis.
Needless to say, all of these images represent a patrimony for the scientific community that has to be preserved.They are hence stored in good order in a database (called Images in Fig. 2).Organizing the false-color images is only one aspect of the investigator's work.She/he would also like to annotate certain portions of the image with information regarding the pigment used.The first step is that of highlighting the portions of the artwork which are significant under this point of view.To do so, another piece of software is responsible for allowing the operator to select a certain area within the false-color IR image and automatically highlight all the areas that would have the same appearance under a human eye (step D).
A more comprehensive analysis on the artwork could be performed enlarging the group of investigators and adding some information technology aid to support their work.To this aim, Fig. 2 shows that our system includes a groupware that generates a cooperative work environment to foster the interaction and discussion among investigators.Furthermore, it also includes a module designed to provide computer-based suggestions based on past classifications that can help investigators in their evaluation.
More in detail, the operator selects the areas she/he intends to highlight; the resulting study image will be a combination of the original image and some highlighted areas.This study image is saved and visible through the application (step E in Fig. 2) Each of the emphasized areas of the study image is now clickable in order to access other information related to that area, including the comparison of the colors of a certain area with a database of pigments standards (step F in Fig. 2).
Clearly, to push further the collaboration among investigators, the groupware should be accessible via web; we have included this option in the design of our system.Therefore, the pigment identification system can be run both offline and online; in the latter case, when clicking an emphasized areas of the study image, another web page related to that area is loaded.All the areas belonging to the same group are linked to the same web page.
A web page related to a certain area of the study image represents both a source of information about the pigment used by the author for that area and a virtual environment for cooperative work and shared discussion among investigators (this is indicated as FORUM in Fig. 2).

IV. TECHNICAL ANALYSIS OF THE GROUPWARE
In this section we discuss the technical options and challenges related to the main components of our groupware.

Offline vs. Online Work
Two approaches can be considered when designing the way an operator has to interact with our application.A first approach (i.e., offline) is a classic one: an operator sits in front of her/his computer and runs a locally installed program.Alternatively, another approach (i.e., online) is that of having a web applet running on a remote server and interacting with it through the Internet.
Both solutions have advantages and disadvantages.In the offline case, we have a single application that has the capability of acquiring the image, analyzing it, highlighting certain areas, providing a software based interpretation of the pigments, and annotating the image.The discussion among investigators has hence to pass through having one of them providing in some way her/his study images to others.This file transfer can be done through an Internet connection or through some portable mass storage (e.g., a flash drive memory, a CD, a DVD, a portable hard disk); in any case, both the investigators need to have the main application for handling the study images installed on their computers.
Therefore, the main advantage of operating through an offline application is that even when an Internet connection is not available, the investigator can work on the image.Moreover, study images are usually huge TIFF files that can reach upto 100 MB of size; it is hence much easier to work on them while they are stored in a local hard disk, without the need of continuous downloads and uploads from/to a remote server.
On the other hand, an online version of the considered application would allow a much higher degree of interaction among investigators that could operate simultaneously on the same painting and exchange in real time opinions about it.A single version of the study image, including the annotations of all the operators, may be stored in the system, thus having a consistent progression of the cooperative analysis work as seen by the whole research community.Furthermore, to participate to the analysis and discussion of a certain painting, or to utilize any other feature of the system, an operator has simply to connect to the Internet and use her/his favorite browser without the need of obtaining and installing a specific software.
In summary, both operative paradigms have pros and cons with respect to each other.We deem that the best solution is that of having the possibility of working both offline and online, with replicated functionalities so that each investigator can choose which modality fits most her/his needs.At the same time, this dual solution adds some complexity, as the two systems have to be able to cooperate and synchronize.For instance, an investigator may work offline on a certain painting and then decide to upload her/his work on the online version so as to broaden the group of investigators participating to the analysis, or even just to be able to keep working on her/his analysis even from other computers that do not have the program installed.

Handling Image Resolution and Layers
As already mentioned, study images may be of the order of 10 to 100 MB.Therefore, efficient elaboration of the image becomes crucial; this is even more true if we consider the online version of the application: in this case, it would be impossible to continuously download and upload such file for every single operation performed by an investigator.At the same time, we need to provide online investigators with the highest possible resolution of the investigated area; otherwise, they would not be able to perform their task.
A possible way to solve this conundrum is represented by adopting a technique similar to what Google does for its Google Maps service.In essence, when watching a whole image (i.e., the world in Google Maps, a painting in our case) on a screen, a very high resolution is not necessary as no difference would be noticeable with respect to using a lower resolution.
As long as we zoom in on an image, then we could appreciate higher and higher resolutions.At the same time, only a smaller portion of the original image is now visible within the limited frame of a screen (see Fig. 3).Therefore, to find the best tradeoff among high resolution and low byte size, the original high resolution image can be divided into blocks.These blocks have the same high resolution but, as they are just chunks of the original image, they have a smaller physical and byte size.
When an investigator zooms in on a certain area of the image, only few chunks of the high resolution image belonging to the zoomed area need to be transferred.Clearly, these chunks are locally cached for future use.They hence need to store information such as download time and last modification time.
An investigator may also decide to highlight a certain area of the image and to add notes (e.g., the possible pigment composition).Instead of creating a new image also including this information, layers can be used as done by popular photo editing software such as Photoshop and GIMP [12].
In essence, a layer is an image with some transparent parts and some semitransparent ones (the highlighted areas).This layer can be superimposed on the existing image to show as highlighted a certain area.This solution allows to add new information to the image with little bytes overhead.Furthermore, multiple layers can be simultaneously combined to create complex compositions in a flexible and efficient way.

An "Intelligent" Selection Aid
An important tool designed to support the investigator is represented by the selection aid.When an investigator selects a certain area within the false-color IR image, this tool takes care of highlighting the area.Moreover, it may also highlight other areas on the painting that possess the same pigment features.This option, as well as the threshold used to determine the boundaries of the highlighted area(s), can be set by the investigator while using the program.
More in detail, the automatic selection of areas to be highlighted is not that trivial if we want to create a tool that is able to interpret the image as a human eye would do.Indeed, we would like to have a selection aid able to mimic a human eye (and brain), rather than precisely consider each tiny single pixel.To this aim a possible solution can come from a technique known as photomosaic (or photographic mosaic) [13]- [15].
An image obtained through photomosaic is made up of many smaller, even completely different, tile images.Taking a close look, one can see all the small tile images; however, when seen together at a distance, they suggest to a human eye an entirely different and bigger image (Fig. 4).This is due to how humans perceive images and to the well known property by which we look with our eyes but we see with our brains.Photomosaic exploits this principle and transforms an input image into a grid of thumbnail images preserving the overall appearance.More in detail, a photomosaic algorithm logically divides an image into a matrix of pixel blocks.Then, the algorithm delves into a database of thumbnail images to extract those ones that best approximate the various pixel blocks composing the main image.This can be done by simply matching the average RGB color, luminosity, and contrast of the candidate tile images with each considered block of pixel in the original one.

Fig. 3 .
Fig. 3. Low resolution version of the whole image (above); when zooming in, higher resolution sections are transferred (below).

Fig. 6 .
Fig. 6.Example of use of the groupware integrated with the standards' database.