Creating a custom Writablecomparator class

Question

I’m creating an application to process distributed video in Hadoop using the paradigm map-reduce. I have several videos stored in the cluster and wish from them to create a single video file.

For this, I am extracting the frames of each video in the phase map and send them to phase reduce. Each video is numbered from 1 to 9, and the start of the key of each frame corresponds to the video that originated it, for example the key 11788847 corresponds to a frame of the video 1, on the other hand the key 57845 corresponds to the video number 5. Note that even though the key 11788847 is larger than the key 57845, it must come first so that the frames are grouped according to the video.

For this, I implemented the class SortingCustomComparator who inherits from WiritableComparator. Follow the code below:

public class SortingCustomComparator extends WritableComparator{

protected SortingCustomComparator(){
    super(Text.class, true);
}
@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2){
    // Converte a string em um inteiro
    String key1 = null;
    String key2 = null;
    try {
        key1 = Text.decode(b1, s1, l1);
        key2 = Text.decode(b2, s2, l2);
    } catch (CharacterCodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


    int keyInt = Integer.valueOf(key1.trim());
    int key2Int = Integer.valueOf(key2.trim());
    return (keyInt >key2Int) ? 1 : -1;

}  }

The problem is that the above code does not have the expected behavior. The created video contains some frames that are placed in random position. I would like the keys to be grouped by blocks. The keys of the first video must be sorted in ascending order and must belong to the set of the first video as well as that of the second one belonging to that of the second one and so on. Ex.: {11, 12, 13, 14, ..., 111, 112}, {2, 21, 22, ...}. At the end these key blocks should be ordered {11, 12, 13, 14, ..., 2, 21, 22}. So the frames will be grouped in the correct order the video will not get bugged. Would anyone know how to implement this correctly?

Already I thank you.

Answer 1

Try to change that:

int keyInt = Integer.valueOf(key1.trim());
int key2Int = Integer.valueOf(key2.trim());
return (keyInt >key2Int) ? 1 : -1;

That’s why:

return key1.compareTo(key2);

Answer 2

I solved the problem as follows:

1 - Each video starts with a number that corresponds to its source video; 2 - The source number is concatenated with the frame number that is generated in the application; 3 - I created a custom class that performs the comparison according to the code below:

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2){
    /*
     * Converte a string em um inteiro
     */ 
    String key1 = null;
    String key2 = null;
    try {
        key1 = Text.decode(b1, s1, l1);
        key2 = Text.decode(b2, s2, l2);
    } catch (CharacterCodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


    int keyInt = Integer.valueOf(key1.trim());
    int key2Int = Integer.valueOf(key2.trim());

    String strKey1 = Integer.toString(keyInt);
    String strKey2 = Integer.toString(key2Int);

    int keyFirstP1 = Integer.parseInt(strKey1.substring(0, 1));
    int key2FirstP1 = Integer.parseInt(strKey2.substring(0, 1));

    strKey1 = modifieString(strKey1);
    strKey2 = modifieString(strKey2);


    //return cmp.compare(strKey1, strKey2);

    if(keyFirstP1 == key2FirstP1){
            return (keyInt > key2Int) ? 1 : -1; 
    }

    if(keyFirstP1 != key2FirstP1){
         return (keyFirstP1 > key2FirstP1) ? 1 : -1;
    }
    return 0;

First we compare the first element of each key. If they are equal the key is fully compared, otherwise we only return the comparison between the first element of each key, thus ensuring the correct order of the frames.