Image classifiers are a staple in deep learning, particularly within the realm of computer vision. While the theoretical frameworks for these models have been thoroughly explored, real-world applications present unique challenges that require innovative solutions. One such challenge lies in classifying gestures captured in real-time, where dealing with unknown or unseen gestures becomes critical.

Challenges in Real-Time Gesture Classification

Our team recently embarked on a project to develop a real-time gesture classifier using webcam footage. The primary objectives included creating a neural network capable of recognizing specific gestures and accurately identifying when no valid gesture is shown.

The key challenges included:

• Insufficient training data.
• Absence of an explicit 'unknown' category.

While data scarcity is a common hurdle in deep learning, it can often be mitigated through data augmentation techniques, such as altering perspectives, exposure, and lighting conditions. However, the second challenge required a novel approach to effectively handle unknown gestures, ensuring that the classifier could acknowledge that no meaningful gesture was being performed.

Conventional methods for handling unknowns typically involve adding an additional category trained with random, unrelated images. This approach, although somewhat effective, is not comprehensive enough to cover the unknown category thoroughly.

The Importance of Handling Unknown Gestures

A classifier that does not explicitly account for unknown categories can result in misleading confidence levels for incorrect predictions. This scenario is highlighted by comparing confusion matrices from two different validation sets—one with known categories and one including unknown categories. The latter reveals how deceptive validation results can be when unknown inputs are mistakenly classified as known categories.

Understanding Classifier Loss Functions

To address the issue of classification with unknown categories, it is essential to understand the function of loss functions in neural networks. These functions measure how well the network's predictions align with the actual data and guide the model's learning process.

Standard Categorical Cross-Entropy

The standard approach uses the categorical cross-entropy loss function. This function compares predicted values to actual values by converting the output through a softmax function and using one-hot encoding for target labels. While effective for single category predictions, this method falls short in multi-label or 'no category' scenarios.

Transitioning to Binary Cross-Entropy

To improve the classifier's ability to handle multiple categories or none at all, we switch from softmax to the sigmoid activation function and apply binary cross-entropy individually. This approach treats each category prediction as an independent probability, allowing the model to determine the likelihood of each category occurring.

Incorporating an 'Unknown' Category

To address unknown gestures, it is crucial to distinguish between three types of categories within the dataset:

• Known Categories: Standard categories the network is trained to recognize.
• Unknown Categories: Additional categories labeled as 'unknown' to indicate they should not be classified separately.
• Unseen Categories: Categories used only in validation to assess the network's ability to recognize unknown inputs.

Initially, adding an 'unknown' category (labeled 'na') to the dataset and converting it to a one-hot encoded vector can improve results. However, a more sophisticated approach involves modifying the loss function to better handle unknown categories by training the network to predict vectors of zeros for such inputs.

Modified Loss Function Implementation

# Convert input using sigmoid activation
input = input.sigmoid()

# Apply one-hot encoding to the target
target = F.one_hot(target, input.shape[1]).float()

# Set all target 'na' categories (index 0) to zero
target[:, 0] = 0

# Calculate binary cross-entropy loss
loss = F.binary_cross_entropy(input, target)


This modification significantly improves accuracy for unknown categories while maintaining reasonable accuracy for known categories, marking a substantial step toward better classifier performance.

Evaluation Metrics

Lastly, it is vital to adjust evaluation metrics to interpret the model's output correctly. For example, updating the accuracy function to consider vectors with values below a certain threshold as unknown improves evaluation consistency:

def accuracy(input, target, thresh=0.4, na_idx=0):
valm, argm = input.max(-1)
argm[valm < thresh] = na_idx
return (argm == target).float().mean()


Conclusion and Future Work

While our modifications to the loss function and metrics significantly improved classification of unknown gestures, there remains room for further enhancements, particularly in gathering more extensive training data and experimenting with different architectures. For a detailed implementation and to explore various other techniques, you can access our GitHub repository and follow the experiments in our notebook.

By integrating these advanced techniques and continuously refining our models, we move closer to creating robust, accurate classifiers capable of handling a wide range of real-world scenarios.

Any feedback, comments, or questions are greatly appreciated. Let's continue to push the boundaries of what's possible with AI and deep learning!