Building a Sign Language Classifier using CNN & Keras.

Building a Sign Language Classifier using CNN & Keras

Computer vision plays a critical role in developing assistive technologies. In this post, we build a Convolutional Neural Network (CNN) model trained on the Sign Language MNIST dataset to recognize American Sign Language (ASL) gestures from pixel data.

---

1. Environment Setup & Data Loading

We begin by downloading the datamunge/sign-language-mnist dataset using KaggleHub and loading the training set via Pandas.

import os
import pandas as pd
import kagglehub

# Download dataset
path = kagglehub.dataset_download("datamunge/sign-language-mnist")

# Load training data
train_df = pd.read_csv(os.path.join(path, 'sign_mnist_train.csv'))
print(f"Dataset shape: {train_df.shape}")

The dataset contains 27,455 samples with 785 columns (1 label column + 784 pixel columns representing a 28x28 grayscale image).

2. Data Preprocessing & Visualization

Before passing our images into a deep learning model, we perform three essential preprocessing steps:

• Separate target labels from feature vectors.
• Normalize pixel intensity values from [0, 255] to [0.0, 1.0].
• Reshape pixel arrays into standard image dimensions (28, 28, 1).

from sklearn.model_selection import train_test_split
from tensorflow.keras.utils import to_categorical

# Feature / Label split & Normalization
X = train_df.drop('label', axis=1) / 255.0
y = train_df['label']

# Train / Test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Reshape for CNN input
X_train_reshaped = X_train.values.reshape(-1, 28, 28, 1)
X_test_reshaped = X_test.values.reshape(-1, 28, 28, 1)

# One-hot encode targets
num_classes = y_train.max() + 1
y_train_encoded = to_categorical(y_train, num_classes=num_classes)
y_test_encoded = to_categorical(y_test, num_classes=num_classes)

Sample Preview

Each gesture maps to an alphabet character (excluding motion-based letters J and Z):

Figure 1: Visualization of a single 28x28 grayscale ASL letter sample.

3. CNN Architecture Definition

We implement a multi-layer Convolutional Neural Network with Max-Pooling and Dropout layers to prevent overfitting:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)),
    MaxPooling2D((2, 2)),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D((2, 2)),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.5),
    Dense(num_classes, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

4. Training and Evaluation

The network is trained over 10 epochs using a batch size of 32:

history = model.fit(
    X_train_reshaped, y_train_encoded,
    epochs=10,
    batch_size=32,
    validation_data=(X_test_reshaped, y_test_encoded)
)

Final Model Evaluation Metrics

Test Loss: 0.0014

Test Accuracy: 99.96%

5. Model Inference Verification

Testing the model on unseen test samples demonstrates strong prediction capability with high probability confidence scores.

import numpy as np

# Select test sample
sample_image = X_test_reshaped[0]
processed_image = np.expand_dims(sample_image, axis=0)

# Prediction
predictions = model.predict(processed_image)
predicted_label = np.argmax(predictions[0])

print(f"Predicted Class Index: {predicted_label}")

Figure 2: Successful model inference comparing ground truth vs. model prediction.

---

Conclusion: Convolutional architectures provide exceptional performance for static sign language character classification tasks. Next steps include expanding to video stream processing for real-time gesture recognition.