Early Detection of Active Shooters: Leveraging Open Source Computer Vision and AWS for Weapon Detection

Written By Ben Rodrigue

An Open-Source Security Solution: Free and Accessible for Everyone

Imagine if you could lock down a school while a shooter is still in the parking lot? Or, as we saw in recent news, imagine if you can identify a shooter before they can try to assassinate a former president?

Well, we just saw a huge shift in computer vision that is making it possible for the average school, mall, concert venue, and literally anyone else to put a system in place that does this. META’s open-source SAM2 Model is a pre-trained, highly capable computer vision model that identifies objects in video and images via text prompting.

What we are presenting to the world, is a design pattern to use SAM2 with AWS services like Amazon SageMaker, and cheap hardware devices, like a raspberry pi, to monitor existing video systems and flag objects like rifles, handguns, and knives. The goal is to identify active shooters the moment a weapon is visible to shorten response times and, hopefully, save lives.

Shooter with rifle identified by model

Segment All Objects in an Image or Video

Generate multiple valid masks for ambiguous prompts.

What Does the Platform Do and How Does it Work?

Currently, there are services that attempt to identify weapons the second they are visible, but SAM2 brings a unique advantage: it’s an open source, high accuracy model requiring minimal development effort at minimal costs.

In the event of weapon identification, as we are discussing here, notifications can be sent to the police, school administration, or even directly to citizens. This allows a human to review the image, verify the threat, and respond accordingly. The model's ability to process and segment objects in real-time could dramatically improve response times and potentially save lives.

Man with handgun low resolution image

Identify objects at a distance and notify authorities

SAM can take input prompts to identify objects using CCTV systems, cell phones, body cams, and, in the future, VR headsets.

The Power of Open Source

One of the most inspiring aspects of SAM2 is its open-source nature. By making this technology freely available, META is encouraging innovation and collaboration. Developers, researchers, and tech enthusiasts can build on this foundation, creating customized solutions tailored to specific needs and contexts. The goal is to inspire others to take this technology, adapt it, and implement it in ways that can enhance public safety and security.

Identify Potential Threats

SAM has learned a general notion of what objects are -- this understanding enables zero-shot generalization to unfamiliar objects and images without requiring additional training.

A Call to Action

While we do not claim that this is a complete solution, the intent of sharing this design is to inspire others. We believe that by putting powerful tools like SAM2 in the hands of the community, we can collectively build platforms that have the potential to save lives.

This project is a call to action for developers and innovators. It's about coming together to leverage cutting-edge technology to address critical issues like school shootings and public safety threats. By sharing our designs and ideas, we hope to spark creativity and encourage the development of robust, reliable solutions that can make a real difference in the world.

Implementing an Opensource Weapon Detection Platform

To leverage META's SAM2 Model for identifying weapons such as handguns, rifles, and knives from CCTV footage, we can deploy the model using AWS services. Here’s a detailed design and implementation guide, including the necessary infrastructure setup using Terraform.

AWS Infrastructure Design Explanation

  1. S3 Bucket:

    • Used to store video frames for further processing.

  2. Kinesis Video Streams:

    • Captures and processes video streams in real-time, ensuring that video data is available for analysis.

  3. IAM Roles and Policies:

    • Ensure appropriate permissions for SageMaker, Lambda, and other AWS services.

  4. SageMaker Model and Endpoint:

    • Deploy the SAM2 model on SageMaker, allowing for real-time inference.

    • The model processes the video frames and identifies weapons.

  5. SNS Topic:

    • Used for sending notifications and alerts when a weapon is detected.

  6. Lambda Function:

    • Handles video frame processing, invokes the SAM2 model, and sends alerts if a weapon is detected.

AWS Infrastructure with Terraform

main.tf

provider "aws" {
  region = "us-west-2"
}

# Create S3 bucket for storing video frames
resource "aws_s3_bucket" "cctv_frames" {
  bucket = "school-cctv-frames"
  acl    = "private"
}

# Create Kinesis Video Stream
resource "aws_kinesis_video_stream" "cctv_stream" {
  name                = "SchoolCCTVStream"
  data_retention_in_hours = 24
  media_type          = "video/h264"
}

# Create IAM role for SageMaker
resource "aws_iam_role" "sagemaker_execution_role" {
  name = "sagemaker_execution_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "sagemaker.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "sagemaker_policy_attachment" {
  role       = aws_iam_role.sagemaker_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
}

# Create SageMaker endpoint for SAM model
resource "aws_sagemaker_model" "sam_model" {
  name                 = "sam-model"
  execution_role_arn   = aws_iam_role.sagemaker_execution_role.arn
  primary_container {
    image = "763104351884.dkr.ecr.us-west-2.amazonaws.com/sagemaker-tensorflow:2.3.0-cpu-py37"
    mode  = "SingleModel"
  }
}

resource "aws_sagemaker_endpoint_configuration" "sam_endpoint_config" {
  name = "sam-endpoint-config"

  production_variants {
    variant_name           = "AllTraffic"
    model_name             = aws_sagemaker_model.sam_model.name
    initial_instance_count = 1
    instance_type          = "ml.m5.large"
  }
}

resource "aws_sagemaker_endpoint" "sam_endpoint" {
  endpoint_name = "sam-endpoint"
  endpoint_config_name = aws_sagemaker_endpoint_configuration.sam_endpoint_config.name
}

# Create SNS topic for alerts
resource "aws_sns_topic" "weapon_alerts" {
  name = "WeaponAlerts"
}

# Create Lambda function for processing video frames and invoking SAM model
data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "lambda_execution_role" {
  name               = "lambda_execution_role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}

resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role       = aws_iam_role.lambda_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_lambda_function" "process_video_frame" {
  function_name = "ProcessVideoFrame"
  role          = aws_iam_role.lambda_execution_role.arn
  handler       = "lambda_function.lambda_handler"
  runtime       = "python3.8"

  environment {
    variables = {
      SAGEMAKER_ENDPOINT = aws_sagemaker_endpoint.sam_endpoint.endpoint_name
      SNS_TOPIC_ARN      = aws_sns_topic.weapon_alerts.arn
    }
  }

  s3_bucket        = aws_s3_bucket.cctv_frames.id
  s3_key           = "lambda_code/process_video_frame.zip"
  source_code_hash = filebase64sha256("lambda_code/process_video_frame.zip")
}

resource "aws_lambda_permission" "allow_s3_invoke" {
  statement_id  = "AllowExecutionFromS3"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.process_video_frame.function_name
  principal     = "s3.amazonaws.com"
}

Lambda Function for Weapon Detection

Create a Lambda function to process video frames, invoke the SAM model, and send alerts if a weapon is detected.

lambda_function.py

import json
import boto3
import cv2
import numpy as np
import os

# Initialize AWS clients
sagemaker_runtime = boto3.client('runtime.sagemaker')
sns = boto3.client('sns')
s3 = boto3.client('s3')

# Environment variables
SAGEMAKER_ENDPOINT = os.environ['SAGEMAKER_ENDPOINT']
SNS_TOPIC_ARN = os.environ['SNS_TOPIC_ARN']

def lambda_handler(event, context):
    # Process each record
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        # Get the image from S3
        response = s3.get_object(Bucket=bucket, Key=key)
        image_bytes = response['Body'].read()

        # Preprocess the image
        image_np = np.frombuffer(image_bytes, np.uint8)
        image = cv2.imdecode(image_np, cv2.IMREAD_COLOR)

        # Convert image to appropriate format for SAM
        _, buffer = cv2.imencode('.jpg', image)
        payload = buffer.tobytes()

        # Invoke the SageMaker endpoint
        response = sagemaker_runtime.invoke_endpoint(
            EndpointName=SAGEMAKER_ENDPOINT,
            ContentType='application/x-image',
            Body=payload
        )

        # Parse the response
        result = json.loads(response['Body'].read().decode())
        detected_objects = result.get('detected_objects', [])

        # Check for weapons in the result
        weapons = ['handgun', 'rifle', 'IED', 'knife']
        if any(obj in weapons for obj in detected_objects):
            send_alert(key, detected_objects)

    return {
        'statusCode': 200,
        'body': json.dumps('Processed successfully')
    }

def send_alert(image_key, detected_objects):
    message = f"Weapon detected in CCTV footage: {image_key}\nDetected objects: {', '.join(detected_objects)}"
    sns.publish(
        TopicArn=SNS_TOPIC_ARN,
        Message=message
    )

Edge Device Design Explanation

Hardware Requirements and Edge Device Setup

Edge Device:

  • Raspberry Pi 4: A popular choice for edge processing with AWS IoT Greengrass. It offers sufficient computational power for initial preprocessing tasks.

  • NVIDIA Jetson Nano: Ideal for more intensive machine learning tasks, providing better performance for real-time video processing.

  • Other Options: Intel NUC, BeagleBone Black, or any other device capable of running Linux and meeting the hardware requirements for Greengrass.

CCTV Cameras and IoT Core:

  • Ensure CCTV cameras are high-resolution and capable of real-time streaming.

  • Use AWS IoT Core to connect and manage the cameras, ensuring secure data transfer to AWS.

AWS IoT Greengrass:

  • Deploy AWS IoT Greengrass on edge devices like NVIDIA Jetson.

  • Greengrass allows local execution of AWS Lambda functions and Docker containers, enabling edge processing to reduce latency and bandwidth usage.

AWS IoT Greengrass Installation

  1. Set Up the Edge Device:

    • Install an appropriate OS (e.g., Raspbian for Raspberry Pi or Ubuntu for NVIDIA Jetson).

  2. Install AWS CLI:

    • Ensure AWS CLI is installed and configured on the edge device to manage AWS services and resources.

  3. Create and Configure AWS IoT Core Resources:

    • Create IoT Core resources (things, certificates, policies) using the AWS Management Console or AWS CLI.

    • Attach the policy to the certificate and associate the certificate with the IoT thing.

  4. Download and Install Greengrass Core Software:

  5. Set Up Greengrass Core:

    • Follow the detailed instructions provided in the AWS IoT Greengrass Developer Guide to install and set up Greengrass Core software.

    • Use the following commands as an example to set up Greengrass on a Raspberry Pi:

# Download the Greengrass Core software
wget https://d1onfpft10uf5o.cloudfront.net/greengrass-core/downloads/1.11.0/greengrass-linux-armv7l-1.11.0.tar.gz

# Extract the tarball
tar -xzvf greengrass-linux-armv7l-1.11.0.tar.gz -C /greengrass

# Navigate to the Greengrass directory
cd /greengrass/ggc/core/

# Start the Greengrass daemon
sudo ./greengrassd start

Configure the Greengrass Group:

  • In the AWS Management Console, create a Greengrass group and add the necessary components such as Lambda functions, connectors, and resources.

  • Deploy the Greengrass group configuration to the core device.

Connecting and Integrating with AWS Services

Greengrass Lambda Functions:

  • Develop and deploy Lambda functions that can process video streams and perform initial preprocessing before sending the data to AWS cloud services.

  • Use AWS IoT Greengrass to run these Lambda functions locally on your edge device.

Video Stream Processing:

  • Set up the edge device to capture video streams from the CCTV cameras.

  • Use OpenCV or similar libraries within the Lambda functions for frame extraction and preprocessing.

Secure Communication:

  • Ensure secure communication between the edge device and AWS IoT Core using the certificates and policies configured earlier.

Example Setup Code for Edge Processing

Here’s an example of a simple Greengrass Lambda function for preprocessing video frames on a Raspberry Pi:

import greengrasssdk
import platform
from threading import Timer
import time
import cv2

# Create a Greengrass Core SDK client
client = greengrasssdk.client('iot-data')

def greengrass_infinite_infer_run():
    cap = cv2.VideoCapture(0)  # Adjust the index based on your camera setup
    while True:
        ret, frame = cap.read()
        if ret:
            # Perform any preprocessing needed (e.g., resizing, normalization)
            resized_frame = cv2.resize(frame, (224, 224))
            # Convert frame to bytes and publish to an IoT topic
            _, buffer = cv2.imencode('.jpg', resized_frame)
            payload = buffer.tobytes()
            client.publish(topic='your/topic/here', payload=payload)
        time.sleep(1)

# Execute the function above
greengrass_infinite_infer_run()

# Start the infinite loop to keep the lambda running
while True:
    time.sleep(1)

Conclusion

By leveraging META’s SAM2 Model and AWS services, you can create an application that monitors CCTV footage for weapons, potentially enhancing public safety. This technology, while not a complete solution, is designed to inspire others to build and share their innovations. The goal is to democratize advanced vision models, making powerful tools accessible to everyone and encouraging the development of life-saving applications.

Again, we do not guarantee the accuracy or outcome of this implementation, and customization will be required, but share it to inspire further development and collaboration. Together, we can create solutions that protect our communities and make a real difference in public safety.

Note: All images are screenshots from: https://segment-anything.com/

Ben Rodrigue https://www.softstackers.com
Previous

Generative AI: Strategic Guide for Businesses
Next

How Our Clients are Operationalizing Generative AI