Skip to content

Open In Colab

Inference using IceVision

Install IceVision

The following downloads and runs a short shell script. The script installs IceVision, IceData, the MMDetection library, and Yolo v5 as well as the fastai and pytorch lightning engines.

!wget https://raw.githubusercontent.com/airctic/icevision/master/install_colab.sh
!bash install_colab.sh

Imports

All of the IceVision components can be easily imported with a single line.

from icevision.all import *
The history saving thread hit an unexpected error (DatabaseError('database disk image is malformed')).History will not be written to the database.

INFO     - The mmdet config folder already exists. No need to downloaded it. Path : /home/farid/.icevision/mmdetection_configs/mmdetection_configs-2.10.0/configs | icevision.models.mmdet.download_configs:download_mmdet_configs:17

List of images for inference

Please store your images in a folder, and populate the path_to_folder variable with the corresponding folder name.

# Pick your images folder
path_to_image_folder = "../samples/fridge/odFridgeObjects/images"
img_files = get_image_files(path_to_image_folder)
# img_files
img = PIL.Image.open(img_files[4])
img

png

Loading a checkpoint and creating the corresponding model

The checkpoint file can either a local file or an URL

# checkpoint_path = 'checkpoints/fridge-retinanet-save-checkpoint-full.pth'

checkpoint_path = 'http://download.openmmlab.com/mmdetection/v2.0/retinanet/retinanet_r50_fpn_1x_coco/retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth'
from icevision.models import *

# The model is automatically recreated in the evaluation mode. To unset that mode, you only need to pass `eval_mode=Fales`)
checkpoint_and_model = model_from_checkpoint(checkpoint_path, 
    model_name='mmdet.retinanet', 
    backbone_name='resnet50_fpn_1x',
    img_size=640, 
    is_coco=True)
# Just logging the info
model_type = checkpoint_and_model["model_type"]
backbone = checkpoint_and_model["backbone"]
class_map = checkpoint_and_model["class_map"]
img_size = checkpoint_and_model["img_size"]
model_type, backbone, class_map, img_size
Use load_from_http loader

/home/farid/anaconda3/envs/icevision/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py:99: UserWarning: init_cfg without layer key, if you do not define override key either, this init_cfg will do nothing
  warnings.warn(

(<module 'icevision.models.mmdet.models.retinanet' from '/home/farid/repos/airctic/icevision/icevision/models/mmdet/models/retinanet/__init__.py'>,
 <icevision.models.mmdet.models.retinanet.backbones.resnet_fpn.MMDetRetinanetBackboneConfig at 0x7fa2024c34f0>,
 <ClassMap: {'background': 0, 'person': 1, 'bicycle': 2, 'car': 3, 'motorcycle': 4, 'airplane': 5, 'bus': 6, 'train': 7, 'truck': 8, 'boat': 9, 'traffic light': 10, 'fire hydrant': 11, 'stop sign': 12, 'parking meter': 13, 'bench': 14, 'bird': 15, 'cat': 16, 'dog': 17, 'horse': 18, 'sheep': 19, 'cow': 20, 'elephant': 21, 'bear': 22, 'zebra': 23, 'giraffe': 24, 'backpack': 25, 'umbrella': 26, 'handbag': 27, 'tie': 28, 'suitcase': 29, 'frisbee': 30, 'skis': 31, 'snowboard': 32, 'sports ball': 33, 'kite': 34, 'baseball bat': 35, 'baseball glove': 36, 'skateboard': 37, 'surfboard': 38, 'tennis racket': 39, 'bottle': 40, 'wine glass': 41, 'cup': 42, 'fork': 43, 'knife': 44, 'spoon': 45, 'bowl': 46, 'banana': 47, 'apple': 48, 'sandwich': 49, 'orange': 50, 'broccoli': 51, 'carrot': 52, 'hot dog': 53, 'pizza': 54, 'donut': 55, 'cake': 56, 'chair': 57, 'couch': 58, 'potted plant': 59, 'bed': 60, 'dining table': 61, 'toilet': 62, 'tv': 63, 'laptop': 64, 'mouse': 65, 'remote': 66, 'keyboard': 67, 'cell phone': 68, 'microwave': 69, 'oven': 70, 'toaster': 71, 'sink': 72, 'refrigerator': 73, 'book': 74, 'clock': 75, 'vase': 76, 'scissors': 77, 'teddy bear': 78, 'hair drier': 79, 'toothbrush': 80}>,
 640)

Get Model Object

model_from_checkpoint(checkpoint_path) returns a dictionary: checkpoint_and_model. The model object is stored in checkpoint_and_model["model"].

# Get model object
# The model is automatically set in the evaluation mode
model = checkpoint_and_model["model"]

# Check device
device=next(model.parameters()).device
device
# Transforms
img_size = checkpoint_and_model["img_size"]
valid_tfms = tfms.A.Adapter([*tfms.A.resize_and_pad(img_size), tfms.A.Normalize()])
device(type='cpu')

Single Image Inference

The end2end_detect() not only compute predictions for a single image but also automatically adjust predicted boxes to the original image size

img = PIL.Image.open(img_files[4])

pred_dict  = model_type.end2end_detect(img, valid_tfms, model, class_map=class_map, detection_threshold=0.5)
pred_dict['img']

png

Batch Inference

The following option shows to do generate inference for a set of images. The latter is processed in batches.

# Create a dataset
imgs_array = [PIL.Image.open(Path(fname)) for fname in img_files]
infer_ds = Dataset.from_images(imgs_array, valid_tfms, class_map=class_map)

# Batch Inference
infer_dl = model_type.infer_dl(infer_ds, batch_size=4, shuffle=False)
preds = model_type.predict_from_dl(model, infer_dl, keep_images=True)
show_preds(preds=preds[0:5])
  0%|          | 0/2 [00:00<?, ?it/s]

png

How to save a model and its metadata in IceVision

When saving a model weights, we could also store the model metadata that are retrieved by the model_from_checkpoint(checkpoint_path) method

# How to save a model and its metadata
checkpoint_path = 'coco-retinanet-checkpoint-full.pth'

save_icevision_checkpoint(model, 
                        model_name='mmdet.retinanet', 
                        backbone_name='resnet50_fpn_1x',
                        classes =  class_map.get_classes(), 
                        img_size=img_size, 
                        filename=checkpoint_path,
                        meta={'icevision_version': '0.9.1'})

Loading models already containing metadata

If you have saved your model weights with the model metadata, you only need to call model_from_checkpoint(checkpoint_path): No other arguments (model_name, backbone_name, classes, img_size) are needed. All the information is already embedded in the checkpoint file.

checkpoint_path = 'https://github.com/airctic/model_zoo/releases/download/m6/fridge-retinanet-checkpoint-full.pth'

checkpoint_and_model = model_from_checkpoint(checkpoint_path)

# Just logging the info
model_type = checkpoint_and_model["model_type"]
backbone = checkpoint_and_model["backbone"]
class_map = checkpoint_and_model["class_map"]
img_size = checkpoint_and_model["img_size"]
model_type, backbone, class_map, img_size
# Inference

# Model
model = checkpoint_and_model["model"]

# Transforms
img_size = checkpoint_and_model["img_size"]
valid_tfms = tfms.A.Adapter([*tfms.A.resize_and_pad(img_size), tfms.A.Normalize()])

# Pick your images folder
path_to_image_folder = "../samples/fridge/odFridgeObjects/images"
img_files = get_image_files(path_to_image_folder)

# Create a dataset with appropriate images 
imgs_array = [PIL.Image.open(Path(fname)) for fname in img_files]
infer_ds = Dataset.from_images(imgs_array, valid_tfms, class_map=class_map)

# Batch Inference
infer_dl = model_type.infer_dl(infer_ds, batch_size=4, shuffle=False)
preds = model_type.predict_from_dl(model, infer_dl, keep_images=True)
show_preds(preds=preds[0:2])

Use load_from_http loader

(<module 'icevision.models.mmdet.models.retinanet' from '/home/farid/repos/airctic/icevision/icevision/models/mmdet/models/retinanet/__init__.py'>,
 <icevision.models.mmdet.models.retinanet.backbones.resnet_fpn.MMDetRetinanetBackboneConfig at 0x7fa2024c34f0>,
 <ClassMap: {'background': 0, 'carton': 1, 'milk_bottle': 2, 'can': 3, 'water_bottle': 4}>,
 384)

png

Happy Learning!

If you need any assistance, feel free to join our forum.