Skip to content

Open In Colab

How to use negative examples

In some scenarios it might be useful to explicitly show the model images that should be considered background, these are called "negative examples" and are images that do not contain any annotations.

In this tutorial we're going to be training two raccoons detectors and observe how they perform on images of dogs and cats. One of the models will be trained only with images of raccoons, while the other will also have access to images of dogs and cats (the negative examples).

As you might already have imagined, the model that was trained with raccoon images predicts all animals to be raccoons!

Installing IceVision and IceData

If on Colab run the following cell, else check the installation instructions

# IceVision - IceData - MMDetection - YOLO v5 Installation
!wget https://raw.githubusercontent.com/airctic/icevision/master/install_colab.sh
!chmod +x install_colab.sh && ./install_colab.sh

Imports

from icevision.all import *

Raccoon dataset

The dataset is stored on github, so a simple git clone will do.

!git clone https://github.com/datitran/raccoon_dataset

The raccoon dataset uses the VOC annotation format, icevision natively supports this format:

raccoon_data_dir = Path('raccoon_dataset')
raccoon_parser = parsers.VOCBBoxParser(annotations_dir=raccoon_data_dir /'annotations', images_dir=raccoon_data_dir /'images')

Let's go ahead and parse our data with the default 80% train, 20% valid, split.

raccoon_train_records, raccoon_valid_records = raccoon_parser.parse()
show_records(random.choices(raccoon_train_records, k=3), ncols=3, class_map=class_map)

png

Pets dataset

With icedata we can easily download the pets dataset:

pets_data_dir = icedata.pets.load_data() / 'images'

Here we have a twist, instead of using the standard parser (icedata.pets.parser) which would parse all annotations, we will instead create a custom parser that only parsers the images.

Remember the steps for generating a custom parser (check this tutorial for more information).

pets_template_record = ObjectDetectionRecord()
Parser.generate_template(pets_template_record)
class MyParser(Parser):
    def __init__(self, template_record):
        super().__init__(template_record=template_record)
    def __iter__(self) -> Any:
    def __len__(self) -> int:
    def record_id(self, o) -> Hashable:
    def parse_fields(self, o, record, is_new):
        record.set_img_size(<ImgSize>)
        record.set_filepath(<Union[str, Path]>)
        record.detection.set_class_map(<ClassMap>)
        record.detection.add_labels(<Sequence[Hashable]>)
        record.detection.add_bboxes(<Sequence[BBox]>)

And now we use that to fill the required methods. We don't have to use the .detection methods since we don't want bboxes for these images.

class PetsImageParser(Parser):
    def __init__(self, template_record, data_dir):
        super().__init__(template_record=template_record)
        self.image_filepaths = get_image_files(data_dir)

    def __iter__(self) -> Any:
        yield from self.image_filepaths

    def __len__(self) -> int:
        return len(self.image_filepaths)

    def record_id(self, o) -> Hashable:
        return o.stem

    def parse_fields(self, o, record, is_new):
        if is_new:
            record.set_img_size(get_img_size(o))
            record.set_filepath(o)

Now we're ready to instantiate the parser and parse the data:

pets_parser = PetsImageParser(pets_template_record, pets_data_dir)
pets_train_records, pets_valid_records = pets_parser.parse()
show_records(random.choices(pets_train_records, k=3), ncols=3)

png

Transforms

Let's define a simple list of transforms, they are the same for both datasets.

presize = 512
size = 384

train_tfms = tfms.A.Adapter([*tfms.A.aug_tfms(size=size, presize=presize), tfms.A.Normalize()])
valid_tfms = tfms.A.Adapter([*tfms.A.resize_and_pad(size=size), tfms.A.Normalize()])

Datasets and DataLoaders

We create the raccoon dataset and dataloader as normal:

raccoon_train_ds = Dataset(raccoon_train_records, train_tfms)
raccoon_valid_ds = Dataset(raccoon_valid_records, valid_tfms)

raccoon_train_dl = efficientdet.train_dl(raccoon_train_ds, batch_size=16, num_workers=4, shuffle=True)
raccoon_valid_dl = efficientdet.valid_dl(raccoon_valid_ds, batch_size=16, num_workers=4, shuffle=False)

For adding the pets data, we simply have to combine the list of records. Note that the pets dataset contains a lot more images than the raccoon dataset, so we'll get only 100 images for train and 30 for valid, feel free to change these numbers and explore the results!

combined_train_ds = Dataset(raccoon_train_records + pets_train_records[:100], train_tfms)
combined_valid_ds = Dataset(raccoon_valid_records + pets_valid_records[:30], valid_tfms)

combined_train_dl = efficientdet.train_dl(combined_train_ds, batch_size=16, num_workers=4, shuffle=True)
combined_valid_dl = efficientdet.valid_dl(combined_valid_ds, batch_size=16, num_workers=4, shuffle=False)

Let's take a look at the combined dataset:

show_samples(random.choices(combined_train_ds, k=6), class_map=class_map, ncols=3)

png

Metrics

As usual, let's stick with our COCOMetric:

metrics = [COCOMetric(metric_type=COCOMetricType.bbox)]

Models

We're now ready to train a separate model for each dataset and see how the results change!

Raccoons only

model_type = models.ross.efficientdet
backbone = model_type.backbones.tf_lite0()
raccoon_model = efficientdet.model(backbone, num_classes=len(raccoon_parser.class_map), img_size=size)
raccoon_learn = efficientdet.fastai.learner(dls=[raccoon_train_dl, raccoon_valid_dl], model=raccoon_model, metrics=metrics)
raccoon_learn.fine_tune(30, 1e-2, freeze_epochs=5)
epoch train_loss valid_loss COCOMetric time
0 1.747288 1.354049 0.000296 00:08
1 1.536247 1.372290 0.002133 00:06
2 1.346514 1.235068 0.065862 00:06
3 1.147257 1.136243 0.159568 00:06
4 1.003974 1.082577 0.131702 00:06
epoch train_loss valid_loss COCOMetric time
0 0.556644 0.960585 0.284111 00:08
1 0.536918 0.872882 0.365284 00:08
2 0.508285 0.826196 0.341167 00:08
3 0.491788 0.718738 0.389929 00:08
4 0.477697 0.623467 0.394723 00:08
5 0.464236 0.636479 0.403623 00:07
6 0.453317 0.608919 0.409568 00:08
7 0.445977 0.692406 0.330556 00:07
8 0.431024 0.654473 0.411304 00:07
9 0.417715 0.603621 0.394563 00:07
10 0.404465 0.664019 0.410722 00:08
11 0.393347 0.791837 0.349207 00:08
12 0.380127 0.571296 0.441543 00:07
13 0.365624 0.581066 0.493645 00:07
14 0.356092 0.560838 0.485373 00:07
15 0.347702 0.596353 0.479297 00:07
16 0.336457 0.518080 0.526282 00:07
17 0.326160 0.517299 0.506950 00:07
18 0.312393 0.527800 0.478262 00:08
19 0.303629 0.503981 0.503308 00:07
20 0.297672 0.505081 0.494659 00:07
21 0.291259 0.499499 0.513842 00:08
22 0.281656 0.492497 0.513260 00:07
23 0.275641 0.488630 0.534297 00:07
24 0.273479 0.515775 0.521623 00:07
25 0.270986 0.533966 0.491943 00:07
26 0.264893 0.516566 0.528829 00:07
27 0.262945 0.497911 0.532670 00:07
28 0.260062 0.489442 0.524715 00:08
29 0.254210 0.488340 0.534543 00:08

If only raccoon photos are showed during training, everything is a raccoon!

efficientdet.show_results(raccoon_model, combined_valid_ds)

png

Raccoons + pets

combined_model = efficientdet.model(backbone, num_classes=len(raccoon_parser.class_map), img_size=size)
combined_learn = efficientdet.fastai.learner(dls=[combined_train_dl, combined_valid_dl], model=combined_model, metrics=metrics)
combined_learn.fine_tune(30, 1e-2, freeze_epochs=5)
epoch train_loss valid_loss COCOMetric time
0 2.629423 16.484606 0.000362 00:10
1 2.326190 10.975664 0.006295 00:10
2 1.823228 16.245941 0.109865 00:10
3 1.527442 49.796238 0.086900 00:10
4 1.251809 39.998779 0.069102 00:10
epoch train_loss valid_loss COCOMetric time
0 0.872557 20.347977 0.154616 00:12
1 0.763502 10.243771 0.224684 00:11
2 0.711195 10.047447 0.234970 00:11
3 0.661595 6.806975 0.285452 00:11
4 0.609079 4.103896 0.270586 00:11
5 0.577907 2.187822 0.322366 00:12
6 0.557430 2.126654 0.268614 00:11
7 0.535719 1.525599 0.330552 00:11
8 0.537557 7.515461 0.302180 00:11
9 0.516761 6.857810 0.349205 00:11
10 0.491282 4.321424 0.428389 00:12
11 0.474612 4.784684 0.376503 00:11
12 0.459131 4.136829 0.285586 00:11
13 0.445324 2.872945 0.374402 00:11
14 0.422465 1.711623 0.399955 00:11
15 0.402938 2.038269 0.424111 00:12
16 0.382965 1.591949 0.442361 00:11
17 0.368094 0.904978 0.485397 00:11
18 0.357877 0.632512 0.498205 00:11
19 0.336768 0.472029 0.469013 00:11
20 0.338500 0.427688 0.511560 00:11
21 0.333879 0.548746 0.465288 00:11
22 0.323725 0.739504 0.543710 00:11
23 0.312402 0.875822 0.529362 00:11
24 0.383150 0.737657 0.549364 00:11
25 0.364561 1.356293 0.508299 00:11
26 0.365999 1.194988 0.515767 00:11
27 0.353583 0.990942 0.525482 00:12
28 0.344147 0.887491 0.531592 00:11
29 0.332661 0.863412 0.521443 00:11

When negative samples are used during training, the model get's way better understading what is not a raccoon.

efficientdet.show_results(combined_model, combined_valid_ds)

png

Happy Learning!

That's it folks! If you need any assistance, feel free to join our forum.