PyTorch torchvision - EfficientNet

[1. torchvision module code](#1. torchvision module code)
[2. torchvision.models.efficientnet](#2. torchvision.models.efficientnet)
[3. EfficientNet](#3. EfficientNet)
- [3.1. efficientnet_b0](#3.1. efficientnet_b0)
- [3.2. efficientnet_b1](#3.2. efficientnet_b1)
- [3.3. efficientnet_b2](#3.3. efficientnet_b2)
- [3.4. efficientnet_b3](#3.4. efficientnet_b3)
- [3.5. efficientnet_b4](#3.5. efficientnet_b4)
- [3.6. efficientnet_b5](#3.6. efficientnet_b5)
- [3.7. efficientnet_b6](#3.7. efficientnet_b6)
- [3.8. efficientnet_b7](#3.8. efficientnet_b7)
[4. PyTorch SiLU](#4. PyTorch SiLU)
- [4.1. EfficientNet-PyTorch](#4.1. EfficientNet-PyTorch)
[5. TensorFlow SiLU](#5. TensorFlow SiLU)
- [5.1. efficientnet](#5.1. efficientnet)
References

PyTorch torchvision - Docs
https://pytorch.org/vision/main/index.html

1. torchvision module code

https://pytorch.org/vision/main/_modules/index.html

torchvision.datasets._optical_flow
torchvision.datasets._stereo_matching
torchvision.datasets.caltech
torchvision.datasets.celeba
torchvision.datasets.cifar
torchvision.datasets.cityscapes
torchvision.datasets.coco
torchvision.datasets.country211
torchvision.datasets.dtd
torchvision.datasets.eurosat
torchvision.datasets.fakedata
torchvision.datasets.fer2013
torchvision.datasets.fgvc_aircraft
torchvision.datasets.flickr
torchvision.datasets.flowers102
torchvision.datasets.folder
torchvision.datasets.food101
torchvision.datasets.gtsrb
torchvision.datasets.hmdb51
torchvision.datasets.imagenet
torchvision.datasets.imagenette
torchvision.datasets.inaturalist
torchvision.datasets.kinetics
torchvision.datasets.kitti
torchvision.datasets.lfw
torchvision.datasets.lsun
torchvision.datasets.mnist
torchvision.datasets.moving_mnist
torchvision.datasets.omniglot
torchvision.datasets.oxford_iiit_pet
torchvision.datasets.pcam
torchvision.datasets.phototour
torchvision.datasets.places365
torchvision.datasets.rendered_sst2
torchvision.datasets.sbd
torchvision.datasets.sbu
torchvision.datasets.semeion
torchvision.datasets.stanford_cars
torchvision.datasets.stl10
torchvision.datasets.sun397
torchvision.datasets.svhn
torchvision.datasets.ucf101
torchvision.datasets.usps
torchvision.datasets.vision
torchvision.datasets.voc
torchvision.datasets.widerface
torchvision.io.image
torchvision.io.video
torchvision.io.video_reader
torchvision.models._api
torchvision.models.alexnet
torchvision.models.convnext
torchvision.models.densenet
torchvision.models.detection.faster_rcnn
torchvision.models.detection.fcos
torchvision.models.detection.keypoint_rcnn
torchvision.models.detection.mask_rcnn
torchvision.models.detection.retinanet
torchvision.models.detection.ssd
torchvision.models.detection.ssdlite
torchvision.models.efficientnet
torchvision.models.feature_extraction
torchvision.models.googlenet
torchvision.models.inception
torchvision.models.maxvit
torchvision.models.mnasnet
torchvision.models.mobilenetv2
torchvision.models.mobilenetv3
torchvision.models.optical_flow.raft
torchvision.models.quantization.googlenet
torchvision.models.quantization.inception
torchvision.models.quantization.mobilenetv2
torchvision.models.quantization.mobilenetv3
torchvision.models.quantization.resnet
torchvision.models.quantization.shufflenetv2
torchvision.models.regnet
torchvision.models.resnet
torchvision.models.segmentation.deeplabv3
torchvision.models.segmentation.fcn
torchvision.models.segmentation.lraspp
torchvision.models.shufflenetv2
torchvision.models.squeezenet
torchvision.models.swin_transformer
torchvision.models.vgg
torchvision.models.video.mvit
torchvision.models.video.resnet
torchvision.models.video.s3d
torchvision.models.video.swin_transformer
torchvision.models.vision_transformer
torchvision.ops.boxes
torchvision.ops.ciou_loss
torchvision.ops.deform_conv
torchvision.ops.diou_loss
torchvision.ops.drop_block
torchvision.ops.feature_pyramid_network
torchvision.ops.focal_loss
torchvision.ops.giou_loss
torchvision.ops.misc
torchvision.ops.poolers
torchvision.ops.ps_roi_align
torchvision.ops.ps_roi_pool
torchvision.ops.roi_align
torchvision.ops.roi_pool
torchvision.ops.stochastic_depth
torchvision.transforms.autoaugment
torchvision.transforms.functional
torchvision.transforms.transforms
torchvision.transforms.v2._augment
torchvision.transforms.v2._auto_augment
torchvision.transforms.v2._color
torchvision.transforms.v2._container
torchvision.transforms.v2._deprecated
torchvision.transforms.v2._geometry
torchvision.transforms.v2._meta
torchvision.transforms.v2._misc
torchvision.transforms.v2._temporal
torchvision.transforms.v2._type_conversion
torchvision.transforms.v2.functional._augment
torchvision.transforms.v2.functional._color
torchvision.transforms.v2.functional._deprecated
torchvision.transforms.v2.functional._geometry
torchvision.transforms.v2.functional._meta
torchvision.transforms.v2.functional._misc
torchvision.transforms.v2.functional._temporal
torchvision.transforms.v2.functional._type_conversion
torchvision.transforms.v2.functional._utils
torchvision.tv_tensors
torchvision.tv_tensors._bounding_boxes
torchvision.tv_tensors._dataset_wrapper
torchvision.tv_tensors._image
torchvision.tv_tensors._mask
torchvision.tv_tensors._torch_function_helpers
torchvision.tv_tensors._tv_tensor
torchvision.tv_tensors._video
torchvision.utils

2. torchvision.models.efficientnet

https://pytorch.org/vision/stable/_modules/torchvision/models/efficientnet.html

activation_layer = nn.SiLU

self.avgpool = nn.AdaptiveAvgPool2d(1)

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
https://arxiv.org/abs/1905.11946

torchvision/models/efficientnet.py
https://github.com/pytorch/vision/blob/main/torchvision/models/efficientnet.py

3. EfficientNet

https://pytorch.org/vision/stable/models/efficientnet.html

The EfficientNet model is based on the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

All the model builders internally rely on the torchvision.models.efficientnet.EfficientNet base class.

torch/nn/modules/activation.py
https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/activation.py

3.1. efficientnet_b0

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b0.html

EfficientNet B0 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.2. efficientnet_b1

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b1.html

EfficientNet B1 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.3. efficientnet_b2

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b2.html

EfficientNet B2 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.4. efficientnet_b3

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b3.html

EfficientNet B3 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.5. efficientnet_b4

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b4.html

EfficientNet B4 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.6. efficientnet_b5

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b5.html

EfficientNet B5 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.7. efficientnet_b6

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b6.html

EfficientNet B6 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

3.8. efficientnet_b7

https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_b7.html

EfficientNet B7 model architecture from the EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper.

4. PyTorch SiLU

https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html

Applies the Sigmoid Linear Unit (SiLU) function, element-wise.

The SiLU function is also known as the swish function.

silu ( x ) = x ∗ σ ( x ) , where σ ( x ) is the logistic sigmoid. \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.} silu(x)=x∗σ(x),where σ(x) is the logistic sigmoid.

Note

See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later.

Gaussian Error Linear Units (GELUs), https://arxiv.org/abs/1606.08415
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, https://arxiv.org/abs/1702.03118
Swish: a Self-Gated Activation Function, https://arxiv.org/abs/1710.05941v1

Shape:

Input: (*), where * means any number of dimensions.
Output: (*), same shape as the input.

Examples:

>>> m = nn.SiLU()
>>> input = torch.randn(2)
>>> output = m(input)

4.1. EfficientNet-PyTorch

EfficientNet-PyTorch
https://github.com/lukemelas/EfficientNet-PyTorch

efficientnet_pytorch/utils.py
https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/utils.py

# Swish activation function
if hasattr(nn, 'SiLU'):
    Swish = nn.SiLU
else:
    # For compatibility with old PyTorch versions
    class Swish(nn.Module):
        def forward(self, x):
            return x * torch.sigmoid(x)

5. TensorFlow SiLU

5.1. efficientnet

efficientnet
https://github.com/qubvel/efficientnet

efficientnet/model.py
https://github.com/qubvel/efficientnet/blob/master/efficientnet/model.py

def get_swish(**kwargs):
    backend, layers, models, keras_utils = get_submodules_from_kwargs(kwargs)

    def swish(x):
        """Swish activation function: x * sigmoid(x).
        Reference: [Searching for Activation Functions](https://arxiv.org/abs/1710.05941)
        """

        if backend.backend() == 'tensorflow':
            try:
                # The native TF implementation has a more
                # memory-efficient gradient implementation
                return backend.tf.nn.swish(x)
            except AttributeError:
                pass

        return x * backend.sigmoid(x)

    return swish

References

[1] Yongqiang Cheng, https://yongqiang.blog.csdn.net/

[2] module code, https://pytorch.org/vision/main/_modules/index.html