{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TF Watcher Demo.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Rishit-dagli/TF-Watcher/blob/main/docs/source/TF-Watcher-Quickstart.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABLsHz3eq5g5"
      },
      "source": [
        "# Quick Start: TF Watcher\n",
        "\n",
        "![](https://i.imgur.com/C1EuWC2.png)\n",
        "\n",
        "What if you could monitor your Machine Learning jobs on Colab, Kaggle, AzureML, or pretty much anywhere on your mobile phones, so you could monitor your models on the fly. Even while taking a walk🚶in < 5 lines of code!\n",
        "\n",
        "Consider giving a star⭐ to [the project](https://github.com/Rishit-dagli/TF-Watcher) if you like it.\n",
        "\n",
        "In this example we’ll use the [Fashion MNIST dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/fashion_mnist) to serve as a quickstart.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qo1MNu3MrphA"
      },
      "source": [
        "## Setup\n",
        "\n",
        "We start off by installing TF Watcher from PyPI."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "thIaiMelqOmg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ceb39235-a259-46a6-8d2c-6ef82299e5cd"
      },
      "source": [
        "!pip install tf-watcher"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting tf-watcher\n",
            "  Downloading tf_watcher-0.1.1-py3-none-any.whl (22 kB)\n",
            "Collecting pyrebase4~=4.5.0\n",
            "  Downloading Pyrebase4-4.5.0-py3-none-any.whl (8.9 kB)\n",
            "Collecting tensorflow~=2.5.0\n",
            "  Downloading tensorflow-2.5.1-cp37-cp37m-manylinux2010_x86_64.whl (454.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 454.4 MB 8.9 kB/s \n",
            "\u001b[?25hCollecting pycryptodome>=3.6.4\n",
            "  Downloading pycryptodome-3.10.1-cp35-abi3-manylinux2010_x86_64.whl (1.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.9 MB 66.0 MB/s \n",
            "\u001b[?25hCollecting requests-toolbelt>=0.7.1\n",
            "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
            "\u001b[K     |████████████████████████████████| 54 kB 2.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: oauth2client>=4.1.2 in /usr/local/lib/python3.7/dist-packages (from pyrebase4~=4.5.0->tf-watcher) (4.1.3)\n",
            "Requirement already satisfied: requests>=2.19.1 in /usr/local/lib/python3.7/dist-packages (from pyrebase4~=4.5.0->tf-watcher) (2.23.0)\n",
            "Collecting python-jwt>=2.0.1\n",
            "  Downloading python_jwt-3.3.0-py2.py3-none-any.whl (6.8 kB)\n",
            "Collecting gcloud>=0.18.3\n",
            "  Downloading gcloud-0.18.3.tar.gz (454 kB)\n",
            "\u001b[K     |████████████████████████████████| 454 kB 53.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python3.7/dist-packages (from gcloud>=0.18.3->pyrebase4~=4.5.0->tf-watcher) (0.17.4)\n",
            "Requirement already satisfied: googleapis-common-protos in /usr/local/lib/python3.7/dist-packages (from gcloud>=0.18.3->pyrebase4~=4.5.0->tf-watcher) (1.53.0)\n",
            "Requirement already satisfied: protobuf!=3.0.0.b2.post1,>=3.0.0b2 in /usr/local/lib/python3.7/dist-packages (from gcloud>=0.18.3->pyrebase4~=4.5.0->tf-watcher) (3.17.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from gcloud>=0.18.3->pyrebase4~=4.5.0->tf-watcher) (1.15.0)\n",
            "Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.1.2->pyrebase4~=4.5.0->tf-watcher) (4.7.2)\n",
            "Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.1.2->pyrebase4~=4.5.0->tf-watcher) (0.4.8)\n",
            "Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from oauth2client>=4.1.2->pyrebase4~=4.5.0->tf-watcher) (0.2.8)\n",
            "Collecting jwcrypto>=0.8\n",
            "  Downloading jwcrypto-1.0-py2.py3-none-any.whl (82 kB)\n",
            "\u001b[K     |████████████████████████████████| 82 kB 900 kB/s \n",
            "\u001b[?25hCollecting deprecated\n",
            "  Downloading Deprecated-1.2.12-py2.py3-none-any.whl (9.5 kB)\n",
            "Collecting cryptography>=2.3\n",
            "  Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.2 MB 41.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=2.3->jwcrypto>=0.8->python-jwt>=2.0.1->pyrebase4~=4.5.0->tf-watcher) (1.14.6)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=2.3->jwcrypto>=0.8->python-jwt>=2.0.1->pyrebase4~=4.5.0->tf-watcher) (2.20)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.1->pyrebase4~=4.5.0->tf-watcher) (2021.5.30)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.1->pyrebase4~=4.5.0->tf-watcher) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.1->pyrebase4~=4.5.0->tf-watcher) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.1->pyrebase4~=4.5.0->tf-watcher) (1.24.3)\n",
            "Collecting keras-nightly~=2.5.0.dev\n",
            "  Downloading keras_nightly-2.5.0.dev2021032900-py2.py3-none-any.whl (1.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.2 MB 68.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.19.5)\n",
            "Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (3.3.0)\n",
            "Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (0.2.0)\n",
            "Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (2.6.0)\n",
            "Collecting tensorflow-estimator<2.6.0,>=2.5.0\n",
            "  Downloading tensorflow_estimator-2.5.0-py2.py3-none-any.whl (462 kB)\n",
            "\u001b[K     |████████████████████████████████| 462 kB 74.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (0.12.0)\n",
            "Collecting grpcio~=1.34.0\n",
            "  Downloading grpcio-1.34.1-cp37-cp37m-manylinux2014_x86_64.whl (4.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.0 MB 41.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (3.7.4.3)\n",
            "Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.1.2)\n",
            "Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.1.0)\n",
            "Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.12)\n",
            "Requirement already satisfied: h5py~=3.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (3.1.0)\n",
            "Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (0.4.0)\n",
            "Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (1.12.1)\n",
            "Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.5.0->tf-watcher) (0.37.0)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py~=3.1.0->tensorflow~=2.5.0->tf-watcher) (1.5.2)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (0.6.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (57.4.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (0.4.5)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (1.8.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (3.3.4)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (1.34.0)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (4.2.2)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (4.6.4)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (3.1.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard~=2.5->tensorflow~=2.5.0->tf-watcher) (3.5.0)\n",
            "Building wheels for collected packages: gcloud\n",
            "  Building wheel for gcloud (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gcloud: filename=gcloud-0.18.3-py3-none-any.whl size=602936 sha256=207e73d727a7400f3c987feb991b513d6d82ec58ce461b1ae54dc4284ccc2c94\n",
            "  Stored in directory: /root/.cache/pip/wheels/50/9d/19/dc1a639905431252dbf8a61388a83459c5cfef7add445bebe4\n",
            "Successfully built gcloud\n",
            "Installing collected packages: deprecated, cryptography, jwcrypto, grpcio, tensorflow-estimator, requests-toolbelt, python-jwt, pycryptodome, keras-nightly, gcloud, tensorflow, pyrebase4, tf-watcher\n",
            "  Attempting uninstall: grpcio\n",
            "    Found existing installation: grpcio 1.39.0\n",
            "    Uninstalling grpcio-1.39.0:\n",
            "      Successfully uninstalled grpcio-1.39.0\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.6.0\n",
            "    Uninstalling tensorflow-estimator-2.6.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.6.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.6.0\n",
            "    Uninstalling tensorflow-2.6.0:\n",
            "      Successfully uninstalled tensorflow-2.6.0\n",
            "Successfully installed cryptography-3.4.7 deprecated-1.2.12 gcloud-0.18.3 grpcio-1.34.1 jwcrypto-1.0 keras-nightly-2.5.0.dev2021032900 pycryptodome-3.10.1 pyrebase4-4.5.0 python-jwt-3.3.0 requests-toolbelt-0.9.1 tensorflow-2.5.1 tensorflow-estimator-2.5.0 tf-watcher-0.1.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ZUHwRXar0d3"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import tfwatcher"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bs8H7Jbar88W"
      },
      "source": [
        "## Get the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FjJwLfWNqiNg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86db0eab-0d00-403d-a879-dfda1ea50c2b"
      },
      "source": [
        "# Load example MNIST data and pre-process it\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWpMfsOfsP4-"
      },
      "source": [
        "Some simple pre-processing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lyHgcA9_sPLW"
      },
      "source": [
        "x_train = x_train.reshape(-1, 784).astype(\"float32\") / 255.0\n",
        "x_test = x_test.reshape(-1, 784).astype(\"float32\") / 255.0\n",
        "\n",
        "# Limit the data to 1000 samples\n",
        "x_train = x_train[:1000]\n",
        "y_train = y_train[:1000]\n",
        "x_test = x_test[:1000]\n",
        "y_test = y_test[:1000]"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kgTdz3fBsT9G"
      },
      "source": [
        "## Create a simple model\n",
        "\n",
        "Note the `metrics` argument in `model.compile`, specify any additional metrics you want to log from here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BJV8gVVbqdTV"
      },
      "source": [
        "# Define the Keras model to add callbacks to\n",
        "def get_model():\n",
        "    model = keras.Sequential()\n",
        "    model.add(keras.layers.Dense(1, input_dim=784))\n",
        "    model.compile(\n",
        "        optimizer=keras.optimizers.RMSprop(learning_rate=0.1),\n",
        "        loss=\"mean_squared_error\",\n",
        "        metrics=[\"accuracy\"],\n",
        "    )\n",
        "    return model"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9K1GyQYrsj_q"
      },
      "source": [
        "## Use TF Watcher while training\n",
        "\n",
        "- We use the `EpochEnd` class from TF Watcher to specify we are interested in operating in the epoch level\n",
        "- We pass in `schedule` as 1 to monitor after every epoch. You could pass in this as 3 to monitor after every 3 epochs or also pass in a list of the specific epoch number you want to monitor on"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bCFgAzIvqrR-",
        "outputId": "e5be2f19-99d0-4ddb-b317-c09bf8fb3b9d"
      },
      "source": [
        "MonitorCallback = tfwatcher.callbacks.EpochEnd(schedule = 1)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Use this ID to monitor training for this session: ybhzyxK\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGfjyhEjb0A4"
      },
      "source": [
        "We will now use this ID on https://www.tfwatcher.tech to monitor our model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E7xAlx5rqqDu",
        "outputId": "6163f0a2-b1d2-4f5b-cf62-d9cd3e35ea16"
      },
      "source": [
        "model = get_model()\n",
        "history = model.fit(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    batch_size=128,\n",
        "    epochs=100,\n",
        "    validation_split=0.5,\n",
        "    callbacks = [MonitorCallback]\n",
        ")"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "4/4 [==============================] - 1s 58ms/step - loss: 220.0279 - accuracy: 0.1400 - val_loss: 7.1787 - val_accuracy: 0.1000\n",
            "Epoch 2/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 5.8449 - accuracy: 0.1400 - val_loss: 5.6531 - val_accuracy: 0.1240\n",
            "Epoch 3/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 4.3850 - accuracy: 0.1580 - val_loss: 4.7036 - val_accuracy: 0.1160\n",
            "Epoch 4/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 3.6359 - accuracy: 0.1700 - val_loss: 5.1514 - val_accuracy: 0.1040\n",
            "Epoch 5/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 22.7958 - accuracy: 0.1500 - val_loss: 94.3385 - val_accuracy: 0.1000\n",
            "Epoch 6/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 66.1149 - accuracy: 0.1160 - val_loss: 38.9990 - val_accuracy: 0.1000\n",
            "Epoch 7/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 23.7571 - accuracy: 0.1280 - val_loss: 16.4039 - val_accuracy: 0.1000\n",
            "Epoch 8/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 16.1609 - accuracy: 0.1080 - val_loss: 31.8129 - val_accuracy: 0.1000\n",
            "Epoch 9/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 36.0135 - accuracy: 0.1200 - val_loss: 47.4181 - val_accuracy: 0.1000\n",
            "Epoch 10/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 38.0579 - accuracy: 0.1220 - val_loss: 24.0960 - val_accuracy: 0.1000\n",
            "Epoch 11/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 23.2553 - accuracy: 0.1040 - val_loss: 24.5393 - val_accuracy: 0.1000\n",
            "Epoch 12/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 25.9761 - accuracy: 0.1180 - val_loss: 43.4903 - val_accuracy: 0.1000\n",
            "Epoch 13/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 36.0534 - accuracy: 0.1320 - val_loss: 27.3967 - val_accuracy: 0.1000\n",
            "Epoch 14/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 23.6804 - accuracy: 0.1200 - val_loss: 30.0495 - val_accuracy: 0.1000\n",
            "Epoch 15/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 34.2381 - accuracy: 0.1220 - val_loss: 41.0987 - val_accuracy: 0.1000\n",
            "Epoch 16/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 28.6988 - accuracy: 0.1220 - val_loss: 38.2743 - val_accuracy: 0.1000\n",
            "Epoch 17/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 29.1761 - accuracy: 0.1180 - val_loss: 38.2930 - val_accuracy: 0.1000\n",
            "Epoch 18/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 29.0422 - accuracy: 0.1320 - val_loss: 33.8774 - val_accuracy: 0.1000\n",
            "Epoch 19/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 25.2365 - accuracy: 0.1360 - val_loss: 36.8045 - val_accuracy: 0.1000\n",
            "Epoch 20/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 33.1871 - accuracy: 0.1080 - val_loss: 35.7805 - val_accuracy: 0.1000\n",
            "Epoch 21/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 30.1454 - accuracy: 0.1140 - val_loss: 29.5536 - val_accuracy: 0.1000\n",
            "Epoch 22/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 24.0112 - accuracy: 0.1120 - val_loss: 30.0866 - val_accuracy: 0.1000\n",
            "Epoch 23/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 34.7015 - accuracy: 0.1040 - val_loss: 29.4295 - val_accuracy: 0.1000\n",
            "Epoch 24/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 24.5739 - accuracy: 0.1340 - val_loss: 23.0243 - val_accuracy: 0.1000\n",
            "Epoch 25/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 20.7962 - accuracy: 0.1360 - val_loss: 50.8549 - val_accuracy: 0.1000\n",
            "Epoch 26/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 41.6597 - accuracy: 0.1160 - val_loss: 32.8466 - val_accuracy: 0.1000\n",
            "Epoch 27/100\n",
            "4/4 [==============================] - 0s 14ms/step - loss: 26.6706 - accuracy: 0.1260 - val_loss: 31.4648 - val_accuracy: 0.1000\n",
            "Epoch 28/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 28.9898 - accuracy: 0.1060 - val_loss: 34.4348 - val_accuracy: 0.1000\n",
            "Epoch 29/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 32.6081 - accuracy: 0.1180 - val_loss: 37.6328 - val_accuracy: 0.1000\n",
            "Epoch 30/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.6324 - accuracy: 0.1340 - val_loss: 26.2719 - val_accuracy: 0.1000\n",
            "Epoch 31/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 17.5867 - accuracy: 0.1200 - val_loss: 32.1177 - val_accuracy: 0.1000\n",
            "Epoch 32/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 34.4390 - accuracy: 0.1240 - val_loss: 46.8234 - val_accuracy: 0.1000\n",
            "Epoch 33/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 35.8227 - accuracy: 0.1020 - val_loss: 24.2219 - val_accuracy: 0.1000\n",
            "Epoch 34/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 19.7204 - accuracy: 0.1160 - val_loss: 32.0298 - val_accuracy: 0.1000\n",
            "Epoch 35/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 34.6682 - accuracy: 0.1300 - val_loss: 43.5553 - val_accuracy: 0.1000\n",
            "Epoch 36/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.7967 - accuracy: 0.1360 - val_loss: 35.5064 - val_accuracy: 0.1000\n",
            "Epoch 37/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 24.1768 - accuracy: 0.1060 - val_loss: 33.8834 - val_accuracy: 0.1000\n",
            "Epoch 38/100\n",
            "4/4 [==============================] - 0s 13ms/step - loss: 35.0535 - accuracy: 0.0980 - val_loss: 33.7792 - val_accuracy: 0.1000\n",
            "Epoch 39/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 27.3413 - accuracy: 0.1040 - val_loss: 26.7047 - val_accuracy: 0.1000\n",
            "Epoch 40/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 26.9778 - accuracy: 0.1180 - val_loss: 34.2578 - val_accuracy: 0.1000\n",
            "Epoch 41/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 28.9345 - accuracy: 0.0940 - val_loss: 38.9659 - val_accuracy: 0.1000\n",
            "Epoch 42/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 31.8423 - accuracy: 0.1180 - val_loss: 33.2312 - val_accuracy: 0.1000\n",
            "Epoch 43/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.6944 - accuracy: 0.1100 - val_loss: 23.8519 - val_accuracy: 0.1000\n",
            "Epoch 44/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 21.8561 - accuracy: 0.1160 - val_loss: 37.2674 - val_accuracy: 0.1000\n",
            "Epoch 45/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 28.3689 - accuracy: 0.1200 - val_loss: 50.9949 - val_accuracy: 0.1000\n",
            "Epoch 46/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 37.0924 - accuracy: 0.1120 - val_loss: 37.8075 - val_accuracy: 0.1000\n",
            "Epoch 47/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 22.4427 - accuracy: 0.1220 - val_loss: 31.0246 - val_accuracy: 0.1000\n",
            "Epoch 48/100\n",
            "4/4 [==============================] - 0s 16ms/step - loss: 30.8454 - accuracy: 0.1160 - val_loss: 40.3705 - val_accuracy: 0.1000\n",
            "Epoch 49/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 30.1953 - accuracy: 0.1080 - val_loss: 39.0889 - val_accuracy: 0.1000\n",
            "Epoch 50/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 19.4889 - accuracy: 0.1080 - val_loss: 34.6825 - val_accuracy: 0.1000\n",
            "Epoch 51/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 34.8757 - accuracy: 0.1300 - val_loss: 49.4359 - val_accuracy: 0.1000\n",
            "Epoch 52/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 34.9451 - accuracy: 0.1140 - val_loss: 31.5136 - val_accuracy: 0.1000\n",
            "Epoch 53/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 22.9106 - accuracy: 0.1380 - val_loss: 26.9605 - val_accuracy: 0.1000\n",
            "Epoch 54/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.2229 - accuracy: 0.1040 - val_loss: 38.6458 - val_accuracy: 0.1000\n",
            "Epoch 55/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.1122 - accuracy: 0.1140 - val_loss: 45.1972 - val_accuracy: 0.1000\n",
            "Epoch 56/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 36.5613 - accuracy: 0.1280 - val_loss: 33.8477 - val_accuracy: 0.1000\n",
            "Epoch 57/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.9645 - accuracy: 0.1020 - val_loss: 29.3765 - val_accuracy: 0.1000\n",
            "Epoch 58/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 23.7523 - accuracy: 0.1140 - val_loss: 38.5822 - val_accuracy: 0.1000\n",
            "Epoch 59/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 30.7082 - accuracy: 0.1440 - val_loss: 40.9264 - val_accuracy: 0.1000\n",
            "Epoch 60/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 31.7553 - accuracy: 0.1200 - val_loss: 28.5086 - val_accuracy: 0.1000\n",
            "Epoch 61/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.3221 - accuracy: 0.1240 - val_loss: 32.6517 - val_accuracy: 0.1000\n",
            "Epoch 62/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.3470 - accuracy: 0.1200 - val_loss: 39.8263 - val_accuracy: 0.1000\n",
            "Epoch 63/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.0612 - accuracy: 0.1240 - val_loss: 34.5444 - val_accuracy: 0.1000\n",
            "Epoch 64/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 34.7778 - accuracy: 0.1200 - val_loss: 44.2647 - val_accuracy: 0.1000\n",
            "Epoch 65/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 34.8926 - accuracy: 0.1080 - val_loss: 34.5197 - val_accuracy: 0.1000\n",
            "Epoch 66/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 24.3221 - accuracy: 0.1220 - val_loss: 30.9930 - val_accuracy: 0.1000\n",
            "Epoch 67/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.4201 - accuracy: 0.1160 - val_loss: 37.4984 - val_accuracy: 0.1000\n",
            "Epoch 68/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 28.9322 - accuracy: 0.1280 - val_loss: 36.6450 - val_accuracy: 0.1000\n",
            "Epoch 69/100\n",
            "4/4 [==============================] - 0s 14ms/step - loss: 28.5764 - accuracy: 0.1320 - val_loss: 34.3183 - val_accuracy: 0.1000\n",
            "Epoch 70/100\n",
            "4/4 [==============================] - 0s 16ms/step - loss: 23.6287 - accuracy: 0.1200 - val_loss: 40.6597 - val_accuracy: 0.1000\n",
            "Epoch 71/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 30.6631 - accuracy: 0.1160 - val_loss: 43.1053 - val_accuracy: 0.1000\n",
            "Epoch 72/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 31.3124 - accuracy: 0.1160 - val_loss: 31.0489 - val_accuracy: 0.1000\n",
            "Epoch 73/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 24.3881 - accuracy: 0.1100 - val_loss: 36.0015 - val_accuracy: 0.1000\n",
            "Epoch 74/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 32.8749 - accuracy: 0.1060 - val_loss: 31.0803 - val_accuracy: 0.1000\n",
            "Epoch 75/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 23.9160 - accuracy: 0.1160 - val_loss: 32.9105 - val_accuracy: 0.1000\n",
            "Epoch 76/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 31.2406 - accuracy: 0.1140 - val_loss: 34.7558 - val_accuracy: 0.1000\n",
            "Epoch 77/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 26.3004 - accuracy: 0.1220 - val_loss: 40.3014 - val_accuracy: 0.1000\n",
            "Epoch 78/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 37.0920 - accuracy: 0.1340 - val_loss: 27.9852 - val_accuracy: 0.1000\n",
            "Epoch 79/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 20.3414 - accuracy: 0.1140 - val_loss: 27.2020 - val_accuracy: 0.1000\n",
            "Epoch 80/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 29.3138 - accuracy: 0.1340 - val_loss: 47.7526 - val_accuracy: 0.1000\n",
            "Epoch 81/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 34.6481 - accuracy: 0.1320 - val_loss: 32.0029 - val_accuracy: 0.1000\n",
            "Epoch 82/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 19.7531 - accuracy: 0.1260 - val_loss: 23.1984 - val_accuracy: 0.1000\n",
            "Epoch 83/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.6197 - accuracy: 0.1200 - val_loss: 44.1223 - val_accuracy: 0.1000\n",
            "Epoch 84/100\n",
            "4/4 [==============================] - 0s 14ms/step - loss: 42.5353 - accuracy: 0.1200 - val_loss: 33.2591 - val_accuracy: 0.1000\n",
            "Epoch 85/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 19.8397 - accuracy: 0.1200 - val_loss: 18.8762 - val_accuracy: 0.1000\n",
            "Epoch 86/100\n",
            "4/4 [==============================] - 0s 15ms/step - loss: 19.0276 - accuracy: 0.1200 - val_loss: 39.6742 - val_accuracy: 0.1000\n",
            "Epoch 87/100\n",
            "4/4 [==============================] - 0s 11ms/step - loss: 45.4169 - accuracy: 0.1160 - val_loss: 33.8931 - val_accuracy: 0.1000\n",
            "Epoch 88/100\n",
            "4/4 [==============================] - 0s 12ms/step - loss: 26.4559 - accuracy: 0.1100 - val_loss: 16.6738 - val_accuracy: 0.1040\n",
            "Epoch 89/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 12.4969 - accuracy: 0.1260 - val_loss: 33.8079 - val_accuracy: 0.1000\n",
            "Epoch 90/100\n",
            "4/4 [==============================] - 0s 16ms/step - loss: 38.4502 - accuracy: 0.1000 - val_loss: 49.4086 - val_accuracy: 0.1000\n",
            "Epoch 91/100\n",
            "4/4 [==============================] - 0s 14ms/step - loss: 29.9915 - accuracy: 0.1280 - val_loss: 36.8529 - val_accuracy: 0.1000\n",
            "Epoch 92/100\n",
            "4/4 [==============================] - 0s 14ms/step - loss: 25.7386 - accuracy: 0.1040 - val_loss: 31.6561 - val_accuracy: 0.1000\n",
            "Epoch 93/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 27.4582 - accuracy: 0.1160 - val_loss: 50.1114 - val_accuracy: 0.1000\n",
            "Epoch 94/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 36.5150 - accuracy: 0.0920 - val_loss: 28.7904 - val_accuracy: 0.1000\n",
            "Epoch 95/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 19.6982 - accuracy: 0.1280 - val_loss: 35.2172 - val_accuracy: 0.1000\n",
            "Epoch 96/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 31.5606 - accuracy: 0.1000 - val_loss: 39.6383 - val_accuracy: 0.1000\n",
            "Epoch 97/100\n",
            "4/4 [==============================] - 0s 9ms/step - loss: 32.0419 - accuracy: 0.1100 - val_loss: 37.4634 - val_accuracy: 0.1000\n",
            "Epoch 98/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 25.6015 - accuracy: 0.1280 - val_loss: 23.4898 - val_accuracy: 0.1000\n",
            "Epoch 99/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 22.4975 - accuracy: 0.1100 - val_loss: 41.4927 - val_accuracy: 0.1000\n",
            "Epoch 100/100\n",
            "4/4 [==============================] - 0s 10ms/step - loss: 38.1962 - accuracy: 0.1000 - val_loss: 33.3583 - val_accuracy: 0.1000\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}