{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 4. Using Experiments\n",
"\n",
"Although the process of training and evaluating models becomes easier due to the abstractions and facilities provided by this framework and Pytorch Lightning, we also aims to standarize the way we conduct experiments, in order to allow for a more systematic and organized approach to the development of models.\n",
"\n",
"The `LightningExperiment` implements a default pipeline (similar to ones used in previous notebooks) and provides a set of default configurations and settings for the experiments. \n",
"This includes the default configurations for the Lightning Trainer, Logger, and Callbacks, as well as model and data module configurations.\n",
"Also, it standardize the ouputs of the experiments, and the way we log the hyperparameters and results.\n",
"However, it also provides flexibility as the user can customize the experiment by overriding the default configurations and settings.\n",
"\n",
"In this notebook, we will demonstrate how to use the `LightningExperiment` class to conduct experiments in a systematic and organized way."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Experiment Structure\n",
"\n",
"The `LightningExperiment` follows the structure below. Each rectangle (vertex of the graph) corresponds to a class, and the arrows (edges of the graph) correspond to the inheritance relationship between the classes. Inside each rectangle, there are three boxes. The first box is the name of the class, the second box is the name of the attributes and their type, and the third box is the methods of that class, the input parameters and return type. \n",
"\n",
"As derived classes inherit the attributes and methods of their parent classes, they have access to all the attributes and methods of the parent class. \n",
"Methods named in italic are abstract methods, that is, they must be implemented in some of the derived classes (below him). \n",
"Some methods are not abstract, thus they have a default implementation, but can be overriden by the user to customize the experiment.\n",
"You may want to check some useful material, if you are not familiar with the concept of inheritance in object-oriented programming, such as [this one from Real Python](https://realpython.com/inheritance-composition-python/), that gives a comprehensive overview of inheritance in Python, or [this one from Geeks For Geeks](https://www.geeksforgeeks.org/inheritance-in-python/).\n",
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The `Experiment` class\n",
"\n",
"The `Experiment` class is the base class for all experiments and includes the `experiment_dir` (where logs, checkpoints, and outputs are saved), the `name` and `run_id` (tipically, the time).\n",
"The experiment directory is created when the experiment is instantiated, and the `experiment_dir` attribute is set to the path of the created directory.\n",
"The experiment consist in 3 stages: `setup`, `run` and `teardown`.\n",
"The `run` method is an abstract method, and must be implemented in the derived classes.\n",
"\n",
"You can use the `execute` method to run the experiment, that will call the `setup`, `run` and `teardown` methods in sequence."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The `LightningExperiment` class\n",
"\n",
"The `LightningExperiment` adds common parameters when using Pytorch Lightning for training or testing. \n",
"Usually this is the base class for any experiment that uses Pytorch Lightning.\n",
"This class also implements the `run` method, that execute a generic Pytorch Lightning pipeline, similar to ones used in previous notebooks. \n",
"This pipeline calls some methods that it defines.\n",
"In fact, the pseudo-code for pipeline implemented by the `run` method is:\n",
"\n",
"1. Get the model and data module using `get_model` and `get_data_module` methods.\n",
"2. If `self.load` is provided (path to the checkpoint), load the checkpoint using the `load_checkpoint` method.\n",
"3. Get the callbacks and logger using `get_callbacks` and `get_logger` methods.\n",
"4. Log the hyperparameters of experiment and model using the `log_hyperparameters` method.\n",
"5. Get the trainer using the `get_trainer` method and attach the logger and callbacks.\n",
"6. Run the model using the `run_model` method.\n",
"\n",
"The user can override these methods to customize the experiment. By default, `get_callbacks`, `get_logger`, `load_checkpoint`, and `log_hyperparameters` have default implementations, and `get_data_module`, `get_model`, `get_trainer`, and `run_model` are abstract methods that must be implemented by the derived class."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The `LightningTrain` and `LightningTest` classes\n",
"\n",
"The `LightningTrain` and `LightningTest` classes are derived from `LightningExperiment` and are used to train and test models, respectively. \n",
"These classes adds more specific parameters for different contexts, such as training and testing. \n",
"For instance, training usually requires the number of epochs, learning rate, and other parameters.\n",
"Both classes implement parent's `get_callbacks`, `get_trainer` and `run_model` methods. \n",
"\n",
"This standardizes the way we train and test models, logging the same information and using the same callbacks and loggers. \n",
"It allows the user to focus on the model and data module, and not on the training and testing process, that is already standardized (and can be customized) and can be reused in different experiments.\n",
"In fact, `get_model` and `get_data_module` are abstract methods that must be implemented by the derived class, as it varies according to the model and data module used in the experiment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `LightningSSLTrain` class\n",
"\n",
"While `LightningTrain` class allows training arbitrary models, the `LightningSSLTrain` class is a derived class used to train models using self-supervised learning. This class adds more 4 new abstract methods: \n",
"\n",
"* `get_pretrain_model` and `get_pretrain_data_module`: the user must return the model and data module used to pretrain the model.\n",
"* `get_finetune_model` and `get_finetune_data_module`: the user must return the model and data module used to finetune the model.\n",
"\n",
"The `training_mode` variable is also introduced and it is used to indicate if the model is in pretrain or finetune mode. \n",
"Then, the `get_model` and `get_data_module` methods will call the `get_pretrain_model` and `get_pretrain_data_module` methods if `training_mode` is `pretrain`, and the `get_finetune_model` and `get_finetune_data_module` methods if `training_mode` is `finetune`. \n",
"\n",
"One important thing to note is about `load` parameter. \n",
"If it is provided, it will load the checkpoint for the model, in order to resume the training. This is valid both for pretrain and finetune modes. \n",
"The `load_backbone` parameter is only used in finetune mode, in order to load the checkpoint for the backbone model, that is, load a model that was pretrained using self-supervised learning. This is usually used to start a finetuning from a model that was pretrained using self-supervised learning. If you want to resume a finetuning from a checkpoint, you should use the `load` parameter."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running CPC Experiment\n",
"\n",
"In this notebook, we will demonstrate how to run a CPC experiment, from pretrain to finetune and, finally, test. \n",
"The `CPCTrain` class derives from `LightningSSLTrain` and implements the `get_pretrain_model`, `get_pretrain_data_module`, `get_finetune_model` and `get_finetune_data_module` methods, while the `CPCTest` class derives from `LightningTest` and implements the `get_model` and `get_data_module` methods.\n",
"Both classes add specific parameters to instantiate CPC model and the respective data module.\n",
"\n",
"Let's first start by pretraining the CPC model, using KuHAR dataset, as in previous notebooks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Experiment: Pretraining CPC\n",
"\n",
"The `CPCTrain` class will encapsuate the default code for creating models and data modules from previous notebooks into the `get_pretrain_model` and `get_pretrain_data_module` methods. \n",
"Thus, we just need to pass the required parameters to the `CPCTrain` class constructor and call the `execute` method to run the experiment.\n",
"As `CPCTrain` is a derived class, we can pass the parameters from all parent classes (`epochs`, `accelerator`, `seed`, *etc.*), as well as the parameters from the `CPCTrain` class (`window_size`, `num_classes`, *etc.*) in the class constructor.\n",
"\n",
"These main parameters include for `CPCTrain` class are:\n",
"\n",
"* `data`: the path to the dataset folder. For pretrain, the data must be the path to a dataset where the samples are the whole time-series of an user. For finetune, the data must be the path to a dataset where the samples are the windows of the time-series, as in previous notebooks.\n",
"* `encoding_size`: the size of the latent representation of the CPC model.\n",
"* `in_channel`: the number of features in the input data.\n",
"* `window_size`: size of the input windows (`X_t`) to be fed to the encoder.\n",
"* `pad_length`: boolean indicating if the input windows should be padded to the `window_size` or not.\n",
"* `num_classes`: number of classes in the dataset.\n",
"* `update_backbone`: boolean indicating if the backbone should be updated during finetuning (only useful for fine-tuning process).\n",
"\n",
"Only the `data` parameter is required, the others have default values. \n",
"Please check the documentation of the `CPCTrain` class for more details.\n",
"\n",
"Let's create the `CPCTrain` class and run the pretraining experiment."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1706883882.274114] [aae107fc745c:2257856:f] vfs_fuse.c:281 UCX ERROR inotify_add_watch(/tmp) failed: No space left on device\n"
]
},
{
"data": {
"text/plain": [
"LightningExperiment(experiment_dir=logs/pretrain/CPC/2024-02-02_11-24-49, model=CPC, run_id=2024-02-02_11-24-49, finished=False)"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from ssl_tools.experiments.har_classification.cpc import CPCTrain\n",
"\n",
"data_path = \"/workspaces/hiaac-m4/ssl_tools/data/view_concatenated/KuHar_cpc\"\n",
"\n",
"cpc_experiment = CPCTrain(\n",
" # General params\n",
" training_mode=\"pretrain\",\n",
" # Data Module params\n",
" data=data_path,\n",
" # CPC model params\n",
" encoding_size=150,\n",
" window_size=60,\n",
" in_channel=6,\n",
" num_classes=6,\n",
" # Trainer params\n",
" epochs=10,\n",
" batch_size=1,\n",
" accelerator=\"gpu\",\n",
" devices=1,\n",
")\n",
"\n",
"cpc_experiment"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/pretrain/CPC/2024-02-02_11-24-49 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..\n",
"`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Training will start\n",
"\tExperiment path: logs/pretrain/CPC/2024-02-02_11-24-49\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"data": {
"text/html": [
"┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓\n",
"┃ ┃ Name ┃ Type ┃ Params ┃\n",
"┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩\n",
"│ 0 │ encoder │ GRUEncoder │ 95.0 K │\n",
"│ 1 │ density_estimator │ Linear │ 22.7 K │\n",
"│ 2 │ auto_regressor │ GRU │ 135 K │\n",
"│ 3 │ loss_func │ CrossEntropyLoss │ 0 │\n",
"└───┴───────────────────┴──────────────────┴────────┘\n",
"\n"
],
"text/plain": [
"┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓\n",
"┃\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mName \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mType \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mParams\u001b[0m\u001b[1;35m \u001b[0m┃\n",
"┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩\n",
"│\u001b[2m \u001b[0m\u001b[2m0\u001b[0m\u001b[2m \u001b[0m│ encoder │ GRUEncoder │ 95.0 K │\n",
"│\u001b[2m \u001b[0m\u001b[2m1\u001b[0m\u001b[2m \u001b[0m│ density_estimator │ Linear │ 22.7 K │\n",
"│\u001b[2m \u001b[0m\u001b[2m2\u001b[0m\u001b[2m \u001b[0m│ auto_regressor │ GRU │ 135 K │\n",
"│\u001b[2m \u001b[0m\u001b[2m3\u001b[0m\u001b[2m \u001b[0m│ loss_func │ CrossEntropyLoss │ 0 │\n",
"└───┴───────────────────┴──────────────────┴────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Trainable params: 253 K \n",
"Non-trainable params: 0 \n",
"Total params: 253 K \n",
"Total estimated model params size (MB): 1 \n",
"\n"
],
"text/plain": [
"\u001b[1mTrainable params\u001b[0m: 253 K \n",
"\u001b[1mNon-trainable params\u001b[0m: 0 \n",
"\u001b[1mTotal params\u001b[0m: 253 K \n",
"\u001b[1mTotal estimated model params size (MB)\u001b[0m: 1 \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "84a17deffcaa4fa992ad2c2e009290d2",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"`Trainer.fit` stopped: `max_epochs=10` reached.\n"
]
},
{
"data": {
"text/html": [
"--> Overall fit time: 31.599 seconds\n",
"\n"
],
"text/plain": [
"--> Overall fit time: 31.599 seconds\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training finished\n",
"Last checkpoint saved at: logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt\n",
"Teardown experiment: CPC...\n"
]
}
],
"source": [
"# Executing the experiment. Result is the output of the run() method\n",
"result = cpc_experiment.execute() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once the experiment finished, we may have a directory structure similar to this:\n",
"\n",
"```\n",
"logs/\n",
" pretrain/\n",
" CPC/\n",
" 2024-02-02_10-53-31/\n",
" checkpoints/\n",
" epoch=9-step=570.ckpt\n",
" last.ckpt\n",
" hparams.yaml\n",
" metrics.csv\n",
"```\n",
"\n",
"This is the default directory structure for experiments. The experiment directory is `logs/pretrain/CPC/2024-02-01_22-01-31/`, and can be accessed using the `cpc_experiment.experiment_dir` attribute. The `checkpoints directory` contains the saved checkpoints and inside it we may have a `last.ckpt` file which is the last checkpoint saved. It can be accessed using the `cpc_experiment.checkpoint_dir` attribute. The `hparams.yaml` file contains the hyperparameters, and the `metrics.csv` file contains the metrics logged during training.\n",
"\n",
"We can obtain the experiment's model, data module, logger, checkpoint directory, callbacks, trianer, and hyperparameters using the `cpc_experiment.model`, `cpc_experiment.data_module`, `cpc_experiment.logger`, `cpc_experiment.checkpoint_dir`, `cpc_experiment.callbacks`, `cpc_experiment.trainer`, and `cpc_experiment.hyperparameters` attributes, respectively. \n",
"These objects are cached in the `cpc_experiment` object, thus, it is instantiated only once, and can be accessed multiple times.\n",
"Also, the `cpc_experiment.finished` attribute is a boolean indicating if the experiment has finished sucessfuly or not.\n",
"\n",
"For fine-tunning, we will need this checkpoint to load the weights of the backbone.\n",
"Let's obtain the checkpoint file and then run the finetuning experiment."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"PosixPath('logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"backbone_checkpoint_path = cpc_experiment.checkpoint_dir / \"last.ckpt\"\n",
"backbone_checkpoint_path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Experiment: Fine-tunning CPC\n",
"\n",
"The `CPCTrain` class also encapsuate the default code for creating models and data modules from previous notebooks into the `get_finetune_model` and `get_finetune_data_module` methods. \n",
"The behaviour of these methods is similar to the `get_pretrain_model` and `get_pretrain_data_module` methods, but they are used to create the model and data module for the finetuning process.\n",
"In fact, the `get_finetune_model` will encapsulate the CPC code inside `SSLDisriminator` class, as seen in previous notebooks.\n",
"\n",
"As we use the same class for pretrain and finetune, we just need to set the `training_mode` attribute to `finetune` and set the `load_backbone` parameter to the checkpoint file obtained in the pretrain process, in order to load the weights of the backbone model. \n",
"Then, we can call the `execute` method to run the experiment.\n",
"\n",
"However, it worth to notice that fine tune is an supervised learning process and uses windowed time-series as input. \n",
"Thus, the `data` parameter must be the path to a dataset where the samples are the windows of the time-series, as in previous notebooks. \n",
"In our case, we will use the standardized balanced view of the KuHar dataset."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"LightningExperiment(experiment_dir=logs/finetune/CPC/2024-02-02_11-31-12, model=CPC, run_id=2024-02-02_11-31-12, finished=False)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_path = \"/workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/KuHar/\"\n",
"\n",
"cpc_experiment = CPCTrain(\n",
" # General params\n",
" training_mode=\"finetune\",\n",
" load_backbone=backbone_checkpoint_path,\n",
" # Data Module params\n",
" data=data_path,\n",
" # CPC model params\n",
" encoding_size=150,\n",
" window_size=60,\n",
" in_channel=6,\n",
" num_classes=6,\n",
" # Trainer params\n",
" epochs=10,\n",
" num_workers=12,\n",
" batch_size=128,\n",
" accelerator=\"gpu\",\n",
" devices=1,\n",
")\n",
"\n",
"cpc_experiment"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/finetune/CPC/2024-02-02_11-31-12 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..\n",
"`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt...\n",
"Model loaded successfully\n",
"Training will start\n",
"\tExperiment path: logs/finetune/CPC/2024-02-02_11-31-12\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"data": {
"text/html": [
"┏━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓\n",
"┃ ┃ Name ┃ Type ┃ Params ┃\n",
"┡━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩\n",
"│ 0 │ backbone │ CPC │ 253 K │\n",
"│ 1 │ head │ CPCPredictionHead │ 14.2 K │\n",
"│ 2 │ loss_fn │ CrossEntropyLoss │ 0 │\n",
"└───┴──────────┴───────────────────┴────────┘\n",
"\n"
],
"text/plain": [
"┏━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓\n",
"┃\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mName \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mType \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mParams\u001b[0m\u001b[1;35m \u001b[0m┃\n",
"┡━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩\n",
"│\u001b[2m \u001b[0m\u001b[2m0\u001b[0m\u001b[2m \u001b[0m│ backbone │ CPC │ 253 K │\n",
"│\u001b[2m \u001b[0m\u001b[2m1\u001b[0m\u001b[2m \u001b[0m│ head │ CPCPredictionHead │ 14.2 K │\n",
"│\u001b[2m \u001b[0m\u001b[2m2\u001b[0m\u001b[2m \u001b[0m│ loss_fn │ CrossEntropyLoss │ 0 │\n",
"└───┴──────────┴───────────────────┴────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Trainable params: 14.2 K \n",
"Non-trainable params: 253 K \n",
"Total params: 267 K \n",
"Total estimated model params size (MB): 1 \n",
"\n"
],
"text/plain": [
"\u001b[1mTrainable params\u001b[0m: 14.2 K \n",
"\u001b[1mNon-trainable params\u001b[0m: 253 K \n",
"\u001b[1mTotal params\u001b[0m: 267 K \n",
"\u001b[1mTotal estimated model params size (MB)\u001b[0m: 1 \n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "92081a68a29343e18b11f8ae49a8f37c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py:293: The number of training batches (11) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
"\n"
],
"text/plain": [
"/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py:293: The number of training batches (11) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"`Trainer.fit` stopped: `max_epochs=10` reached.\n"
]
},
{
"data": {
"text/html": [
"--> Overall fit time: 12.987 seconds\n",
"\n"
],
"text/plain": [
"--> Overall fit time: 12.987 seconds\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training finished\n",
"Last checkpoint saved at: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt\n",
"Teardown experiment: CPC...\n"
]
}
],
"source": [
"# Executing the experiment. Result is the output of the run() method\n",
"result = cpc_experiment.execute() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will pick the last checkpoint from the fine-tuning process to evaluate the model."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"PosixPath('logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fine_tuned_checkpoint_path = cpc_experiment.checkpoint_dir / \"last.ckpt\"\n",
"fine_tuned_checkpoint_path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiment: Evaluating CPC Performance\n",
"\n",
"Finally, we can evaluate the performance of the CPC model using the `CPCTest` class. \n",
"This class inherits from `LightningTest` and encapsulate the default code for creating models and data modules from previous notebooks into the `get_model` and `get_data_module` methods.\n",
"\n",
"The signature of the `CPCTest` class is very similar to the `CPCTrain` class. However, differently from the train process the test process uses the `.test()` method in the Trainer and not the `.fit()` method.\n",
"Also, the `load` parameter is used to load the checkpoint obtained in the finetuning process (that load the weights from `SSLDiscriminator`, backbone and prediction head).\n",
"\n",
"Let's create experiments to test the CPC model, using the test set from different datasets besides KuHAR."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-55 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n",
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/KuHar\n",
"Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at KuHar...\n",
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...\n",
"Model loaded successfully\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d2398c86ef214c4d96783616d573caf7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Test metric ┃ DataLoader 0 ┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ test_acc │ 0.4652777910232544 │\n",
"│ test_loss │ 1.5481091737747192 │\n",
"└───────────────────────────┴───────────────────────────┘\n",
"\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.4652777910232544 \u001b[0m\u001b[35m \u001b[0m│\n",
"│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 1.5481091737747192 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-57 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Teardown experiment: CPC...\n",
"Test on dataset KuHar finished!\n",
"Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/MotionSense\n",
"Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at MotionSense...\n",
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...\n",
"Model loaded successfully\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "813175344f71488aaf2c634d00c5f080",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Test metric ┃ DataLoader 0 ┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ test_acc │ 0.43879473209381104 │\n",
"│ test_loss │ 1.5955957174301147 │\n",
"└───────────────────────────┴───────────────────────────┘\n",
"\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.43879473209381104 \u001b[0m\u001b[35m \u001b[0m│\n",
"│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 1.5955957174301147 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-59 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Teardown experiment: CPC...\n",
"Test on dataset MotionSense finished!\n",
"Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/RealWorld_thigh\n",
"Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at RealWorld_thigh...\n",
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...\n",
"Model loaded successfully\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "612ec0d361884c7399217888425f69d8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Test metric ┃ DataLoader 0 ┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ test_acc │ 0.40372908115386963 │\n",
"│ test_loss │ 1.643248438835144 │\n",
"└───────────────────────────┴───────────────────────────┘\n",
"\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.40372908115386963 \u001b[0m\u001b[35m \u001b[0m│\n",
"│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 1.643248438835144 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-33-01 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Teardown experiment: CPC...\n",
"Test on dataset RealWorld_thigh finished!\n",
"Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/RealWorld_waist\n",
"Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at RealWorld_waist...\n",
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...\n",
"Model loaded successfully\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1d507d05b32d4f5e9563d6393d53b022",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Test metric ┃ DataLoader 0 ┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ test_acc │ 0.4031635820865631 │\n",
"│ test_loss │ 1.6421446800231934 │\n",
"└───────────────────────────┴───────────────────────────┘\n",
"\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.4031635820865631 \u001b[0m\u001b[35m \u001b[0m│\n",
"│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 1.6421446800231934 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-33-03 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!\n",
"GPU available: True (cuda), used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"HPU available: False, using: 0 HPUs\n",
"`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..\n",
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Teardown experiment: CPC...\n",
"Test on dataset RealWorld_waist finished!\n",
"Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/UCI\n",
"Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at UCI...\n",
"Setting up experiment: CPC...\n",
"Running experiment: CPC...\n",
"Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...\n",
"Model loaded successfully\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a48db052fd0141dea8947f2ffebf7fbf",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Output()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃ Test metric ┃ DataLoader 0 ┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│ test_acc │ 0.35652172565460205 │\n",
"│ test_loss │ 1.6707664728164673 │\n",
"└───────────────────────────┴───────────────────────────┘\n",
"\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.35652172565460205 \u001b[0m\u001b[35m \u001b[0m│\n",
"│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 1.6707664728164673 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Teardown experiment: CPC...\n",
"Test on dataset UCI finished!\n"
]
}
],
"source": [
"from pathlib import Path\n",
"from ssl_tools.experiments.har_classification.cpc import CPCTest\n",
"\n",
"root_datasets_path = Path(\"/workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/\")\n",
"\n",
"datasets = [\n",
" \"KuHar\",\n",
" \"MotionSense\",\n",
" \"RealWorld_thigh\",\n",
" \"RealWorld_waist\",\n",
" \"UCI\",\n",
"]\n",
"\n",
"results = dict()\n",
"for dataset in datasets:\n",
" data_path = root_datasets_path / dataset\n",
" print(f\"Dataset at: {data_path}\")\n",
" cpc_experiment = CPCTest(\n",
" # General params\n",
" load=fine_tuned_checkpoint_path,\n",
" # Data Module params\n",
" data=data_path,\n",
" # CPC model params\n",
" encoding_size=150,\n",
" window_size=60,\n",
" in_channel=6,\n",
" num_classes=6,\n",
" # Trainer params\n",
" batch_size=256,\n",
" accelerator=\"gpu\",\n",
" devices=1,\n",
" )\n",
" print(f\"Loading model from {fine_tuned_checkpoint_path} and executing test using dataset at {dataset}...\")\n",
" results[dataset] = cpc_experiment.execute()\n",
" print(f\"Test on dataset {dataset} finished!\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'KuHar': [{'test_loss': 1.5481091737747192, 'test_acc': 0.4652777910232544}],\n",
" 'MotionSense': [{'test_loss': 1.5955957174301147,\n",
" 'test_acc': 0.43879473209381104}],\n",
" 'RealWorld_thigh': [{'test_loss': 1.643248438835144,\n",
" 'test_acc': 0.40372908115386963}],\n",
" 'RealWorld_waist': [{'test_loss': 1.6421446800231934,\n",
" 'test_acc': 0.4031635820865631}],\n",
" 'UCI': [{'test_loss': 1.6707664728164673, 'test_acc': 0.35652172565460205}]}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# We can acess the results\n",
"results"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other advantages of using `LightningExperiment`\n",
"\n",
"The `LightningExperiment` class also provides other advantages, such as:\n",
"\n",
"* Automatically generate CLI applications for the experiments, using the `jsonargparse` library. In fact, every parameter in the class constructor is automatically converted to a command line argument. This allows the user to run the experiment from the command line, using the same parameters as in the class constructor.\n",
"* Default `metrics.csv` and `hparams.yaml` files are created, and the hyperparameters are logged in the `hparams.yaml` file. The `metrics.csv` file contains the metrics logged during training, and can be used to analyze the performance of the model."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}