4. Using Experiments

Although the process of training and evaluating models becomes easier due to the abstractions and facilities provided by this framework and Pytorch Lightning, we also aims to standarize the way we conduct experiments, in order to allow for a more systematic and organized approach to the development of models.

The LightningExperiment implements a default pipeline (similar to ones used in previous notebooks) and provides a set of default configurations and settings for the experiments. This includes the default configurations for the Lightning Trainer, Logger, and Callbacks, as well as model and data module configurations. Also, it standardize the ouputs of the experiments, and the way we log the hyperparameters and results. However, it also provides flexibility as the user can customize the experiment by overriding the default configurations and settings.

In this notebook, we will demonstrate how to use the LightningExperiment class to conduct experiments in a systematic and organized way.

Experiment Structure

The LightningExperiment follows the structure below. Each rectangle (vertex of the graph) corresponds to a class, and the arrows (edges of the graph) correspond to the inheritance relationship between the classes. Inside each rectangle, there are three boxes. The first box is the name of the class, the second box is the name of the attributes and their type, and the third box is the methods of that class, the input parameters and return type.

As derived classes inherit the attributes and methods of their parent classes, they have access to all the attributes and methods of the parent class. Methods named in italic are abstract methods, that is, they must be implemented in some of the derived classes (below him). Some methods are not abstract, thus they have a default implementation, but can be overriden by the user to customize the experiment. You may want to check some useful material, if you are not familiar with the concept of inheritance in object-oriented programming, such as this one from Real Python, that gives a comprehensive overview of inheritance in Python, or this one from Geeks For Geeks.

The Experiment class

The Experiment class is the base class for all experiments and includes the experiment_dir (where logs, checkpoints, and outputs are saved), the name and run_id (tipically, the time). The experiment directory is created when the experiment is instantiated, and the experiment_dir attribute is set to the path of the created directory. The experiment consist in 3 stages: setup, run and teardown. The run method is an abstract method, and must be implemented in the derived classes.

You can use the execute method to run the experiment, that will call the setup, run and teardown methods in sequence.

The LightningExperiment class

The LightningExperiment adds common parameters when using Pytorch Lightning for training or testing. Usually this is the base class for any experiment that uses Pytorch Lightning. This class also implements the run method, that execute a generic Pytorch Lightning pipeline, similar to ones used in previous notebooks. This pipeline calls some methods that it defines. In fact, the pseudo-code for pipeline implemented by the run method is:

1. Get the model and data module using get_model and get_data_module methods.

2. If self.load is provided (path to the checkpoint), load the checkpoint using the load_checkpoint method.

3. Get the callbacks and logger using get_callbacks and get_logger methods.

4. Log the hyperparameters of experiment and model using the log_hyperparameters method.

5. Get the trainer using the get_trainer method and attach the logger and callbacks.

6. Run the model using the run_model method.

The user can override these methods to customize the experiment. By default, get_callbacks, get_logger, load_checkpoint, and log_hyperparameters have default implementations, and get_data_module, get_model, get_trainer, and run_model are abstract methods that must be implemented by the derived class.

The LightningTrain and LightningTest classes

The LightningTrain and LightningTest classes are derived from LightningExperiment and are used to train and test models, respectively. These classes adds more specific parameters for different contexts, such as training and testing. For instance, training usually requires the number of epochs, learning rate, and other parameters. Both classes implement parent’s get_callbacks, get_trainer and run_model methods.

This standardizes the way we train and test models, logging the same information and using the same callbacks and loggers. It allows the user to focus on the model and data module, and not on the training and testing process, that is already standardized (and can be customized) and can be reused in different experiments. In fact, get_model and get_data_module are abstract methods that must be implemented by the derived class, as it varies according to the model and data module used in the experiment.

LightningSSLTrain class

While LightningTrain class allows training arbitrary models, the LightningSSLTrain class is a derived class used to train models using self-supervised learning. This class adds more 4 new abstract methods:

• get_pretrain_model and get_pretrain_data_module: the user must return the model and data module used to pretrain the model.

• get_finetune_model and get_finetune_data_module: the user must return the model and data module used to finetune the model.

The training_mode variable is also introduced and it is used to indicate if the model is in pretrain or finetune mode. Then, the get_model and get_data_module methods will call the get_pretrain_model and get_pretrain_data_module methods if training_mode is pretrain, and the get_finetune_model and get_finetune_data_module methods if training_mode is finetune.

One important thing to note is about load parameter. If it is provided, it will load the checkpoint for the model, in order to resume the training. This is valid both for pretrain and finetune modes. The load_backbone parameter is only used in finetune mode, in order to load the checkpoint for the backbone model, that is, load a model that was pretrained using self-supervised learning. This is usually used to start a finetuning from a model that was pretrained using self-supervised learning. If you want to resume a finetuning from a checkpoint, you should use the load parameter.

Running CPC Experiment

In this notebook, we will demonstrate how to run a CPC experiment, from pretrain to finetune and, finally, test. The CPCTrain class derives from LightningSSLTrain and implements the get_pretrain_model, get_pretrain_data_module, get_finetune_model and get_finetune_data_module methods, while the CPCTest class derives from LightningTest and implements the get_model and get_data_module methods. Both classes add specific parameters to instantiate CPC model and the respective data module.

Let’s first start by pretraining the CPC model, using KuHAR dataset, as in previous notebooks.

Experiment: Pretraining CPC

The CPCTrain class will encapsuate the default code for creating models and data modules from previous notebooks into the get_pretrain_model and get_pretrain_data_module methods. Thus, we just need to pass the required parameters to the CPCTrain class constructor and call the execute method to run the experiment. As CPCTrain is a derived class, we can pass the parameters from all parent classes (epochs, accelerator, seed, etc.), as well as the parameters from the CPCTrain class (window_size, num_classes, etc.) in the class constructor.

These main parameters include for CPCTrain class are:

• data: the path to the dataset folder. For pretrain, the data must be the path to a dataset where the samples are the whole time-series of an user. For finetune, the data must be the path to a dataset where the samples are the windows of the time-series, as in previous notebooks.

• encoding_size: the size of the latent representation of the CPC model.

• in_channel: the number of features in the input data.

• window_size: size of the input windows (X_t) to be fed to the encoder.

• pad_length: boolean indicating if the input windows should be padded to the window_size or not.

• num_classes: number of classes in the dataset.

• update_backbone: boolean indicating if the backbone should be updated during finetuning (only useful for fine-tuning process).

Only the data parameter is required, the others have default values. Please check the documentation of the CPCTrain class for more details.

Let’s create the CPCTrain class and run the pretraining experiment.

from ssl_tools.experiments.har_classification.cpc import CPCTrain

data_path = "/workspaces/hiaac-m4/ssl_tools/data/view_concatenated/KuHar_cpc"

cpc_experiment = CPCTrain(
    # General params
    training_mode="pretrain",
    # Data Module params
    data=data_path,
    # CPC model params
    encoding_size=150,
    window_size=60,
    in_channel=6,
    num_classes=6,
    # Trainer params
    epochs=10,
    batch_size=1,
    accelerator="gpu",
    devices=1,
)

cpc_experiment

[1706883882.274114] [aae107fc745c:2257856:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device

LightningExperiment(experiment_dir=logs/pretrain/CPC/2024-02-02_11-24-49, model=CPC, run_id=2024-02-02_11-24-49, finished=False)

# Executing the experiment. Result is the output of the run() method
result = cpc_experiment.execute()

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/pretrain/CPC/2024-02-02_11-24-49 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..

Setting up experiment: CPC...
Running experiment: CPC...
Training will start
        Experiment path: logs/pretrain/CPC/2024-02-02_11-24-49

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃   ┃ Name              ┃ Type             ┃ Params ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ encoder           │ GRUEncoder       │ 95.0 K │
│ 1 │ density_estimator │ Linear           │ 22.7 K │
│ 2 │ auto_regressor    │ GRU              │  135 K │
│ 3 │ loss_func         │ CrossEntropyLoss │      0 │
└───┴───────────────────┴──────────────────┴────────┘

Trainable params: 253 K
Non-trainable params: 0
Total params: 253 K
Total estimated model params size (MB): 1

`Trainer.fit` stopped: `max_epochs=10` reached.

--> Overall fit time: 31.599 seconds

Training finished
Last checkpoint saved at: logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt
Teardown experiment: CPC...

Once the experiment finished, we may have a directory structure similar to this:

logs/
    pretrain/
        CPC/
            2024-02-02_10-53-31/
                checkpoints/
                    epoch=9-step=570.ckpt
                    last.ckpt
                hparams.yaml
                metrics.csv

This is the default directory structure for experiments. The experiment directory is logs/pretrain/CPC/2024-02-01_22-01-31/, and can be accessed using the cpc_experiment.experiment_dir attribute. The checkpoints directory contains the saved checkpoints and inside it we may have a last.ckpt file which is the last checkpoint saved. It can be accessed using the cpc_experiment.checkpoint_dir attribute. The hparams.yaml file contains the hyperparameters, and the metrics.csv file contains the metrics logged during training.

We can obtain the experiment’s model, data module, logger, checkpoint directory, callbacks, trianer, and hyperparameters using the cpc_experiment.model, cpc_experiment.data_module, cpc_experiment.logger, cpc_experiment.checkpoint_dir, cpc_experiment.callbacks, cpc_experiment.trainer, and cpc_experiment.hyperparameters attributes, respectively. These objects are cached in the cpc_experiment object, thus, it is instantiated only once, and can be accessed multiple times. Also, the cpc_experiment.finished attribute is a boolean indicating if the experiment has finished sucessfuly or not.

For fine-tunning, we will need this checkpoint to load the weights of the backbone. Let’s obtain the checkpoint file and then run the finetuning experiment.

backbone_checkpoint_path = cpc_experiment.checkpoint_dir / "last.ckpt"
backbone_checkpoint_path

PosixPath('logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt')

Experiment: Fine-tunning CPC

The CPCTrain class also encapsuate the default code for creating models and data modules from previous notebooks into the get_finetune_model and get_finetune_data_module methods. The behaviour of these methods is similar to the get_pretrain_model and get_pretrain_data_module methods, but they are used to create the model and data module for the finetuning process. In fact, the get_finetune_model will encapsulate the CPC code inside SSLDisriminator class, as seen in previous notebooks.

As we use the same class for pretrain and finetune, we just need to set the training_mode attribute to finetune and set the load_backbone parameter to the checkpoint file obtained in the pretrain process, in order to load the weights of the backbone model. Then, we can call the execute method to run the experiment.

However, it worth to notice that fine tune is an supervised learning process and uses windowed time-series as input. Thus, the data parameter must be the path to a dataset where the samples are the windows of the time-series, as in previous notebooks. In our case, we will use the standardized balanced view of the KuHar dataset.

data_path = "/workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/KuHar/"

cpc_experiment = CPCTrain(
    # General params
    training_mode="finetune",
    load_backbone=backbone_checkpoint_path,
    # Data Module params
    data=data_path,
    # CPC model params
    encoding_size=150,
    window_size=60,
    in_channel=6,
    num_classes=6,
    # Trainer params
    epochs=10,
    num_workers=12,
    batch_size=128,
    accelerator="gpu",
    devices=1,
)

cpc_experiment

LightningExperiment(experiment_dir=logs/finetune/CPC/2024-02-02_11-31-12, model=CPC, run_id=2024-02-02_11-31-12, finished=False)

# Executing the experiment. Result is the output of the run() method
result = cpc_experiment.execute()

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/finetune/CPC/2024-02-02_11-31-12 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..

Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/pretrain/CPC/2024-02-02_11-24-49/checkpoints/last.ckpt...
Model loaded successfully
Training will start
        Experiment path: logs/finetune/CPC/2024-02-02_11-31-12

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

┏━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃   ┃ Name     ┃ Type              ┃ Params ┃
┡━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ backbone │ CPC               │  253 K │
│ 1 │ head     │ CPCPredictionHead │ 14.2 K │
│ 2 │ loss_fn  │ CrossEntropyLoss  │      0 │
└───┴──────────┴───────────────────┴────────┘

Trainable params: 14.2 K
Non-trainable params: 253 K
Total params: 267 K
Total estimated model params size (MB): 1

/usr/local/lib/python3.10/dist-packages/lightning/pytorch/loops/fit_loop.py:293: The number of training batches (11) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.

`Trainer.fit` stopped: `max_epochs=10` reached.

--> Overall fit time: 12.987 seconds

Training finished
Last checkpoint saved at: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt
Teardown experiment: CPC...

We will pick the last checkpoint from the fine-tuning process to evaluate the model.

fine_tuned_checkpoint_path = cpc_experiment.checkpoint_dir / "last.ckpt"
fine_tuned_checkpoint_path

PosixPath('logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt')

Experiment: Evaluating CPC Performance

Finally, we can evaluate the performance of the CPC model using the CPCTest class. This class inherits from LightningTest and encapsulate the default code for creating models and data modules from previous notebooks into the get_model and get_data_module methods.

The signature of the CPCTest class is very similar to the CPCTrain class. However, differently from the train process the test process uses the .test() method in the Trainer and not the .fit() method. Also, the load parameter is used to load the checkpoint obtained in the finetuning process (that load the weights from SSLDiscriminator, backbone and prediction head).

Let’s create experiments to test the CPC model, using the test set from different datasets besides KuHAR.

from pathlib import Path
from ssl_tools.experiments.har_classification.cpc import CPCTest

root_datasets_path = Path("/workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/")

datasets = [
    "KuHar",
    "MotionSense",
    "RealWorld_thigh",
    "RealWorld_waist",
    "UCI",
]

results = dict()
for dataset in datasets:
    data_path = root_datasets_path / dataset
    print(f"Dataset at: {data_path}")
    cpc_experiment = CPCTest(
        # General params
        load=fine_tuned_checkpoint_path,
        # Data Module params
        data=data_path,
        # CPC model params
        encoding_size=150,
        window_size=60,
        in_channel=6,
        num_classes=6,
        # Trainer params
        batch_size=256,
        accelerator="gpu",
        devices=1,
    )
    print(f"Loading model from {fine_tuned_checkpoint_path} and executing test using dataset at {dataset}...")
    results[dataset] = cpc_experiment.execute()
    print(f"Test on dataset {dataset} finished!")

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-55 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/KuHar
Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at KuHar...
Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...
Model loaded successfully

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test_acc          │    0.4652777910232544     │
│         test_loss         │    1.5481091737747192     │
└───────────────────────────┴───────────────────────────┘

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-57 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..

Teardown experiment: CPC...
Test on dataset KuHar finished!
Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/MotionSense
Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at MotionSense...
Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...
Model loaded successfully

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test_acc          │    0.43879473209381104    │
│         test_loss         │    1.5955957174301147     │
└───────────────────────────┴───────────────────────────┘

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-32-59 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..

Teardown experiment: CPC...
Test on dataset MotionSense finished!
Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/RealWorld_thigh
Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at RealWorld_thigh...
Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...
Model loaded successfully

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test_acc          │    0.40372908115386963    │
│         test_loss         │     1.643248438835144     │
└───────────────────────────┴───────────────────────────┘

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-33-01 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..

Teardown experiment: CPC...
Test on dataset RealWorld_thigh finished!
Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/RealWorld_waist
Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at RealWorld_waist...
Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...
Model loaded successfully

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test_acc          │    0.4031635820865631     │
│         test_loss         │    1.6421446800231934     │
└───────────────────────────┴───────────────────────────┘

/usr/local/lib/python3.10/dist-packages/lightning/fabric/loggers/csv_logs.py:198: Experiment logs directory logs/test/CPC/2024-02-02_11-33-03 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]

Teardown experiment: CPC...
Test on dataset RealWorld_waist finished!
Dataset at: /workspaces/hiaac-m4/ssl_tools/data/standartized_balanced/UCI
Loading model from logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt and executing test using dataset at UCI...
Setting up experiment: CPC...
Running experiment: CPC...
Loading model from: logs/finetune/CPC/2024-02-02_11-31-12/checkpoints/last.ckpt...
Model loaded successfully

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         test_acc          │    0.35652172565460205    │
│         test_loss         │    1.6707664728164673     │
└───────────────────────────┴───────────────────────────┘

Teardown experiment: CPC...
Test on dataset UCI finished!

# We can acess the results
results

{'KuHar': [{'test_loss': 1.5481091737747192, 'test_acc': 0.4652777910232544}],
 'MotionSense': [{'test_loss': 1.5955957174301147,
   'test_acc': 0.43879473209381104}],
 'RealWorld_thigh': [{'test_loss': 1.643248438835144,
   'test_acc': 0.40372908115386963}],
 'RealWorld_waist': [{'test_loss': 1.6421446800231934,
   'test_acc': 0.4031635820865631}],
 'UCI': [{'test_loss': 1.6707664728164673, 'test_acc': 0.35652172565460205}]}

Other advantages of using LightningExperiment

The LightningExperiment class also provides other advantages, such as:

• Automatically generate CLI applications for the experiments, using the jsonargparse library. In fact, every parameter in the class constructor is automatically converted to a command line argument. This allows the user to run the experiment from the command line, using the same parameters as in the class constructor.

• Default metrics.csv and hparams.yaml files are created, and the hyperparameters are logged in the hparams.yaml file. The metrics.csv file contains the metrics logged during training, and can be used to analyze the performance of the model.