Tuning & optimization

Hyperparameter tuning

Learning rate

• Learning rate scheduler
• ReduceLROnPlateau
• Optuna https://optuna.org/
• CosineAnnealingLR https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html
• How to choose Learning rate scheduler
  • https://neptune.ai/blog/how-to-choose-a-learning-rate-scheduler
  • https://machinelearningmastery.com/using-learning-rate-schedules-deep-learning-models-python-keras/

Optimizer by hugginface

https://huggingface.co/docs/transformers/master/en/main_classes/optimizer_schedules#transformers.SchedulerType

Comments