❓How to Estimate Training Time in FPT AI Studio
Objectives
How to predict fine-tuning time before launching a job.
Understand how to check ETA during training.
Plan GPU resource allocation more effectively.
Overview
Assume fixed GPUs. Fine-tuning time depends on several key factors:
Model size
Number of parameters
Dataset size
Number of training samples
Sequence length
Max token length per sample
Batch size
Samples processed per step
Epochs
Number of full dataset passes
Method 1: Empirical Estimation (Based on Real Benchmark)
Description
This method estimates total training time by running a small benchmark job on a subset of the data and scaling up the results. It’s simple, practical, and highly accurate for your specific hardware setup.
Steps to perform
Run a mini benchmark
Use 100-1000 samples from your dataset.
Train for 1 epoch with your target hyperparameters.
Record the training time for this mini-run (e.g., total time for 1 epoch).
Estimate total training time Use a simple proportional scaling formula:
Example: Training time for 1,000 samples (1 epoch) = 5 minutes Full dataset size = 20,000 samples Planned epochs = 3 scaling_factor = dataset_size / sample_dataset_size scaling_factor = 20,000 / 1,000 = 20 total_time = scaling_factor * epoch_time * epochs total_time = 20 * 5 minutes * 3 = 300 minutes = 5 hours
Method 2: Online Estimation (Dynamic ETA)
Description
Once training starts, FPT AI Studio can dynamically estimate the remaining time (ETA) based on the actual average step time during runtime.
How it works in FPT AI Studio
The backend automatically logs the training time for each logging interval (defined by your
logging_stepssetting).These logs are stored in the Logs → training-* section of the Studio interface.
You can view ETA updates directly in the dashboard or download the full log file for detailed analysis.
Comparison of Both Methods
When to use
Before training
During training
Requires trial run?
Yes
No (uses live data)
Main purpose
GPU/time planning
Real-time progress tracking
Best Practices
Start with Empirical Estimation for a rough pre-training plan.
Use Dynamic ETA to monitor progress and verify your estimate.
Last updated