Comparison of Google TPUv2 and Nvidia V100 on ResNet-50

Google recently added the Tensor Processing Unit v2 (TPUv2) , a processor specifically designed to accelerate deep learning , to the list of cloud services . This is the second generation of the world's first widely available deep learning accelerator, which claims to be an alternative to Nvidia GPUs. Recently, we talked about first impressions . Many asked for a more detailed comparison with the Nvidia V100 GPUs .

To objectively and meaningfully compare deep learning accelerators is a non-trivial task. But because of the future importance of this category of products and the lack of detailed comparisons, we felt the need to conduct independent tests. This includes taking into account the opinions of potentially opposing parties. That's why we contacted Google and Nvidia engineers - and invited them to comment on a draft of this article. In order to guarantee a lack of bias, we also invited independent experts. Thanks to this, as far as we know, the most complete comparison of TPUv2 and V100 to date has turned out.

Experimental setup

Below are compared four TPUv2 (which form one Cloud TPU) with four Nvidia V100. Both have full memory of 64 GB, so you can train the same models on them with the same amount of training sample. In the experiments, we train the models the same way: four TPUv2 in the Cloud TPU and four V100 perform the task of synchronous parallel distributed learning.

As a model, we chose ResNet-50 on ImageNet , the de facto standard and benchmark for classifying images. The reference ResNet-50 implementations are publicly available, but none of them support learning on both the Cloud TPU and several GPUs at the same time.

Nvidia recommends using MXNet for multiple V100sor TensorFlow implementations available as Docker images on the Nvidia GPU Cloud . Unfortunately, it turned out that both implementations do not very well agree with the default settings when working on multiple GPUs with large training samples. It is necessary to make changes, in particular, to the learning rate schedule.

Instead, we took the implementation of ResNet-50 from the benchmark repositoryTensorFlow and launched it as a Docker image (tensorflow / tensorflow: 1.7.0-gpu, CUDA 9.0, CuDNN 7.1.2). It is significantly faster than the TensorFlow implementation recommended by Nvidia and is only slightly inferior (by about 3%, see below) to MXNet implementation. But it converges well. In addition, there is an additional advantage that we compare two implementations on the same version of the framework (TensorFlow 1.7.0).

Google recommends using bfloat16 with TensorFlow 1.7.0 from the official TPU repository for Cloud TPU. Both implementations - TPU and GPU - use mixed precision computations on the corresponding architecture, and most tensors are stored in half-precision numbers.

V100 tests were run on the p3.8xlarge instance (16 cores Xeon [email protected], 244 GB of memory, Ubuntu 16.04) on AWS with four V100 GPUs (each with 16 GB of memory). TPU tests were run on a small instance n1-standard-4 (2 [email protected] cores, 15 GB of memory, Debian 9), for which Cloud TPU (v2-8) from four TPUv2 (each with 16 GB of memory) was allocated .

We made two different comparisons. First, we studied performance in terms of bandwidth (images per second) on synthetic data without augmentation, that is, without creating additional training data from the available data. This comparison does not depend on convergence, there are no bottlenecks in I / O, and data augmentation does not affect the result. The second comparison examined the accuracy and convergence of two implementations on ImageNet.

Bandwidth test

We measured the throughput by the number of images per second on synthetic data , that is, with the creation of data for on-the-fly training, at various batch sizes. Note that for a TPU, only a sample size of 1024 is recommended, but due to numerous requests from readers, we are reporting other results.


Performance (images per second) on various sample sizes on synthetic data and without augmentation. The sample sizes are “global”, that is, 1024 means the size of 256 on each of the GPU / TPU chips at each step

With a training sample size of 1024, there is virtually no difference in bandwidth! TPU is only slightly ahead with a margin of about 2%. On smaller training samples, bandwidth drops on both platforms, and GPUs work slightly better. But as mentioned above, such training sample sizes are not currently recommended for TPUs.

Following the recommendations of Nvidia, we conducted an experiment with the GPU on MXNet . We used the implementation of ResNet-50 in the Docker image ( mxnet: 18.03-py3 ), available in the Nvidia GPU Cloud . With a training sample size of 768 (1024 is too much), GPUs process about 3280 images per second. This is about 3% faster than the best result for TPU. But as mentioned above, the MXNet implementation does not converge very well across multiple GPUs with this training sample size, so here and below we will focus on the TensorFlow implementation.

Cost in the cloud

Cloud TPU (four TPUv2 chips) is currently only available on Google Cloud. It connects on demand to any VM instance only when such calculations are required. For the V100, we reviewed the AWS cloud solution (the V100 is not yet available in the Google Cloud). Based on the results above, we can normalize the number of images per second per dollar for each platform and provider.

Performance: images per second per dollar

	Cloud TPU	4 × V100	4 × V100
Cloud	Google cloud	Aws	AWS Reserved Instance
Price per hour	$ 6.7	$ 12.2	$ 8.4
Images per second	3186	3128	3128
Performance (images per second per dollar)	476	256	374

With these prices, Cloud TPU is the clear winner. However, the situation may look different if you are considering a longer term rental or equipment purchase (although this option is currently not available for Cloud TPU). The table above also includes the price of a reserved instance p3.8xlarge on AWS for rent for 12 months (without prepayment). This greatly improves one-dollar productivity to 374 images / s by $ 1.

There are other interesting options for the GPU. For example, Cirrascale offers a monthly server rental with four V100 for about $ 7500 (about ~ $ 10.3 per hour). But for direct comparison, additional tests are required, since this equipment differs from equipment on AWS (type of CPU, memory, support for NVLink, etc.).

Accuracy and convergence

In addition to performance reports, we wanted to verify that the calculations are actually “meaningful,” that is, implementations converge to good results. Since we compared two different implementations, we can expect some deviation. Therefore, our comparison is not only an indicator of the speed of equipment, but also the quality of implementation. For example, implementing a TPU involves very resource-intensive preprocessing steps and actually sacrifices bandwidth. According to Google, this is the expected behavior. As we will see below, it is justified.

We trained the models on the ImageNet dataset , where the task is to classify the image into one of 1000 categories, such as hummingbirds , burritos orthe pizza . The data set consists of 1.3 million images for training (~ 142 GB) and 50,000 images for validation (~ 7 GB).

Learning takes place over 90 eras with a sample size of 1024, after which the results are compared with control data. The TPU implementation sequentially processes about 2796 images per second , and the GPU implementation - about 2839 images per second . This is different from previous bandwidth results, where we turned off augmentation and used synthetic data to compare the net speed of TPUs and GPUs.


The accuracy of the top 1 (i.e., for each image, only the prediction with the greatest confidence is taken into account) of two implementations after 90 epochs

As shown above, the accuracy of the top 1 after 90 epochs for the implementation of TPU by 0.7 percentage points it's better. This may seem insignificant, but achieving improvement at this very high level is extremely difficult. Depending on the application, such small improvements can significantly affect the result.

Let's look at the accuracy of the top 1 in different eras during the training of models.


Top 1 accuracy on a test set for two implementations

Sudden changes in the above graph coincide with changes in learning speed. The convergence trend is better in implementing TPU. Here the final accuracy is reached 76.4% after 86 eras. The implementation of the GPU lags behind and reaches a final accuracy of 75.7% after 84 eras, while to achieve such accuracy on TPU only 64 eras are required. The improvement in TPU convergence is likely due to better data preprocessing and augmentation, but additional experiments are needed to confirm this hypothesis.

Cost-effective cloud pricing solution

Ultimately, the time and cost necessary to achieve a certain accuracy matters. If we take a solution at the level of 75.7% (the best accuracy achieved by implementing the GPU), then we can calculate the cost of achieving this accuracy based on the required eras and the speed of training in images per second. This eliminates the time for evaluating the model between eras and the time it takes to start learning.


Price to achieve top-1 accuracy of 75.7%. * Reserved for 12 months

As shown above, the current Cloud TPU pricing policy allows you to train the model from scratch to 75.7% accuracy on ImageNet in less than 9 hours for $ 55! Learning up to a convergence of 76.4% costs $ 73. Although the V100s run just as fast, a higher price and slower convergence result in a significantly higher solution cost.

Again, note that the comparison depends on the quality of implementation, as well as the price of the cloud.

It would be interesting to compare the difference in energy consumption. But there is currently no publicly available information on TPUv2 energy consumption.

Conclusion

As for the basic performance on the ResNet-50, then four TPUv2 chips (one Cloud TPU module) and four V100 GPUs are equally fast in our tests (the difference is within 2%). Probably due to future software optimizations (for example, TensorFlow or CUDA), the performance will improve and the ratio will change.

However, in practice, most often the main thing is the time and financial costs necessary to achieve a certain accuracy on a specific task. The current Cloud TPU pricing coupled with the excellent ResNet-50 implementation leads to impressive results in time and cost on ImageNet, which makes it possible to train the model to 76.4% accuracy for about $ 73.

For a detailed comparison, benchmarks are needed on models from other areas and with different network architectures. It is also interesting to understand how much effort is required to effectively use each hardware platform. For example, calculations with mixed accuracy are accompanied by a significant increase in performance, but are implemented differently on GPUs and TPUs.