Google Voice has a large-scale interrupt due to TLS certificate expiration

On February 15th, the Google Voice app suffered a four-hour interruption. After the interruption, a large number of users who depend on the application were unable to make/receive calls.

The Google Voice service mainly provides users with a virtual mobile phone number to make and receive calls. It also supports sending and receiving short messages to protect users’ privacy.

Now the company has announced the root cause of the large-scale outage of the voice service. After a detailed investigation, Google determined that the failure was due to the expiration of the encryption certificate.

Google uses trusted certificates issued by Google itself in its services. These certificates have a relatively short validity period based on security considerations and rely on automated tools.

Under normal circumstances, automated tools will automatically complete the renewal work before the certificate expires. As long as the renewal work is completed before the expiration, the normal operation of the service will not be affected.

The Google Voice service uses encrypted connections throughout the entire process based on security considerations. Google forces the use of TLS encrypted connections in the client and therefore will reject any plaintext connections.

Under the operation of automated tools, this mode can run effectively, so it usually does not a malfunction. However, sometimes automated tools may also get an error.

Google stated in the investigation report that the root cause of the interruption was the certificate expiration, and the system did not automatically renew the certificate after the expiration due to some problems.

Although Google has a rotation policy, it cannot solve the problem that the certificate is not renewed. When the certificate expires, all clients refuse to connect to the Google server.

Via: bleepingcomputer