Datastream error [ bin-log file missing ]

[RC1]

Hi folks I am getting following error on GCP data stream. My datastream streams data from [Mysql of cloud sql ] to GCS bucket

The stream is unable to continue as the file it is trying to read ('mysql-bin.000115') no longer exists on the server. This can happen due to a long disconnection from the server, a purge action on the binary logs, a data source error that lasts for a long time, or insufficient binary log retention policy. Unfortunately the only way to fix this is to recreate the stream.

I know that this issue is on database level. But how to debug this issue and find the root cause analysis? How to know exactly which thing cause this issue from these (due to a long disconnection from the server, a purge action on the binary logs, a data source error that lasts for a long time, or insufficient binary log retention policy) ?

cc: @etaim 

[josegutierrez]

The reason for the error is "Datastream's reader is unable to read the changes from the source log files, because those files aren't available anymore. Log files are usually kept on the server for a defined retention period, and if Datastream was unable to read said file by the time that retention period expires - we won't be able to continue and read the CDC changes for that stream.

So it seems you have to set up a retention period long enough (recommendation is 7 days) as per public documentation. 

Having said that as you have mentioned that you cannot trust in increasing the retention period because your version maximum is 7 days and it could fit in only one binlog file depending on the traffic.

[RC1]

@josegutierrez Our retention period for logs is 7 days. What could be other reasons that could cause such issues ?

[josegutierrez · Accepted solution]

Hi @RC1  I did some research and I didn’t found anything else, my guess should be to increase the retention or to make a support case, so an agent can help you with more specific details.