Digital Ocean - 502 Bad Gateway on one instance

[Con1924]

Hi,
I’m using flowfuse with the digital ocean deployment.
I have 3 instances running. I can access the editors of two of them fine but the third I get a 502 Nginx bad gateway error.

I have tried restarting and stop/starting the instance.

Have also tried a fix from 504 Gateway Timeout when opening editor: { "error": { "code": "unexpected_error", "message": "Error: connect ETIMEDOUT 178.X.X.X:443" } } · Issue #1496 · FlowFuse/flowfuse · GitHub
Which seams similar and to run:
ufw allow http
ufw allow https

On the instance NodeRed logs I have this repeated many times

16/10/2023 09:08:52
[error]
{“name”:“RequestTimedOutError”,“message”:“Request timed out”}

Thanks

[Ben]

The linked issue is very different to what you describe and those commands should already be run as part of the droplet build.

Please run docker ps -a on the droplet and post the output please.

Also please suspend and start the instance and grab the try and grab the whole start up logs

[Con1924]

Thanks,

[Con1924]

[Ben]

I assume the instance in question is one with the UID that starts 8d8be951 (the uuid will be in the URL at the top of the flowforge app when on the instance page) that shows as being up 28mins?

can you try and access the editor again then run docker logs flowforge-nginx-1 and show me any messages mentioning the instance hostname

[Con1924]

@Ben

8d8be951

The output of docker logs flowforge-nginx-1 was constantly outputting too fast to read.

Here is the docker logs via portainer filtered for the UUID, hope thats okay

[Ben]

No, that’s not what I need.

I need you to look for lines that start with the instance hostname. So foo.ff.scnri.io where foo is the name of the instance.

[Con1924]

My bad, is this better?

[Ben]

Please check the Node-RED log again, you may need to scroll back up to see if it crashed and restarted at the point you tried to log into the editor.

It may help to restart in safe mode to prevent the flows running while trying to log in. To do this add the following environment variable under the instance settings:

NODE_RED_ENABLE_SAFE_MODE = true

Once set, suspend, restart then once started try and log in again please.

[Con1924]

Added the Env Var, restarted, now II can access the editor!
Does that mean it’s something in one of my flows? There aren’t that many flows.

Taking some MQTT data, formatting it and sending it to Influxdb

[Ben]

It does imply it is your flow that is causing the problem. You need to fix what ever is flooding the logs with errors then it will be easier to see what might be causing to to crash.

[Con1924]

Okay, so we are sub’d to a wildcard MQTT topic. The sub topics get formatted and sent to an Influxdb 2 DB. The influxdb config had changed which broke the comms from the node to the DB.

Now I’ve fixed the influxdb node everything is okay. I have removed safe mode and everything still works. I guess it was the rate of errors causing the problem.

Is there anyway to mitigate against this? So if it happens again it doesn’t cripple the instance?

Thank you for helping me get to the bottom of it!