Crashed Instance and dissapeared production flows due to "version mismatch"

Support
1

Hi guys. We are having serious issues while deploying a flow in an industrial production environment. There were 2 people working simultaneously, the app asked to merge changes and after that it crashed and all flows dissapeared. We are in a hurry since we are deploying in a factory local environment since the internet connection is pretty bad and we were unable to get cloud services working here.

Im attaching a screenshot with the error it is showing in the flow forge. We are deploying in a linunx VM connected to a local server inside a Lan network.

Any help is truly appreciable since all the flows were removed and it seems the default file was inserted instead. We will be attentive to any support you suggest. We are looking for the local flows.json file, but we are unable to find it.

Captura de pantalla 2023-07-05 102752
Captura de pantalla 2023-07-05 102752782×593 31.5 KB

2

Hello

Thinking firstly how to get your flow running again, do you have a snapshot you can revert to?

1 Like
3

Yes Rob, last snapshot was made yesterday. Firstly we are running the snapshot since the factory is starting opperations and we were making the last adjustments. We are trying to get inside the local files and get the “flows.json” file, however we are unable to find it, could you please kindly let us know how to get access to that file?

4

Some other thoughts / questions.

Are you running in Docker,K8s or as a local install?

What was being changed at the moment the crash occurred? I’ve have not personally seen an instance fail due to merging in edits to flows.

Have you checked the Node-RED logs rather than the audit logs? Go to the instance then Node-RED Logs.

1 Like
5

@rob we are trying to run the snapshot and after confirm rollback nothing happens!!! Please help!!!

6

Thanks! We are running in a local install inside a Linux Virtual Machine. Not in a Docker or K8s.

We were only charging a debug node at the moment the crash occurred.

After looking at the Node-RED logs it is just showing info after the crash, there is nothing for what happened before.

7

Here is what the NODE-RED logs is showing. Hw its only showing info after the crash.

7/5/2023 10:46:13 AM
[system]
Launcher Started
7/5/2023 10:46:13 AM
[system]
Loading project settings
7/5/2023 10:46:13 AM
[system]
Target state is 'running'
7/5/2023 10:46:13 AM
[system]
Starting Node-RED
7/5/2023 10:46:14 AM
[info]
Welcome to Node-RED
===================
7/5/2023 10:46:14 AM
[info]
Node-RED version: v3.0.2
7/5/2023 10:46:14 AM
[info]
Node.js  version: v16.16.0
7/5/2023 10:46:14 AM
[info]
Linux 5.15.0-71-generic x64 LE
7/5/2023 10:46:14 AM
[info]
Loading palette nodes
7/5/2023 10:46:14 AM
[info]
FlowForge HTTP Authentication Plugin loaded
7/5/2023 10:46:14 AM
[info]
FlowForge Team Library Plugin loaded
7/5/2023 10:46:14 AM
[]
FlowForge Light Theme Plugin loaded
7/5/2023 10:46:14 AM
[]
FlowForge Dark Theme Plugin loaded
7/5/2023 10:46:15 AM
[warn]
[RequestValidator] Registered invalid property name 'sampleNodeColour'. Properties for this node must start with 'requestValidator'
7/5/2023 10:46:15 AM
[info]
Dashboard version 3.4.0 started at /ui
7/5/2023 10:46:15 AM
[warn]
------------------------------------------------------
7/5/2023 10:46:15 AM
[warn]
[@flowforge/nr-file-nodes/file] 'file in' already registered by module node-red
7/5/2023 10:46:15 AM
[warn]
------------------------------------------------------
7/5/2023 10:46:15 AM
[info]
Settings file  : /opt/flowforge/var/projects/37d34705-5ef0-4304-9e8e-dd67b76c6b57/settings.js
7/5/2023 10:46:15 AM
[info]
Context store  : 'default' [module=memory]
7/5/2023 10:46:15 AM
[info]
Server now running at http://127.0.0.1:12080/
7/5/2023 10:46:15 AM
[info]
Starting flows
7/5/2023 10:46:15 AM
[info]
Started flows
7/5/2023 10:53:32 AM
[system]
Stopping Node-RED
7/5/2023 10:53:32 AM
[info]
Stopping flows
7/5/2023 10:53:32 AM
[info]
Stopped flows
7/5/2023 10:53:32 AM
[system]
Node-RED exited rc=0 signal=null
7/5/2023 10:53:34 AM
[system]
Loading project settings
7/5/2023 10:53:34 AM
[system]
Target state is 'running'
7/5/2023 10:53:34 AM
[system]
Starting Node-RED
7/5/2023 10:53:34 AM
[info]
Welcome to Node-RED
===================
7/5/2023 10:53:34 AM
[info]
Node-RED version: v3.0.2
7/5/2023 10:53:34 AM
[info]
Node.js  version: v16.16.0
7/5/2023 10:53:34 AM
[info]
Linux 5.15.0-71-generic x64 LE
7/5/2023 10:53:34 AM
[info]
Loading palette nodes
7/5/2023 10:53:34 AM
[info]
FlowForge HTTP Authentication Plugin loaded
7/5/2023 10:53:34 AM
[info]
FlowForge Team Library Plugin loaded
7/5/2023 10:53:34 AM
[]
FlowForge Light Theme Plugin loaded
7/5/2023 10:53:34 AM
[]
FlowForge Dark Theme Plugin loaded
7/5/2023 10:53:35 AM
[warn]
[RequestValidator] Registered invalid property name 'sampleNodeColour'. Properties for this node must start with 'requestValidator'
7/5/2023 10:53:35 AM
[info]
Dashboard version 3.4.0 started at /ui
7/5/2023 10:53:35 AM
[warn]
------------------------------------------------------
7/5/2023 10:53:35 AM
[warn]
[@flowforge/nr-file-nodes/file] 'file in' already registered by module node-red
7/5/2023 10:53:35 AM
[warn]
------------------------------------------------------
7/5/2023 10:53:35 AM
[info]
Settings file  : /opt/flowforge/var/projects/37d34705-5ef0-4304-9e8e-dd67b76c6b57/settings.js
7/5/2023 10:53:35 AM
[info]
Context store  : 'default' [module=memory]
7/5/2023 10:53:35 AM
[info]
Server now running at http://127.0.0.1:12080/
7/5/2023 10:53:35 AM
[info]
Starting flows
7/5/2023 10:53:35 AM
[info]
Started flows
8

Any help in this case??? Someone experienced a similar situation?

9

I’ve asked if anyone else here has any suggestions, I’m sure they will post if they have any ideas.

What OS are you running on?

1 Like
10

Ubuntu Server in a Virtual Machine.

Could you please let us know how to extract the current flow inside the Flowforge folder and try to run it in other instance? Maybe in a single Node-RED instance outside flowforge?

11

Hello, the path is in the logs you sent, it’s ‘/opt/flowforge/var/projects/’ can you see your flow in there?

Also, can confirm that you are not using our device agent to run the Node-RED instance.

12

@jvalencia you have tagged device-agent so it is a little unclear where your issue lies.

Additionally, the snapshot you have - is it set as the target for devices (assuming devices) - or, have you rolled back to the “ok” snapshot?

13

Thanks. In that path there is no any flow inside. In addition, we are also trying to run a snapshot made yesterday, it completes the rollback, however it is still showing the default blank file.

14

Ok I get it, I already removed the labels if that helps to avoid confusion.

The snapshot we had wasn´t set for the target devices. We tried to roll back to the “ok” snampshot and nothing happens. There is a serious bug inside this FlowForge version, maybe Node-RED version, its unclear for us.

15

Maybe we are not using the full FlowForge stack, it is on us. We are just trying to get the flows.json file and make it working in another instance.

16

Bugs do occur but I am not certain this is a bug (since i have used snapshots a lot)

I appreciate that, but while there is a lot of information in your posts, there is not enough details of your setup to understand what might have happened and where your flows may be.

For example, did you export flows at all? Do you have more than one snapshot? Have you tried rolling back other snapshots?

When you say in a linux VM, is that where FlowForge is running?

Assuming you followed the linux install documentation, did you setup postgres (or leave DB as default). If default, then the database for your flowforge will be a SQLITE instance in /opt/flowforge named forge.db. If you can access this file, all snapshots etc will be contained in that.

Can you access files inside the VM?

I am happy to receive a copy and look for you.

1 Like
17

For awareness, this issue has been fixed.

@jvalencia could you please update this thread with information. Did you resolve this via the FlowForge frontend or did you dig into the database?

Additionally, please provide us with details of your FlowForge install version, NodeJS version and the Node-RED version in your stack. We are keen to understand if this is a bug we need to fix.

18

Indeed, the main issue hasnt been fixed!

In this moment we were able to get the last snapshot working, but we are now trying to get the last changes made from that snapshot until today. We fixed several bugs and improved some functionalities.

Do you think is there any way we can try to get those changes working again? It was very weird that after merging changes in 2 users working simoultaneously without performing several changes or flow’s conflicts, (just adding one debug node and disconnecting another) the application went in a blank file.

We are very happy to receive your help, and it would be very awesome, if you could let us know how to try to retrieve the last changes made after that snapshot?

We also lost the Node-RED logs before the problem, we are only looking the logs after the crash, to be honest, if you guys could help us to get those logs or changes, or anything that could give us a way to recover the last changes… we would be in an enormous debt with you looking to get a pro version.

19

Thanks to your support we were able to get the last snapshot from DB, however the main issue is still unresolved. let me place here that versions info that was inside the log

Node-RED version: v3.0.2
Node.js version: v16.16.0
Linux 5.15.0-71-generic x64 LE
FlowForge V1.6.0

1 Like
20

It doesnt matter if its from your end or our mistake, why is there a merge button option while deploying changes if its not capable to detect conflicting changes and crashes the entire application? Is that solved in a paid version?