• 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Docker issue after few hours
#1
I just got 2 times the same issue, I'll try to describe the context as I haven't any log available.

Docker Image running : nodered/node-red-docker:rpi. No change done into the docker image, just loaded with the volume "nodered_data" mounted to "/data".

When the problem occurs:
- the Nodered flows editor raises the following error "Error: EROFS: read-only file system, open '/data/flows.json'
at Error (native)" when trying to deploy the flow. Previous deploys were working before the error.

- then the docker image stops responding, and cannot be restarted
- Portainer says the image is started, when I hit "Start", but status doesn't change and image doesn't start
- Stop of Docker service works into the Control Panel, but then "Start" of the Services does nothing. The Docker Service turns Green then Yellow, but never start
- Only the Edge Gateway Manager page is working, Portainer cannot connect to Docker

Workaround:
- Restart of the NetPi , Docker starts, Images runs again

I cannot tell at this stage how long the NetPi runs until this issue arrive. It mays be a couple of hours before this occur. Looking to Info Center, the resources looks ok : CPU 20%, Memory 27%, Storage 70%
  Reply
#2
Looks like the issue is related to docker or access to storage. I have the same issue, but with another container image. Error taken from the Portainer logs :
"Error grabbing logs: read /run/.system_part/docker/containers/9f6ae847c387c86190f6d9dc047dd4714aca45e4cec78c79eeb0875746846e04/9f6ae847c387c86190f6d9dc047dd4714aca45e4cec78c79eeb0875746846e04-json.log: input/output error"

At this stage , Portainer cannot do anything to docker.
Not possible to restart Docker from Control Panel.
  Reply
#3
Well Vincent,

I have doubts I can help you much on this point. We do not and will not take responsibility for any Docker Container's function. Docker is just a service that stands between the kernel and a containerized user application that is just restricted in its rights. Basically it runs the same kernel functions as on a native system. We from Hilscher can just take the open sources and compile them in the quality and state as they come from the www at the time of compilation.

Currently we are using Docker in the version 17.06.xx. Docker 17.12.xx is already out. You see that Docker software is still improved day by day. There are known issues we will not and can not follow all. We specifically look on security issues. But what we do is to keep pace and compile next Docker versions automatically when we come out with a new software version.

Here is what I can say. Since many months we have a netPI hardware running test system running 24hrs a day, just to see that netPI hardware is running fine all day long. We do it not with Docker software only, but with other Linux images coming without Docker too. So we have a good software mix. We never see any issue related to the hardware nor like in your case issues with the read/writing accesses to folders or the drive itself.

A word to Portainer.io. Portainer.io is really a simple program. It just uses the socket API of Docker. So if portainer hangs in a way then always Docker is the reason why it is not working or is running delayed or slowly. So your issue is Docker related for sure.

Another thing is, since netPI is 100% equal to a Raspberry Pi 3 I would say that the same effect will happen on a Raspberry too. Have you tried that?

In our "nightly builds" for the next coming software release we have integrated Docker in the version 17.12.x already. It will come out in about 2 weeks.




You never fail until you stop trying.“, Albert Einstein (1879 - 1955)

  Reply
#4
I'm trying to investigate the root cause. For the moment, when I let the NetPi running without touching it, with 3 containers, after a while ( I would say a night for instance), I come back to the web interfaces and get the same behaviors as described above.
What I have also is the Portainer not able to connect anymore on the docker daemon, and when I go to the main admin page of the Edge Gateway Manager, I get the "nginx bad gateway 502" page. So not able to access the page too. I can only power off / on, to get things working back for a while...

Currently having un-modified containers like:
- nodered/node-red-docker:rpi
- homeassistant/raspberrypi3-homeassistant:latest
- hilschernetpi/netpi-netx-ethernet-lan:latest

taken directly from docker hub. They all have their configuration folders mounted as local volumes (e.g. for nodered, I have /data -> /run/.system_part/docker/volumes/nodered_data/_data).

I was suspecting a faulty power supply, and changed it to a 24v 2A.

The node-red is polling via the normal interface the Modbus coupler at a rate of 100ms (loading an IO map of 8x8 bits), and load isn't that high (20% CPU, 37% RAM).

I'm running the last version of the firmware V1.1.0.0. Waiting for the next version to see if this problem persist.
Perhaps if we could get a log from the host layer, it will be easier to understand. I assume by default the OS layer / Docker doesn't store any log on the internal flash...?

I will follow your advice and install the same images on a fresh Raspi installation. However, I won't be able to reproduce the same host layer. So I will do the following: last raspian light version, last docker & portainer.
I keep you informed about the tests.

hope my feedback help,
/incent
  Reply
#5
Well a fresh system will not be needed on your RPi3. Docker encapsulates all calls from a container into the host kernel. So it is just a matter of the kernel. We use kernel 4.9.x.

Using Raspbian (jessie and stretch both work) as basis is like I would do it. Also our video on youtube does the same.

What I wouln't do is to install portainer.io on the RPi3 . Portainer.io is just a GUI of the Docker CLI functions by using the /var/run/docker.sock interface. You can instruct docker in a normal shell directly with the same effect. So just use commands like "docker pull ... " in your console and it will be good.

I haven't analyzed Docker log files yet and don't know the quality of them. So I can't contribute much on that topic.

When you click on a container that is not running, you can watch it's console output in portainer.io. Have you every tried that? So click on the container and then "log". There you see all container outputs you would see as if you call docker commands on a normal shell on a RPi3. Maybe you find valuable informations there after you have restarted the service why the container does not start again.
You never fail until you stop trying.“, Albert Einstein (1879 - 1955)

  Reply
#6
Hello Armin,

thanks for your answers. I did a last test before deciding, but I think for my usage it won't continue with the NetPi.
Here is the reason: I'm getting issues with docker & portainer after few days running.

- I did a fresh install of the firmware , then loaded two standard containers from docker/hub
- One is nodered/node-red-docker:rpi
- Other one is homeassistant/raspberrypi3-homeassistant:latest

I did the same on both NetPi and my RaspberryPi3. Then I loaded the same Node-red flow , which does a simple task: Modbus read 64 inputs, then check Input 1 if On, then cycling 8 DO on/off with a frequency of 100ms. Due to the container, they use the standard LAN interface, so the NetX isn't used here.

I let them run during 72 hours both, then just turned off all containers, and let them running during a week without turning any container ( in fact, I was just doing a freeze between weekends). So both were just running their OS / Docker with all containers turned off.


Today, I returned back to the portainer to turn on the containers:
- On RaspPI, the containers restarted and responded as expected
- On NetPi, no container was able to start. (I checked it wasn't due to Web Interface timeout, so I went to the main URL, entered the portainer and login).

I had a 3rd container into the NetPi (the hilschernetpi/netpi-netx-ethernet-lan:latest ) , so I restarted that one to be sure that my problem wasn't due to the NodeRed or Home Assistant images, but that one didn't start too.

So I just did a power reset of the NetPi, and then all 3 containers turned On and running back again.

I cannot see what's wrong, but it is not working as my reference RaspPi, and makes me feel the usage I want doesn't fit with the present implementation of your product. I really like the concept, but my return window on Amazon is getting close, so I prefer to return it back and focus on my project instead of trying to figure out what doesn't work on the NetPi itself, or my containers on it.

thanks again,
Vincent
  Reply
#7
Oh I am very sad to hear that you want to return the device.

If you say you "turned off" the containers then you mean you use the "stop container" function in portainer.io? Right? Just to understand what we can maybe set up here as a test system to get similar effect visible too. And also if you say you "turned on" the containers you used the "start container" function?

Armin

You never fail until you stop trying.“, Albert Einstein (1879 - 1955)

  Reply
#8
Yes indeed, I meant using the "Start" and "Stop" , not "Pause/Resume".

The reason I'm stoping now is also due to the return policy of amazon. if I keep it few days more, I won't be refund if it doesn't work as I want. I don't have a big budget for my PLC refresh.

While thinking about my usage , here are some points that I couldn't understand by myself:
- when Portainer stops working (e.g. I start a container, it says "Container Started" , but it doesn't): I have no log from docker to check or knows if Docker is still running correctly , no docker cli command

- when the internal flash is full, I see it into the Control Panel , but it is difficult to check why it goes full, we cannot check the volumes size, only the images size are visible ...

- external usb storage : already mentioned

- footprint enclosure (that's minor issue): turn out that with a DIN standard enclosure like Hager Vega D, I cannot close the door with the NetPi mounted (lol). Anyway, I had the same issue with my Wago Power Supplies, so I have no door since a while.

- Soldered SD : well, fully agree for industrial ... psychological concern perhaps for home usage ... RaspPi is know to brick SD cards, so if/when this happen , not possible to swap with a backup cloned SD and restart quickly. My existing PLC has a Compact Flash card mounted read-only, if things goes wrong, I just swap it.

About positive aspects:
- Support and Reactiveness : Great! Really appreciate the answers and explanation


- DIN rail : so nobody touch it when live, embedded into the electric enclose with the IO
- Out of the box docker host with pre-patched kernel : Great! , that what I was looking for. Little bit too sharp for the moment, quite normal for a new product, but keep going, this is the way to go.

- Industrial interfaces: I was expecting to have modbus into the firmware, or have these interfaces managed through docker (Ethernet firmware) so the containers doesn't have to load any particular driver (perhaps lazy?) and use standard docker hub image, map it as single interface / bridge or dual-interface.

- 24volt power supply, that's good so shared with standard IO / PLC ...
  Reply
#9
Dear Vincent,

what do you mean with "the internal flash is full"? Do you mean the 8GB byte SD card memory respectively around 6GB you can use effectively for your own tasks? And you say it is getting written and written and written so that is it getting "full"? But this can only be initiated by a container, cause the host software doesn't do anything with the SD card after booted.

About "container started". Have you ever clicked with you mouse onto a "started" container and cliked to "logs" afterwards to see the containers "STDOUT outputs" as a log file? There you can see what a containers outputs as if you are in a console and use the CLI commands line.

Which version of Docker are you using on Rpi3 by the way? We will come with the latest version 17.12. end of the comint week.

Do you mean Modbus RTU or Modbus TCP? If you load a Node-RED container from us and use then in Node-RED the "manage palette" function you can install a Modbus TCP node from the www and use it right away with netPI. Or you install CODESYS container and get a modbus TCP master for free as well. Since there are so many software stacks out in the world we do not have a Modbus TCP offer for the Industrial network controller netX.


You never fail until you stop trying.“, Albert Einstein (1879 - 1955)

  Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  [INFO] RESTful API for Docker remote control Armin@netPI 0 87 December-17th-2018, 10:15 AM
Last Post: Armin@netPI
  Accessing Docker daemon through remote CLI adeeljsid 2 161 August-30th-2018, 12:32 PM
Last Post: Armin@netPI
  Edge Gateway Manager - Docker Management HagnSchool 1 89 June-8th-2018, 07:50 AM
Last Post: Armin@netPI
  DHCP/DNS issue: Couldn't access NetPI3 out-of-the-box zulliger@software-natives.ch 5 181 March-14th-2018, 06:47 PM
Last Post: adeeljsid
  Portainer as a docker image? vdupont 1 137 February-17th-2018, 09:55 AM
Last Post: Armin@netPI
  Are USB interfaces exposed by docker to the containers? vdupont 1 88 February-17th-2018, 08:48 AM
Last Post: Armin@netPI
  Kann keine Docker Images laden rjenni 1 94 January-25th-2018, 09:23 AM
Last Post: Armin@netPI
  Adressing netPI with IE11 over its hostname let Docker tile fail to open Armin@netPI 0 67 January-11th-2018, 07:33 AM
Last Post: Armin@netPI

Forum Jump:


Users browsing this thread: 1 Guest(s)