OPI half-bricked after the upgrade

Home Forums User Contributions OPI Support OPI half-bricked after the upgrade

This topic contains 10 replies, has 3 voices, and was last updated by  DarS 2 years, 10 months ago.

  • Author
    Posts
  • #676

    DarS
    Participant

    Hello,
    I am quite busy last months so I did not notice the OPI automatic upgrade up until this Monday, when I realized issues with ownCloud client on my notebook. It turned out that OPI has been half-bricked after the upgrade.
    WHAT WORKS
    I can log in into the device (both as ‘admin’ and as regular ‘user’) and then use ‘Settings’ web interface. Here everything seems to work well. I can log out and log in several times, I can reconfigure various parameters, etc. I even enabled ‘ssh’ service but … could not log in, because the password sits somewhere in email inside OPI (and it’s web mail interface does not work – see next point).
    OPI settings - this works
    WHAT DOES NOT WORK
    Once I log into the device and try to access Calendar, Mail or Files, I see nothing but a rotating circle…
    OPI Mail - does not work
    Moreover, ownCloud clients also display signs of errors:
    OPI ownCloud client - does not work

    Of course OPI worked fine before the upgrade, so I guess that what I see is the result of the upgrade itself and not misconfiguration of the device.

    I just performed simple troubleshooting so far. OPI was rebooted (‘nice’ MS Windows habit…) once, but this did not help. After the initialization OPI begun to behave the same way as described above. I’ve checked that ownCloud clients on MS Windows and Linux display similar errors. I did not check e-mail access via IMAP or POP yet – I don’t have email clients configured for OPI. But I will if it is needed for troubleshooting πŸ™‚

    My question: what should be the best way to troubleshoot further my malfunctioning OPI? I wish I could have at least ‘ssh’ access, things could be much easier then πŸ™‚

    Regards,
    -DarS

  • #677
    Tor Krill
    Tor Krill
    Keymaster

    Hi DarS,

    That was a strange error we have not encountered yet. We have noticed that there where a regression on OC and group management and that there under some circumstances the upgrade of OC could fail.

    This however does not explain why your email isn’t working.

    One possible cause could be that something have stopped MySQL from working as it should.

    First off, can you access email some other way than via Web-mail. F.x. with a desktop client such as Thunderbird, outlook, Apple Mail or something similar. That way you should be able to retrieve the ssh password.

    If this isn’t an option or if this does nor work we could try to use our newly released rescue image. The big limitation with that is that it can only be used to examine and fix the internal storage. Not the external SD-card.

    It can be found here: Rescue image 1.0 follow the instructions on (Re)Installing OPI to create the rescue card.

    Use the card to boot from. I.e. power down OPI and change SD card to the rescue card and power up OPI. Unfortunately this image wont tell you when its booted. It just “hangs” with the middle led blinking. It should however be up and running in a minute or two.

    When that is done you should be able to ssh into OPI using username root and password op237.

    You should then always start by doing a filesystem check on the internal storage. When logged in issue:

    fsck.ext4 -y /dev/mmcblk1p2

    Preferably twice if there where any errors. You should get a message that the filesystem where clean when all is good.

    When that is done you should mount the internal storage to hopefully get any clue on what is wrong. Do:

    mkdir -p /mnt
    mount -text4 /dev/mmcblk1p2 /mnt

    Then the interesting stuff is in the /mnt/var/log directories. Take a look into those and see if they might reveal why things aren’t working.

    And as always, contact us at support@openproducts.com if you need deeper assistance.

    /Tor

    • This reply was modified 2 years, 11 months ago by Tor Krill Tor Krill. Reason: Fixed faulty path
  • #680

    DarS
    Participant

    Tor,
    Thanks for great hints on troubleshooting! I am glad I learned about ‘rescue image’, it is good to know that you can rely on such tooling when in emergency.

    I was wrong saying that Email does not work. You inspired me to configure the desktop email client and … voila! It worked. So I can confirm that both engines: Postfix and Dovecot do work. It was a malfunctioning web front-end which caused the impression that ‘Email did not work’.

    Few comments/hints (perhaps someone might find them useful):
    – I noticed that ‘ssh password’ is being send to OPI ADMIN mail account. So if you created several mail accounts/users AND for daily work you use your regular account and not the admin account (desired practice), then you will not find the ‘ssh password’ in your email. You need to log-off and on as OPI ADMIN instead.
    – whatever your OPI ADMIN user name is (let’s assume JOE_ADM), your ssh login is always ‘root’ and not OPI ADMIN (JOE_ADM in our example).

    Further debugging of broken ownCloud and email web frontend will be a breeze with ‘ssh’ enabled πŸ™‚

    Btw – I cannot see any ownCloud process on ‘ps aux’ list. How can I start it manually?

    Regards,
    -DarS

  • #681

    DarS
    Participant

    Sorry… I completely forgot that ownCloud is written in PHP. Why on Earth I expected any daemon to run? πŸ™‚

    But anyway, I am curious how to troubleshoot further the dead email frontend and OC.

    Regards,
    -DarS

  • #682

    DarS
    Participant

    Hi Tor,
    Detailed examination of log files revealed, that the issue I experienced is related to web frontend.

    MISSING FILES
    ‘/var/log/nginx/error.log’ file indicated clearly what went wrong.
    For user web access:

    "/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory)
    "/var/www/static/core/templates/404.php" failed (2: No such file or directory)
    "/usr/share/opi-webadmin/frontend/skins/opi/images/OPI_RGB.png" failed (2: No such file or directory)

    For ownCloud client access:
    PHP Fatal error: Call to a member function getGID() on a non-object in /usr/share/owncloud/lib/private/group.php on line 193

    And indeed, manual check revealed, that
    – ‘modernizr-2.8.0.min.js’ file is missing, although other files (like ‘jquery.cookie.js’ or ‘apiadmin.js’) are in the ‘/usr/share/opi-webadmin/frontend/js/’ directory,
    – ‘/usr/share/opi-webadmin/frontend/skins’ directory is missing, although other dirs (like ‘/css’ or ‘/js’) are there.

    POTENTIAL CAUSE ?
    ‘OPI Admin Webinterface’ package includes majority (all?) of files placed in ‘/usr/share/opi-webadmin/frontend/’ directory. The version installed on my OPI was ‘opi-webfrontend_1.34_all.deb’. I thought that there was perhaps some failure in the OPI upgrade process and therefore my ‘opi-webfrontend’ was not updated to the latest version.
    However, manual upgrade:

    apt-get update
    apt-get upgrade

    surprised me with the output:
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    The source for ‘apt’ seemed to be correct: ‘repo.op237.se/opi’.

    SUMMARY
    ‘apt-get’ suggests that my OPI runs the latest version. However, certain files are missing and therefore web front end and ownCloud do not work.

    Could I ask for hints how to move things forward?

    Regards,
    -DarS

    • #683
      Tor Krill
      Tor Krill
      Keymaster

      Hi again DarS,

      Sorry for the delayed answer.

      It definetly seems like there is something that went wrong with the upgrade. Which is the cause of the initial problem as well. We have noticed that the upgrade have failed for some customers.

      It seemed like the device driver for the storage system ran out of memory and then terminated the upgrade as well. This unfortunately could leave the system in a somewhat inconsistent state

      The owncloud access problem is however a different problem. It is a bug that where revealed when we upgraded OC. We have a fix for that one but unfortunately we are a bit hesitant to release that one until we fully have resolved the upgrade problem that you and a few other customers have been hit by.

      There is however a temporary workaround for the OC client problem. Remove the users that are affected from any groups. Not pretty but it should solve that problem temporarily until the fix is released.

      Regarding the missing files on the upgrade. I would recommend trying to reinstall opi-frontend.

      Issue:
      apt-get clean
      followed by
      apt-get install --reinstall opi-webfrontend

      Hopefully this should resolve the problems.

      Best Regards,

      /Tor

  • #684

    DarS
    Participant

    Hi Tor,
    I failed again πŸ™‚ I am in a hurry, so cannot dig deeper at this time (in the evening perhaps). But this is the outcome of ‘re-install’

    apt-get install --reinstall opi-webfrontend
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
    Need to get 0 B/863 kB of archives.
    After this operation, 0 B of additional disk space will be used.
    debconf: unable to initialize frontend: Dialog
    debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
    debconf: falling back to frontend: Readline
    dpkg: warning: 'ldconfig' not found in PATH or not executable
    dpkg: warning: 'start-stop-daemon' not found in PATH or not executable
    dpkg: error: 2 expected programs not found in PATH or not executable
    Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
    E: Sub-process /usr/bin/dpkg returned an error code (2)
    

    First thoughts – my OPI clearly is missing something (more screws up during upgrade?). Or the file system was corrupted as well.

    And this is what I saw when I tried to install the ‘missing Dialog’

    apt-get install dialog
    [...]
    Package dialog is not available, but is referred to by another package.
    This may mean that the package is missing, has been obsoleted, or
    is only available from another source
    
    E: Package 'dialog' has no installation candidate
    

    Regards,
    -Darek

    • This reply was modified 2 years, 10 months ago by  DarS.
  • #686

    harrijer
    Participant

    Not sure how helpful this is but:

    I had the same problem and did the following:

    export PATH=/sbin:/usr/sbin:$PATH

    Then ran the:

    apt-get clean
    apt-get install --reinstall opi-webfrontend
    

    Also ran:

    apt-get install --reinstall opi-themes

    However this did not cure my problem of not being able to access the owncloud parts of the web interface.

    As Tor advised, I logged in to the web interface as an admin user and removed all members of a group I had setup to share calendars. This brought back access to owncloud. However, obviously my shared calendars are now no longer working. Luckily, our setup is not complicated and we can access the shared calendar using the credentials of the user that created it. Not ideal, but workable for us.

    The reinstall of opi-webfrontend did not cure messages like:

    2014/11/25 08:28:18 [error] 6124#0: *3804 open() "/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: "GET /admin/js/modernizr-2.8.0.min.js HTTP/1.1", host: "xxxxx.op-i.me", referrer: "https://xxxxx.op-i.me/admin/admin.html"

    Not sure what the siginifcance of this is though as the webinterface seems to be working fine but then I don’t use it a lot.

    /Jeremy

    • This reply was modified 2 years, 10 months ago by  harrijer.
  • #688
    Tor Krill
    Tor Krill
    Keymaster

    Hi DarS and Jeremy,

    I looked a bit deeper into the problem with the missing files mentioned above. It seems like these are actually false indications. I.e. they are referenced but seems not to be needed. (I have the same error-messages here.)

    Hence those can be safely ignored and i have added them to our internal bug-tracker. This also means that reinstalling opi-webfrontend won’t solve any problems as Jeremy also found out.

    Regarding the other problem with the group problem this is a known issue and as i said before i have a fix ready to be released here. Unfortunately we have encountered other problems that got a higher priority. (We have a memory problem leaving some users units bricked after the last upgrade. I will write a separate note on this shortly.)

    But as soon as that fix is released my intentions is to release an upgrade to OC with the group fix.

    DarS, you however said that you also where unable to access web-mail? Is this still an issue? And Jeremy, your roundcube install works?

    Finally, DarS the reason that issueing the commands failed where due to the missing PATHs as Jeremy found out. Just follow the advice and start by issue:

    export PATH=/sbin:/usr/sbin:$PATH

    And the commands should execute nicely.

    /Tor

    • This reply was modified 2 years, 10 months ago by Tor Krill Tor Krill. Reason: Removed duplicate "signing"
  • #693

    harrijer
    Participant

    Just to be sure, I have just done the following:

    Logged in as a user that is not a member of a group and all panels work fine.

    Logged in as admin, I added the same user to a group.

    Logged back in as the user, now the only panel that works is the user admin, all others leave me at the spinning clock.

    Remove the user from the group, all working again.

    That bascially confirms your thoughts on the oc bug.

    /Jeremy

  • #701

    DarS
    Participant

    Yes, I do confirm – my OPI behaves exactly the same (btw, thanks Jeremy for testing this scenario).

    The ‘apt-get install –reinstall opi-webfrontend’ reinstall did nothing spectacular to my OPI (apart from knowledge that the $PATH needs to be taken care of πŸ™‚ ). I still had ‘missing files’ errors in log and spinning circle/clock on user interface.

    But when I followed Jeremy’s advice and removed USER from all GROUPS, I could login and see all emails, calendar, contacts and files.

    WHAT WORKS
    For USER not being in any GROUP, I see that all functions work as expected:
    – USER email, calendar, contact and files are accessible via web interface (although ‘/var/log/nginx’ still logs info about missing files and PHP error)
    – USER ownCloud client syncs flawlessly, no more ‘internal server error’ messages
    – ADMIN email, calendar, contact and files are … accessible via web interface as well. Note what happened – only the USER was removed from all GROUPS. For obvious reasons ADMIN user remained in ‘admin’ group. And despite this the ADMIN web interface begun to work well again!
    – ADMIN email, calendar, contact and files ceases its operation when USER belongs to any GROUP.

    So I can confirm now that I can live with the workaround, no real problem at this time.

    Regards,
    -DarS

You must be logged in to reply to this topic.

Posted in