Tagged: Issue with upgrade
November 20, 2014 at 05:52 #676
I am quite busy last months so I did not notice the OPI automatic upgrade up until this Monday, when I realized issues with ownCloud client on my notebook. It turned out that OPI has been half-bricked after the upgrade.
I can log in into the device (both as ‘admin’ and as regular ‘user’) and then use ‘Settings’ web interface. Here everything seems to work well. I can log out and log in several times, I can reconfigure various parameters, etc. I even enabled ‘ssh’ service but … could not log in, because the password sits somewhere in email inside OPI (and it’s web mail interface does not work – see next point).
WHAT DOES NOT WORK
Once I log into the device and try to access Calendar, Mail or Files, I see nothing but a rotating circle…
Moreover, ownCloud clients also display signs of errors:
Of course OPI worked fine before the upgrade, so I guess that what I see is the result of the upgrade itself and not misconfiguration of the device.
I just performed simple troubleshooting so far. OPI was rebooted (‘nice’ MS Windows habit…) once, but this did not help. After the initialization OPI begun to behave the same way as described above. I’ve checked that ownCloud clients on MS Windows and Linux display similar errors. I did not check e-mail access via IMAP or POP yet – I don’t have email clients configured for OPI. But I will if it is needed for troubleshooting 🙂
My question: what should be the best way to troubleshoot further my malfunctioning OPI? I wish I could have at least ‘ssh’ access, things could be much easier then 🙂
November 20, 2014 at 10:32 #677
That was a strange error we have not encountered yet. We have noticed that there where a regression on OC and group management and that there under some circumstances the upgrade of OC could fail.
This however does not explain why your email isn’t working.
One possible cause could be that something have stopped MySQL from working as it should.
First off, can you access email some other way than via Web-mail. F.x. with a desktop client such as Thunderbird, outlook, Apple Mail or something similar. That way you should be able to retrieve the ssh password.
If this isn’t an option or if this does nor work we could try to use our newly released rescue image. The big limitation with that is that it can only be used to examine and fix the internal storage. Not the external SD-card.
Use the card to boot from. I.e. power down OPI and change SD card to the rescue card and power up OPI. Unfortunately this image wont tell you when its booted. It just “hangs” with the middle led blinking. It should however be up and running in a minute or two.
When that is done you should be able to ssh into OPI using username root and password op237.
You should then always start by doing a filesystem check on the internal storage. When logged in issue:
fsck.ext4 -y /dev/mmcblk1p2
Preferably twice if there where any errors. You should get a message that the filesystem where clean when all is good.
When that is done you should mount the internal storage to hopefully get any clue on what is wrong. Do:
mkdir -p /mnt mount -text4 /dev/mmcblk1p2 /mnt
Then the interesting stuff is in the /mnt/var/log directories. Take a look into those and see if they might reveal why things aren’t working.
And as always, contact us at email@example.com if you need deeper assistance.
- This reply was modified 2 years, 11 months ago by Tor Krill. Reason: Fixed faulty path
November 21, 2014 at 06:28 #680
Thanks for great hints on troubleshooting! I am glad I learned about ‘rescue image’, it is good to know that you can rely on such tooling when in emergency.
I was wrong saying that Email does not work. You inspired me to configure the desktop email client and … voila! It worked. So I can confirm that both engines: Postfix and Dovecot do work. It was a malfunctioning web front-end which caused the impression that ‘Email did not work’.
Few comments/hints (perhaps someone might find them useful):
– I noticed that ‘ssh password’ is being send to OPI ADMIN mail account. So if you created several mail accounts/users AND for daily work you use your regular account and not the admin account (desired practice), then you will not find the ‘ssh password’ in your email. You need to log-off and on as OPI ADMIN instead.
– whatever your OPI ADMIN user name is (let’s assume JOE_ADM), your ssh login is always ‘root’ and not OPI ADMIN (JOE_ADM in our example).
Further debugging of broken ownCloud and email web frontend will be a breeze with ‘ssh’ enabled 🙂
Btw – I cannot see any ownCloud process on ‘ps aux’ list. How can I start it manually?
November 21, 2014 at 07:25 #681
Sorry… I completely forgot that ownCloud is written in PHP. Why on Earth I expected any daemon to run? 🙂
But anyway, I am curious how to troubleshoot further the dead email frontend and OC.
November 22, 2014 at 09:14 #682
Detailed examination of log files revealed, that the issue I experienced is related to web frontend.
‘/var/log/nginx/error.log’ file indicated clearly what went wrong.
For user web access:
"/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory) "/var/www/static/core/templates/404.php" failed (2: No such file or directory) "/usr/share/opi-webadmin/frontend/skins/opi/images/OPI_RGB.png" failed (2: No such file or directory)
For ownCloud client access:
PHP Fatal error: Call to a member function getGID() on a non-object in /usr/share/owncloud/lib/private/group.php on line 193
And indeed, manual check revealed, that
– ‘modernizr-2.8.0.min.js’ file is missing, although other files (like ‘jquery.cookie.js’ or ‘apiadmin.js’) are in the ‘/usr/share/opi-webadmin/frontend/js/’ directory,
– ‘/usr/share/opi-webadmin/frontend/skins’ directory is missing, although other dirs (like ‘/css’ or ‘/js’) are there.
POTENTIAL CAUSE ?
‘OPI Admin Webinterface’ package includes majority (all?) of files placed in ‘/usr/share/opi-webadmin/frontend/’ directory. The version installed on my OPI was ‘opi-webfrontend_1.34_all.deb’. I thought that there was perhaps some failure in the OPI upgrade process and therefore my ‘opi-webfrontend’ was not updated to the latest version.
However, manual upgrade:
apt-get update apt-get upgrade
surprised me with the output:
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The source for ‘apt’ seemed to be correct: ‘repo.op237.se/opi’.
‘apt-get’ suggests that my OPI runs the latest version. However, certain files are missing and therefore web front end and ownCloud do not work.
Could I ask for hints how to move things forward?
November 24, 2014 at 08:49 #683
Hi again DarS,
Sorry for the delayed answer.
It definetly seems like there is something that went wrong with the upgrade. Which is the cause of the initial problem as well. We have noticed that the upgrade have failed for some customers.
It seemed like the device driver for the storage system ran out of memory and then terminated the upgrade as well. This unfortunately could leave the system in a somewhat inconsistent state
The owncloud access problem is however a different problem. It is a bug that where revealed when we upgraded OC. We have a fix for that one but unfortunately we are a bit hesitant to release that one until we fully have resolved the upgrade problem that you and a few other customers have been hit by.
There is however a temporary workaround for the OC client problem. Remove the users that are affected from any groups. Not pretty but it should solve that problem temporarily until the fix is released.
Regarding the missing files on the upgrade. I would recommend trying to reinstall opi-frontend.
apt-get install --reinstall opi-webfrontend
Hopefully this should resolve the problems.
November 25, 2014 at 07:14 #684
I failed again 🙂 I am in a hurry, so cannot dig deeper at this time (in the evening perhaps). But this is the outcome of ‘re-install’
apt-get install --reinstall opi-webfrontend Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded. Need to get 0 B/863 kB of archives. After this operation, 0 B of additional disk space will be used. debconf: unable to initialize frontend: Dialog debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.) debconf: falling back to frontend: Readline dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: warning: 'start-stop-daemon' not found in PATH or not executable dpkg: error: 2 expected programs not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin E: Sub-process /usr/bin/dpkg returned an error code (2)
First thoughts – my OPI clearly is missing something (more screws up during upgrade?). Or the file system was corrupted as well.
And this is what I saw when I tried to install the ‘missing Dialog’
apt-get install dialog [...] Package dialog is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source E: Package 'dialog' has no installation candidate
- This reply was modified 2 years, 10 months ago by DarS.
November 25, 2014 at 09:49 #686
Not sure how helpful this is but:
I had the same problem and did the following:
Then ran the:
apt-get clean apt-get install --reinstall opi-webfrontend
apt-get install --reinstall opi-themes
However this did not cure my problem of not being able to access the owncloud parts of the web interface.
As Tor advised, I logged in to the web interface as an admin user and removed all members of a group I had setup to share calendars. This brought back access to owncloud. However, obviously my shared calendars are now no longer working. Luckily, our setup is not complicated and we can access the shared calendar using the credentials of the user that created it. Not ideal, but workable for us.
The reinstall of opi-webfrontend did not cure messages like:
2014/11/25 08:28:18 [error] 6124#0: *3804 open() "/usr/share/opi-webadmin/frontend/js/modernizr-2.8.0.min.js" failed (2: No such file or directory), client: xxx.xxx.xxx.xxx, server: localhost, request: "GET /admin/js/modernizr-2.8.0.min.js HTTP/1.1", host: "xxxxx.op-i.me", referrer: "https://xxxxx.op-i.me/admin/admin.html"
Not sure what the siginifcance of this is though as the webinterface seems to be working fine but then I don’t use it a lot.
- This reply was modified 2 years, 10 months ago by harrijer.
November 25, 2014 at 15:10 #688
Hi DarS and Jeremy,
I looked a bit deeper into the problem with the missing files mentioned above. It seems like these are actually false indications. I.e. they are referenced but seems not to be needed. (I have the same error-messages here.)
Hence those can be safely ignored and i have added them to our internal bug-tracker. This also means that reinstalling opi-webfrontend won’t solve any problems as Jeremy also found out.
Regarding the other problem with the group problem this is a known issue and as i said before i have a fix ready to be released here. Unfortunately we have encountered other problems that got a higher priority. (We have a memory problem leaving some users units bricked after the last upgrade. I will write a separate note on this shortly.)
But as soon as that fix is released my intentions is to release an upgrade to OC with the group fix.
DarS, you however said that you also where unable to access web-mail? Is this still an issue? And Jeremy, your roundcube install works?
Finally, DarS the reason that issueing the commands failed where due to the missing PATHs as Jeremy found out. Just follow the advice and start by issue:
And the commands should execute nicely.
- This reply was modified 2 years, 10 months ago by Tor Krill. Reason: Removed duplicate "signing"
November 25, 2014 at 18:33 #693
Just to be sure, I have just done the following:
Logged in as a user that is not a member of a group and all panels work fine.
Logged in as admin, I added the same user to a group.
Logged back in as the user, now the only panel that works is the user admin, all others leave me at the spinning clock.
Remove the user from the group, all working again.
That bascially confirms your thoughts on the oc bug.
November 26, 2014 at 05:43 #701
Yes, I do confirm – my OPI behaves exactly the same (btw, thanks Jeremy for testing this scenario).
The ‘apt-get install –reinstall opi-webfrontend’ reinstall did nothing spectacular to my OPI (apart from knowledge that the $PATH needs to be taken care of 🙂 ). I still had ‘missing files’ errors in log and spinning circle/clock on user interface.
But when I followed Jeremy’s advice and removed USER from all GROUPS, I could login and see all emails, calendar, contacts and files.
For USER not being in any GROUP, I see that all functions work as expected:
– USER email, calendar, contact and files are accessible via web interface (although ‘/var/log/nginx’ still logs info about missing files and PHP error)
– USER ownCloud client syncs flawlessly, no more ‘internal server error’ messages
– ADMIN email, calendar, contact and files are … accessible via web interface as well. Note what happened – only the USER was removed from all GROUPS. For obvious reasons ADMIN user remained in ‘admin’ group. And despite this the ADMIN web interface begun to work well again!
– ADMIN email, calendar, contact and files ceases its operation when USER belongs to any GROUP.
So I can confirm now that I can live with the workaround, no real problem at this time.
You must be logged in to reply to this topic.