Recommended Posts

So I woke up late today, around 1300, to find that my Nextcloud instance was down.  I'm hosting it on Debian Bullseye via the regular old tarball manually set up with Apache, MariaDB/MySQL, PHP, etc.  It's been running great for literally years across multiple in-place upgrades to both Nextcloud and Debian.

 

After doing some tinkering it came to my attention that I MySQL was complaining it couldn't connect to the database.  Easy enough I figured, I'll just log into MySQL and see what's wrong.  Upon trying to launch the MySQL shell though it would ask for the password and then error out saying it couldn't connect to the server.

"ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (111)"

So I thought maybe the .sock file got messed with during an update or something and wasn't being removed properly, so I verified the location of the correct file by looking at the configs, all of which pointed to the same file, and I then deleted that mysqld.sock file and tried restarting MySQL, but still no dice.  I tried rebooting the whole server just for kicks, no luck.

 

I tried reinstalling MariaDB/MySQL but that apparently doesn't get rid of the existing configuration files, so what I ended up doing was apt purge --autoremove on mariadb-server, deleting /var/run/mysql, then reinstalling it and re-importing my most recent database backup (yesterday).  It's just a personal instance with myself, my wife and kids on it, and I've got it scheduled to do daily backups of the database, so it wasn't a huge issue.  What I'm curious about is why it crapped out in the first place.

 

While poking around in syslog I found the following line:

mariadbd[1115]: 2022-01-01 11:55:02 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption.


That timestamp is hours after any kind of automatic update/reboot would have taken place.

 

So something crazy happened that corrupted the actual database, but why would that have broken my ability to log into the MySQL shell to try and correct it?  It's saying I should dump and reimport the database, but I couldn't do that without having access to the MySQL shell.

 

I've checked the logs for apt and I don't see any kind of updates that would have been applied by unattended-upgrades; my last automatic update was December 18th.

 

Did anybody else have anything happen today with their database?  I guess it's definitely possible that Nextcloud encountered some kind of bug and corrupted its database.  I've done short SMART tests on all the drives in the system and found no issues, and the server is running on an UPS so there shouldn't have been any kind of power fluctuation or outage to cause any issues.  My UPS is reporting no events since the 17th either.

 

I guess I'm posting all this just to try and fish for thoughts from any of you who may have encountered this kind of thing in the past, or who may have some idea as to what happened.  I've restored a backup and everything is fine, but if there's something I can do to prevent the issue in the future, I'd like to do so.

Link to comment
https://www.neowin.net/forum/topic/1414223-mariadbmysql-took-a-dump-last-night/
Share on other sites

Well, I have a bunch of MySQL / MariaDB 5.x and MariaDB 10.x instances which are all running without issue right now.

 

I've had things like that happen before though. One cause is if the filesystem temporarily ran out of diskspace which can cause a table to require fixing. I've a suspicion that MySQL doesn't behave well if the data filesystem is briefly marked as read-only but it's just a hunch.

 

Table corruption can stop MySQL from starting though. That's a thing unfortunately.

 

Personally I'd recommend enabling the binlog and adding "--master-data=2" to your mysqldump line so that you can recover the database right up to the point where corruption occured. If you backup both the database dump file and the associated binlog files then you're pretty well sorted in terms of data recovery I think.

I just checked my install of MySQL running on Raspbian and all is well. With having to do a complete wipe and restore, the last entries are more than likely gone to see what the last thing that was modified or added/removed. The last time I had any corruption on my setup was testing new additions and was completely my own doing. Have you checked any connection logging to see if any weird connections were seen around the after the last time you knew it was working?

 

@DonC has a great point as that missing data between the last backup could be vital to see what happened.

The log entry immediately prior to the error messages is Nextcloud invoking its cron.php script, so I'm guessing it has something to do with that.  I've made copies of syslog from that timeframe so I may dig into it some more later, but I'm tired of reading logs since everything is back up and working I'll save it for later.

Good luck and I am happy that at least everything is back up and going. Keep us posted if you do dig into this. I am interested to see what you find if you do

On 01/01/2022 at 21:05, Gerowen said:

 

Did anybody else have anything happen today with their database? 

 

chinese hackers

On 01/01/2022 at 22:44, Marujan said:

chinese hackers

I thought about hackers of some sort, but there was no indications of any files missing or modified, no suspicious entries in auth.log, nothing banned by Fail2Ban, etc.  On top of that, all the various services hosted by the server are all hosted by their own non-root user accounts/groups and SSH is not open to the world and enforces public key authentication.  I'm fairly certain it was just some weird-ness with the database during the execution of Nextcloud's cron script.

 

Besides, with only 4 users, outside of some script kiddie who happened across a public share link I've posted somewhere, there's not really any incentive to try and bother my personal server.

Edited by Gerowen

So here's a piece of syslog.  You can see that at 11:50 the cron.php script executes and there are no problems.  5 minutes later it runs again (this is scheduled/expected), and this is where the problems begin.  So in the block of time between 11:50 and 11:55, something screwy happened.  I was asleep at the time, so I personally wasn't doing anything on the server directly, but we've all got cell phones and PCs connected to it all the time, plus I've shared several public links for photo albums and such with family members over Facebook, so even if I wasn't logged in, Nextcloud is constantly doing "something" in the background.

image.thumb.png.b4d6e17c3252dbf55a75302c9e5a5541.png

 

Here's the contents of auth.log for that particular block of time.  Nothing suspicious, root running cron and www-data running Nextcloud's cron.php script.

image.thumb.png.f22bf5f0c8231a95da077253eca4d1af.png

 

The database and the Nextcloud server files are stored on the main system drive which is a Western Digital Blue 2.5" SSD.  The actual data directory (user files) is stored however on a separate, encrypted RAID 5 "storage" partition.  Both drives have plenty of free space available.

image.png.f3fed2caa212019ffd5d112db6aa65d9.png

 

image.png.da1083f5449bd822d948673846af89b9.png

 

I never thought to keep a backup copy of the corrupted database for further inspection but once I got the backup copy up and running I deleted it.  I even had a copy of /var/run/mysql as a backup in the event that purging/re-installing MariaDB didn't fix the issue, but once it was clear everything was working again I deleted it.  But as far as I can tell, everything looks fine.  All I can figure is that I encountered some kind of weird bug/edge case.  I am running an older system.  The "server" originally started out as an old HP Pavilion P6803W tower PC that I bought ages ago.  Since then it has received an upgrade to a 6 core AMD Phenom II processor, 16GB of RAM, a new power supply, new case, etc.  The only original part is the motherboard.  However, all of the hardware in it is old and used, and the RAM isn't ECC, so it's totally possible that there was some sort of a bit flip or some other hardware issue.  I haven't had any issues in the past, but that doesn't mean they can't start, especially since the system has been running basically 24/7/365 for going on a decade now.  The temps have always been in great shape because I put an over-sized 125 watt cooler on a 95 watt chip.

image.png.2f78d8657437b73c344eb786920159d6.png

 

There's no indications of this being any kind of an attack either.  No changes made to my firewall rules, no new packages installed or removed, no modifications to any of my systemd service files, no files apparently tampered with or bothered, nobody banned by Fail2Ban, no unexpected auth attempts or blocked traffic on my firewall, no weird entries in syslog/kern.log, (at least that I've noticed) etc.

 

On the hardware front all the drives check out after running some short SMART tests, but I will see about doing a memtest scan on it at some point just to verify whether there are any issues with the RAM.  I'm gonna hope that it was just a software bug and I don't encounter it again because even though I don't mind replacing the server, I'm kind of attached to the old girl, :p  I will also verify that I don't have any other services hogging up my RAM as well just to be safe.

Edited by Gerowen
added screenshot as evidence of free space

At least from the quick views, nothing looks out of place. Knowing the the hardware is as old as it is could be just a really unfortunately timed hiccup. If the drivs check out and no bad sectors found, my next check would be the ram.

 

Keep up on those backs to be safe and I hope it does not happen again.🤞

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Posts

    • Why you need to take back control of your synced passwords and how to go about doing that by Paul Hill Credit: Pixabay Last month, when Google decided to introduce daily and weekly caps for Gemini, it reignited an anxiety of mine, that you can’t really depend on service providers to maintain features forever, and it got me looking into free software (as in freedom) in other areas too. One app I quickly came across was KeePassXC on desktop and KeePassDX on Android as an alternative to password manager lock-in within the Chrome or Firefox ecosystems. I personally like to switch around with browsers, and using either password manager is inconvenient, so something like KeePassXC was interesting to me. The main issue with it now is syncing; I was not sure how to do that. After a bit of research, I came across Syncthing, a tool I was vaguely familiar with but had never used because it seemed complicated. However, I was completely wrong, and honestly, I think everyone should use it if they use multiple devices. It essentially lets you share folders peer to peer across all of your devices, no cloud services that you don’t control necessary! And it was fairly simple to set up, if not a bit clunky. Since setting it up, I’ve also started using Syncthing to back up other apps too, so don’t think it’s limited to just saving password databases. You can use it for pretty much anything you use Dropbox or Google Drive for. Before continuing to talk about those apps a bit more, let’s walk back a bit and talk about browser sync. Ever since the late 2000s and early 2010s, really, since we have been using smartphones, browser sync has been a necessity of life. I don’t know about you, but I have hundreds of passwords saved. For the most part, they’re all unique, so I don’t remember them and rely on software to manage them for me. Until recently, I’ve relied on password managers in Chrome and Firefox, but what I always found annoying was that it can be hard to transfer them between browsers. Sure, on Windows it is simple enough, but on Linux, exporting bookmarks has been temperamental. It works OK nowadays, but not too long ago, Chrome required you to enable exporting passwords in chrome://flags. The situation is even worse on mobile; there is no exporting or importing of passwords of any kind. You literally have to do it on a desktop, which is incredibly annoying in our mobile-first world. Sync also lets us take out bookmarks, history, tabs, and autofill data easily. To enable sync, it’s just a matter of signing into the browser once, and it handles the rest. It’s nice and easy. Obviously, all this has some issues, including those I’ve outlined above about it being hard to transfer data between browsers, but also things such as account suspension, lost account passwords, and other lock-in mechanisms, such as passkeys, being tied to a specific browser. On a sidenote, I have just removed all of my passkeys because they can make it harder to move browsers. I think the biggest threat to your synced passwords, especially if doing this with Google, is having your account suspended. I don’t ever expect mine to be suspended, but you do hear horror stories on Reddit where people lose access to their Google accounts. Imagine if you have hundreds of passwords, then suddenly lose access to them because Google froze your account, what would you do? So yes, it can be nice to use these syncing services for their convenience, but they also have risks. You may have seen me going on about free software quite a bit in my editorials. It’s essentially a concept championed by the Free Software Foundation. It’s software under particular licenses that grant you four freedoms: run the program for any purpose (0), study and change the source code (1), redistribute copies to others (2), and the freedom to distribute modified copies to others (3). For example, if there is an app I use and one day it gets abandoned by the developer, I can keep running it or even clone the software and continue developing it. Look at the myriad of cool services Google has run over the years before killing them. You can’t take the source code for those because they are proprietary, for the most part. Both KeePassXC and Syncthing are free software, so I get the freedoms listed above. In my use case where I’m syncing a database full of my passwords, I also get proper ownership over my data, there is no losing access to the database due to a frozen account, I can access the code of the tools I’m using, and I can get support from real people online if I run into issues, rather than having to consult a vague help page from an opaque company. With the KeePassXC password manager, you create a .kdbx file, which is what will be synced between devices. KeePassXC has cross-platform apps and also has browser extensions so that the browser can fetch passwords from the database once it is unlocked. Meanwhile, Syncthing is a peer-to-peer file sync tool where you can select folders to sync between your devices. Just pop files in the folders you choose, and then they will be available across your other devices whenever they come online. Syncthing is resilient as it works over both LAN and the internet and only ever sends content between your devices, never to a third-party server somewhere else. By combining these two pieces of software, you can essentially replicate the browser sync functionality. I have had a weird, conflicting issue where a new file is appearing, but it doesn’t seem to be impacting my main password database, which is updating between devices just fine. If you want to get a setup similar to what I have, you will need to go here to download KeePassXC for your computer. Once you have that, you will need to download your passwords from your web browser to a CSV file. In Chrome, you can type chrome://password-manager/settings into the URL bar, and you should see an option to download your passwords under Export Passwords. This will give you the CSV file you need for importing into KeePassXC. If you use a different browser, just use a search engine and type “browser-name export passwords” and muddle along. In KeePassXC, you’ll want to press Import File from the home screen, select the CSV file, and create a new database from it. On one of the screens of the wizard, there will be a Title field with a drop-down selected to none. Change this to Title and continue. You’ll select a name for the database, the encryption level (the defaults are fine), and then you will pick a password. I would choose four unrelated words that are easy for you to remember, as you’ll be typing them fairly often to access your passwords. When you have all your passwords in your new database, you will want to set up the browser extension so that your browser can fetch passwords from KeePassXC. Rather than explain how to do that here, refer to KeePassXC’s guide on how to set it up properly. Once you’ve got that set up, you want to install KeePassDX on Android. You can grab it on the F-Droid store and the Google Play Store. For iPhone users, there are other .kdbx-supporting apps, but I haven’t tried any of them, so have a look around and use what suits you. Once you have that done, you will want to install Syncthing on your computer and find a third-party app for your mobile device. On Android, I use an app called BasicSync; there are also options for iOS, but again, I’ve not tried these. Once you’ve got SyncThing, you’ll want to set it up and connect all of your devices together and share a folder between your gadgets. PCWorld has a good tutorial on setting up a synchronized file between your devices using SyncThing. Once you’ve set it up, congrats, you’ll never have to touch that stuff again except for adding or removing devices. I’ll be honest, I didn’t particularly like setting up Syncthing. It didn’t take me a massive amount of time, but I think I had to check online because I found it a bit confusing. That said, I’ve had it running for several weeks now and never need to touch the Syncthing settings, so that’s very nice. I also mentioned a conflicting file. I’m not sure why this is appearing, but the main .kdbx file seems to be updating and syncing just fine. What’s nice is that both KeePassXC and Syncthing are free software, so they won’t just vanish one day; you can take the code and fork the project or use a range of alternative implementations that others have made. It’s also nice that it works over LAN, so even if your ISP is having problems, your passwords will still sync. One area where you will want to be a bit more careful with this setup is if you only have one device. I am OK because I have a computer and two phones, all synced up. If you just have one device, you will probably want to store a backup of your .kdbx file somewhere else. Obviously, you’ll also want to remember your password really well, too. If you get locked out, it's game over. Overall, if you want to take back control of your computing from big tech, taking control of your passwords is an important part of this. You don’t need to immediately clear out your browser’s password manager; try running KeePassXC and the password manager concurrently for a while to see if you run into any problems. If you do try this out, let us know some other creative ways to use Syncthing. I haven’t really come up with a solution about what to do with my bookmarks, for example.
    • If the price was a dollar, someone would complain "Why isn't it free?" If it was free, someone would complain they weren't being paid to play it.
    • That lens of history will burn if you hold it at the right angle... Warn users too late: Shame, Microsoft! That extremely minor update to an obscure Control Panel widget required 2 years of warning. Warn users too early: Shame, Microsoft! We've got better things to do. Pipeline and process be damned, we'll just always be disappointed, eh?
    • Microsoft Paint used to be my favorite Windows app as a kid, and it's still pretty good by Usama Jawad I have been using Windows since the early 2000s, when I was around 10 years old or so. I vaguely remember playing around with Windows 98 and Windows 2000, but that may have been on school PCs which had old operating systems installed. My main OS on the home PC, and the one I recall spending most time with, was Windows XP. At that time, I used the home PC to create Word and PowerPoint documents for school, but a lot of the time, I simply used it to play games. My dad would bring game discs which we would try and install on the PC, sometimes unsuccessfully, and sometimes, we would rely on flash games in the browser, like Bubble Trouble on Miniclip. However, the problem with the latter approach was the internet speed. On a good day, our dial-up internet would offer us speeds of 56 kbps, but on most days, it was closer to 33 kbps. This did not facilitate online gaming as I would often have to wait minutes for a game to load or "draw" on the screen, and trying to download pirated games wasn't simple either. I remember getting tired of waiting for online games to load and just downloading simulator games from the Big Fish Games website instead, only to be disappointed after finding out that I was just being given access to trial versions of the title, and I needed to fork out money to pay for the full version. All of this is to say that it wasn't very easy to find entertainment options on the home PC when I was a kid, due to a number of reasons, mostly outside of my control. This situation pushed me towards a rather unconventional ally: Microsoft Paint. Whenever the internet wasn't working as good as I expected, I would simply spin up Paint and draw complete rubbish on the canvas. Of course, that wasn't always the intention, but it usually happened when I messed up drawing a straight line or something, and then I would give up on that particular piece and simply draw a random collection of objects. Microsoft Paint was extremely accessible and easy to use. Even if you weren't an artist, you could quickly understand the tools at your disposal and how to leverage them on a canvas. The absolute breadth on offer ensured that each painting was truly unique, as you could utilize various combinations of tools like the pencil, paint, spray paint, and more to truly personalize your creation. Since I wasn't particularly good at drawing both on digital screen or a physical screen, I remember that my main style of art would be to insert a bunch of randomly intersecting lines and then fill them with random colors through the paint can. I have trying to replicate that art style in the latest version of Paint below, and as you can see, it's truly Pablo Picasso-esque. The human imagination truly knows no bounds Microsoft Paint kept me occupied for hours and was my best friend when video games on the home PC were inaccessible for one reason or the other. There was no academic or professional reason for which I would need to use Paint, but I still loved using it in my personal time, even if what I created wasn't worth being shown to anyone. It was simply fun. Fast-forward to today, and the situation is mostly the same. Now that I am almost 29 years old, and I still have no reason to use Microsoft Paint in a professional capacity. In fact, I don't even use it in a personal capacity, except to dabble with it from time to time, just to see if core functionalities are still intact. And I'm happy to say that I think Microsoft Paint still offers the same accessibility and inviting experience that it did to me a couple of decades ago, even though its UX has been refreshed and it's been integrated with Copilot features. Interestingly, things could have been a lot different, had Microsoft had its way. Microsoft Paint was marked for deprecation with the Windows 10 Fall Creators Update in 2017, and even began displaying a product retirement alert, urging customers to shift to Paint 3D instead. Fortunately, after consumer backlash, Microsoft reversed course on this decision, and Paint continues to be a native app inside Windows installations that can also be updated quite frequently through the Microsoft Store. Instead, Paint 3D ended up on the chopping block, which is for the better, I think. I have intermittently played around with Microsoft's refreshed Paint experience in the past few years, and I do think it has received worthwhile upgrades. the UI and the UX has been modernized while retaining core functionality, and the app is still fairly easy to use. It doesn't meet any of my use-cases, but I've never really had any use-cases ever, as described previously. Of course, the elephant in the room is the Copilot integration. Personally, I believe that this is one place where Copilot does make sense, environmental concerns aside. I know that a lot of creatives use AI to generate images, and while some may be using professional alternatives, Paint still offers a decent casual experience, with the power of Copilot. Of course, you do need to have a valid Microsoft 365 Copilot license and available credits to use it, but even if you don't, you still get the big Copilot button in the toolbar, unfortunately. All in all, I am glad that Microsoft Paint continues to be a native feature in Windows 11, and a piece of software that has evolved to meet modern needs without cutting off its own roots. It's just an iconic piece of Windows history that was an essential part of my childhood, and while I don't use it anymore, I'm just glad it is still there.
    • 2TB WD_Black SN7100 PCIe Gen4 NVMe SSD drops to its lowest price in over three months by Fiza Ali Amazon is currently offering the 2TB WD_Black SN7100 internal solid-state drive at its lowest price in over three months, so you may want to check it out, if you have been considering a storage upgrade, before the deal dries up (purchase link is toward the end of the article). Featuring a PCIe Gen 4.0 interface and M.2 2280 form factor, the SN7100 promises to deliver sequential read speeds of up to 7,250MB/s and sequential write speeds reaching 6,900MB/s, offering as much as a 35% improvement in performance compared with the previous generation. It also achieves random read speeds of 1,000,000 IOPS and random write speeds of 1,400,000 IOPS. The drive uses Western Digital’s TLC 3D NAND technology for reliable performance and is further supported by a five-year limited warranty. It also offers strong endurance, rated at up to 1,200TBW, making it suitable for demanding workloads such as gaming, content creation, and high-speed recording. Moreover, its DRAM-less architecture claims to improve power efficiency (the SSD relies on system memory for caching via HMB), while the WD_Black Dashboard software enables users to monitor drive health, install firmware updates, and activate Game Mode for potentially better performance. Finally, it operates within an operating temperature range of 0°C to 85°C, and can withstand storage temperatures from -40°C to 85°C. 2TB WD_Black SN7100 PCIe Gen4 NVMe SSD: $242.96 (Amazon US) Check this deal out if you want a 4TB option. Good to know This Amazon deal is U.S. specific, and not available in other regions unless specified. We only use first-party seller links (at the time of article publishing); ensure that you purchase from a first-party seller link only. Check out Today's Deals on Amazon | or our recent tech deals. Become a Prime member (for Students or SNAP) via Neowin Get Prime Access - Prime for half price (for qualifying Medicaid, EBT, SNAP) Subscribe to Prime Video, Audible Plus, Music Unlimited or Kindle Unlimited via Neowin As an Amazon Associate, we earn from qualifying purchases.
  • Recent Achievements

    • Week One Done
      Supreme Spray LV earned a badge
      Week One Done
    • One Month Later
      Genuinetonerink- Dubai earned a badge
      One Month Later
    • Week One Done
      Genuinetonerink- Dubai earned a badge
      Week One Done
    • One Year In
      hhgygy earned a badge
      One Year In
    • Week One Done
      AMV earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      514
    2. 2
      +Edouard
      163
    3. 3
      PsYcHoKiLLa
      87
    4. 4
      Steven P.
      74
    5. 5
      Michael Scrip
      73
  • Tell a friend

    Love Neowin? Tell a friend!