Cron Task to Download CSV and Dump Into MySQL


Recommended Posts

Hey,
Ok..Trying to wrap my head around this..
Basically, what I'm trying to do is every thirty minutes, download a csv file, and then dump it into MySQL database, so that I can view the results from my PHP website...
I haven't tested this yet (Never used a cron before), but from my searching, to setup the cron, I would use:
 

0,30 * * * * wget -O - http://example.com/test.csv

But..Once I have the csv file..I can't seem to figure out what to do with it...Maybe I'm just tired, but I have no idea how to get it into the database...A push in the right direction would help a lot..

 

(In case it matters, using Debian, everything up-to-date)

 

Thanks!

You would then run a sql query via MySQL command line. So you would set up another cron to run a minute or an hour later. Depending on how long it takes the csv to download. So you figure out the time it takes to download the csv and then run a bash script at that time interval. I Think it's cleaner to point cron to a bash script that runs the sql query instead of inside the cron file.

Look up MySQL queries view command line.

Example

$ mysql -u vivek -p -e 'SELECT COUNT(*) FROM quotes' cbzquotes

Source: http://www.cyberciti.biz/faq/run-sql-query-directly-on-the-command-line/

So you would just run an import into the table your php script parses for the data.

A bash script..That's a good idea...Never used one of those either, I'll have to read up on those, thank for the suggestion!

And I should have figured out MySQL from command line...Think I've been staring at lines of code for too long today..Think I was partly trying to over-complicate things...lol Never liked it too much though...I still need to get more comfortable with command line.

But I'll look into the bash script idea, thanks!

Bash scripts are easy and wonderful. Throw a quick file together and name it .sh. Make it executable by running chmod +x filename.sh

Example:

#!/bin/sh

MySQL -u user -p password -h localhost -e 'query'

Put the above 2 lines in a file, save it and have cron run the file.

Same cron entry like above with obviously different interval. The command will be /path/to/script.sh

Cron run it as root so you don't have to worry about permissions just make sure it's executable.

Thanks :)

I had thought about learning about Bash scripts when I was trying to learn Linux, but could never think of a use for it, so never got around to it lol Looks fairly straight forward..Hopefully this weekend I'll have time to play around with it, I'll let you know how it goes!

Ok, finally had a chance to work on this..Had to work on something else for the project quickly.

Anyways, I'm getting commands not found...

#!/bin/sh

wget "http://www.example.com/test.csv"

sleep 40

MySQL -u user -p password

use databseName

load data local infile 'table.csv' into table tableName
IGNORE 1 LINES
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(column1, column2, column3, column4, column5, column6, column7)

And I get this returned:

/var/www/html/scripts/test.sh: line 7: MySQL: command not found
/var/www/html/scripts/test.sh: line 9: use: command not found
/var/www/html/scripts/test.sh: line 11: load: command not found
/var/www/html/scripts/test.sh: line 12: IGNORE: command not found
/var/www/html/scripts/test.sh: line 13: fields: command not found
/var/www/html/scripts/test.sh: line 14: enclosed: command not found
/var/www/html/scripts/test.sh: line 15: lines: command not found
/var/www/html/scripts/test.sh: line 16: column1,: command not found

It seemed to have downloaded the file..However when I check /var/www/html/scripts/, the csv file isn't there..

But still not sure what to do about the commands not found..What did I do wrong..?

Thanks! Using something similar..(I had quotes in there, so changed it to QUERY_INPUT)

#!/bin/sh

wget -O table.csv "http://www.example.com/table.csv"

sleep 40

mysql -u<user> -p<password> dataBase <<QUERY_INPUT

load data local infile 'table.csv' into table testing
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
IGNORE 1 ROWS
(column1, column2, column3, column4, column5, column6, column7)
QUERY_INPUT

the 'mysql' part needed to be lowercase as well.

That's the final file for anyone who needs it.

The 'sleep 40' is there to make sure the file downloads before loading MySQL.

'Ignore 1 Rows' is to ignore the first row

 

Thanks for the help!

 

Marking Sikh's as the answer, however thank you Haggis as well!

Didn't get notifications for this, gurrrr. Thanks haggis for helping out.

 

Glad you figured it out. I would make the sleep a minute or two just incase mysql gets tied up for some reason. Its always good to overestimate then it is to underestimate because if you don't get email notifications that your script is running or failed to run, the next time you look at it, it might have not been working for 2 weeks or 2 days.

 

So I recommend up'ing the sleep

This topic is now closed to further replies.
  • Posts

    • Wall cabinet: Fiber connection (1Gbps) Patch panel 2.5Gbps Router A mini PC with Ryzen 5800H CPU and 32GB DDR4 RAM, for network related tools (Portainer, Pi-Hole, Cloudflared, Tailscale, Uptime Kuma, Wetty) A headless server running Proxmox: AMD Ryzen 9 9950X CPU, 96GB DDR5, RTX 3090 24GB GPU, 10Gbps NIC, and a bunch of NVMs, SATA SSDs, HDDs A VM with TrueNAS for backups and media server via Jellyfin, passing through the SATA storage controller A VM with an Ubuntu server acting as a web and a GPU server for AI workloads, and a TeamCity build/deploy agent for my personal projects This setup has evolved many times for the last 2 years, I have not gone completely crazy, yet (I think), but still experimenting, still learning, it is a fun and rewarding experience having a home lab!
    • Microsoft PC Manager 3.17.4.0 (Offline Installer) by Razvan Serea With Microsoft PC Manager, users can easily perform basic computer maintenance and enhance the speed of their devices with just one click. This app offers a range of features, including disk cleanup, startup app management, virus scanning, Windows Update checks, process monitoring, and storage management. Microsoft PC Manager key features: Storage Manager- easily uninstall infrequently used apps, manage large files, perform a cleanup, and set up Storage Sense to automatically clear temporary files. Health Checkup feature -scans for potential problems, viruses, and startup programs to turn off. It helps you identify unnecessary items to remove, optimizing your system's performance. Pop-up Management - block pop-up windows from appearing in apps. Windows Update - scans your system for any pending updates. Startup Apps - enable or disable startup apps on your PC, allowing you to optimize your system's startup performance. Browser Protection - rest assured that harmful programs cannot alter your default browser. Also enables you to change your default browser. Process Management - allows you to conveniently terminate any active process, ensuring optimal system performance and resource utilization. Anti-virus protection - Fully integrated with Windows Security. Safeguard your PC anytime. Quick Steps: Download Microsoft PC Manager Offline Installer (APPX/MSIX) with Adguard Adguard serves as a third-party online service, offering a user-friendly method for directly downloading appx, appxbundle, and msixbundle files from the Microsoft Store. Official download links will be generated for both the app's various versions and its dependency packages. How to download Microsoft PC Manager Offline Installer (APPX/MSIX) 1. Initially, you must find the app URL within the Microsoft Store. Access the Microsoft Store via your browser and search for "Microsoft PC Manager". Once located, copy the app URL, which includes the product ID, either from the address bar or from the provided link below. https://apps.microsoft.com/detail/9PM860492SZD 2. Now paste the app URL into the designated area, then click the check mark button to produce a direct download link. 3. To download, right-click the relevant link and select “Save link as…” from your browser's menu. Occasionally, Microsoft Edge may flag the download as insecure. In such cases, consider utilizing alternative browsers such as Google Chrome or Firefox to successfully complete the download. Microsoft PC Manager is a completely free tool optimized exclusively for use on Windows 10 (version 1809 or newer) and Windows 11. Download: Microsoft PC Manager 3.17.4.0 | from Microsoft Store View: Microsoft PC Manager Home Page Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • You might want to update to 1TB drive.  They are not very expensive these days, there are plenty of options - simple look I would think you could find a 1TB ssd or even nvme somewhere between 50 and 100 bucks. when I have moved to larger ssds in laptop in the past, you can just connect the drive via usb - image.  Then take the old drive out, put in the new drive and there you go..  If your concerned about how much space you have left this prob your best bet..   
    • I see! Wasn’t aware of that. Even though they started off with desktop app for Mac before Windows. Should be on it’s way soon! 🤞
    • support.microsoft.com/en-us/topic/microsoft-defender-update-for-windows-operating-system-installation-images-1c89630b-61ff-00a1-04e2-2d1f3865450d
  • Recent Achievements

    • First Post
      Johnny Mrkvička earned a badge
      First Post
    • Week One Done
      viraltui earned a badge
      Week One Done
    • One Month Later
      serfegyed earned a badge
      One Month Later
    • Dedicated
      firey earned a badge
      Dedicated
    • Dedicated
      fettermanj earned a badge
      Dedicated
  • Popular Contributors

    1. 1
      +primortal
      627
    2. 2
      ATLien_0
      223
    3. 3
      Michael Scrip
      217
    4. 4
      Xenon
      149
    5. 5
      Steven P.
      139
  • Tell a friend

    Love Neowin? Tell a friend!