• 0

Remove all duplicates from text files. Only leave unique lines.


Question

Hi folks,

 

I am looking for a way to remove all duplicate lines from a text file.  If there are 2 or more of the same line I want both/all removed so that the final text file will just have the totally unique lines left.

 

As a bit of background to what we are doing:

 

We  import images and data relating to the images into a database.

We then import some different data into the same database that, using some reference numbers, is matched with the 1st set of data.

We export the amalgamated data and send it to another company.

Each week we import more of both set of data.

Each week we export this data, but only need to send the "new" data to the other company.

Not all data from week n will be exported in week n.  

 

I am currently running a find & replace (using Useful File Utilities) using week 1 export against week 2 export to see if it will actually work - we have 100,000+ lines to find & replace on(increasing each week).

 

I suppose the question is, if I merge week 1 and week 2 text files and sort into order, is there software/script that will copy totally unique lines to a new text file?

 

Thanks in advance for any help.

2 answers to this question

Recommended Posts

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • The Xbox game bar says hi. It's been able to show you CPU, RAM, GPU and FPS for as long as I can remember.
    • WinSCP 6.5.2 by Razvan Serea  WinSCP is an open source free SFTP client, FTP client, WebDAV client and SCP client for Windows. Its main function is file transfer between a local and a remote computer. Beyond this, WinSCP offers scripting and basic file manager functionality. WinSCP features: Graphical user interface Translated into several languages Integration with Windows (drag&drop, URL, shortcut icons) U3 support All common operations with files Support for SFTP and SCP protocols over SSH-1 and SSH-2 and plain old FTP protocol Batch file scripting and command-line interface Directory synchronization in several semi or fully automatic ways Integrated text editor Support for SSH password, keyboard-interactive, public key and Kerberos (GSS) authentication Integrates with Pageant (PuTTY authentication agent) for full support of public key authentication with SSH Explorer and Commander interfaces Optionally stores session information Optionally supports portable operation using a configuration file in place of registry entries, suitable for operation from removable media WinSCP 6.5.2 changelog: Thumbnail view in file panels. Three selectable sizes of toolbar icons, showing slightly larger size by default. Switching to Segoe UI font with slightly larger size. Improvements to Synchronization checklist window, including resolving file moves and pushing synchronization to background queue. Ongoing local delete operation can be moved to a background queue. Optimized working with large local directories. Compatibility with new OneDrive WebDAV interface. Dark theme for session tabs. Improvements to S3 support, including more options to authentication and display and modification of S3 file/object tags. List of all changes. Download: WinSCP 6.5.2 | 11.6 MB (Open Source) Download: WinSCP MSI | 28.7 MB Download: Standalone Executable | 8.4 MB Link: WinSCP Home page | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • QOwnNotes 25.6.2 by Razvan Serea QOwnNotes is a open source (GPL) plain-text file notepad with markdown support and todo list manager for GNU/Linux, Mac OS X and Windows, that (optionally) works together with the notes application of ownCloud (or Nextcloud). So you are able to write down your thoughts with QOwnNotes and edit or search for them later from your mobile device (like with CloudNotes) or the ownCloud web-service. The notes are stored as plain text files and you can sync them with your ownCloud sync client. Of course other software, like Dropbox, Syncthing, Seafile or BitTorrent Sync can be used too. Features: the notes folder can be freely chosen (multiple note folders can be used) sub-string searching of notes is possible and search results are highlighted in the notes application can be operated with customizable keyboard shortcuts external changes of note files are watched (notes or note list are reloaded) older versions of your notes can be restored from your ownCloud server trashed notes can be restored from your ownCloud server differences between current note and externally changed note are showed in a dialog markdown highlighting of notes and a markdown preview mode notes are getting their name from the first line of the note text (just like in the ownCloud notes web-application) and the note text files are automatically renamed, if the the first line changes compatible with the notes web-application of ownCloud and mobile ownCloud notes applications compatible with ownCloud's selective sync feature by supporting an unlimited amount of note folders with the ability to choose the respective folder on your server manage your ownCloud todo lists (ownCloud tasks or Tasks Plus / Calendar Plus) or use an other CalDAV server to sync your tasks to encryption of notes (AES-256 is built in or you can use custom encryption methods like Keybase.io (encryption-keybase.qml) or PGP (encryption-pgp.qml)) dark mode theme support theming support for the markdown syntax highlighting all panels can be placed wherever you want, they can even float or stack (fully dockable) support for freedesktop theme icons, you can use QOwnNotes with your native desktop icons and with your favorite dark desktop theme support for hierarchical note tagging and note subfolders support for sharing notes on your ownCloud server portable mode for carrying QOwnNotes around on USB sticks Evernote import QOwnNotes is available in many different languages like English, German, French, Polish, Chinese, Japanese, Russian, Portuguese, Hungarian, Dutch and Spanish Changes in QOwnNotes 25.6.2: The Find action dialog is now working again (for #3294) Added more French translation (thank you, jd-develop) Download: QOwnNotes 25.6.2 | 37.3 MB (Open Source) Download: QOwnNotes for Other Operating Systems View: QOwnNotes Home Page | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • So it’s fine he scams people to troll others? 🙄 This is a repeat of his watch fiasco.
  • Recent Achievements

    • First Post
      Fuzz_c earned a badge
      First Post
    • First Post
      TIGOSS earned a badge
      First Post
    • Week One Done
      slackerzz earned a badge
      Week One Done
    • Week One Done
      vivetool earned a badge
      Week One Done
    • Reacting Well
      pnajbar earned a badge
      Reacting Well
  • Popular Contributors

    1. 1
      +primortal
      700
    2. 2
      ATLien_0
      279
    3. 3
      Michael Scrip
      209
    4. 4
      +FloatingFatMan
      195
    5. 5
      Steven P.
      130
  • Tell a friend

    Love Neowin? Tell a friend!