Remove all duplicates from text files. Only leave unique lines.

[metallithrax]

Hi folks,

 

I am looking for a way to remove all duplicate lines from a text file.  If there are 2 or more of the same line I want both/all removed so that the final text file will just have the totally unique lines left.

 

As a bit of background to what we are doing:

 

We  import images and data relating to the images into a database.

We then import some different data into the same database that, using some reference numbers, is matched with the 1st set of data.

We export the amalgamated data and send it to another company.

Each week we import more of both set of data.

Each week we export this data, but only need to send the "new" data to the other company.

Not all data from week n will be exported in week n.  

 

I am currently running a find & replace (using Useful File Utilities) using week 1 export against week 2 export to see if it will actually work - we have 100,000+ lines to find & replace on(increasing each week).

 

I suppose the question is, if I merge week 1 and week 2 text files and sort into order, is there software/script that will copy totally unique lines to a new text file?

 

Thanks in advance for any help.

[fusi0n]

Use NotePad++, RegEx, find replace

^(.*)(\r?\n\1)+$

replacing with \1.

[metallithrax]

Thanks for that, it does seem to work.  Now just a few other things to figure out with this job.