How can I move files with specific filenames and re-create the directory structure they are in?

[Cyber Akuma]

I have literally thousands of files, and all of them are within several levels of subfolders. There are a chunk I want to move to a different location, all of which have the same string of characters at the end of their filename. 
 
While I could just simply search through the root folder for *[string here]* and then move all the files that are found, the issue is if I do it this way I will just end up dumping all of the files into a single location. I want to be able to re-create the directory structure of the files as well in the new location. Is there any way to do this easily? 
 
For example: Say I have the folder C:\something\something\targetfolder 
 
And within \targetfolder there are many levels of subfolders:  
 \targetfolder\01\01\01  
 \targetfolder\01\01\02  
 \targetfolder\01\02\01  
 \targerfolder\02\01\01 
 
etc, and at the end of each set of subfolders are where the files are. Any of the files that match the text string I am searching for, I would like to put them in something like "C:\something\something\otherfolder" and keep the all the subfolders intact for the moved files. 
 
So something like "C:\something\something\targetfolder\01\01\02\file.txt" would end up in "C:\something\something\otherfolder\01\01\02\file.txt" and so on for each file that matches.

[Fleet Command]

Sounds like something that PowerShell would do. Depending on your need, you should use the -Include and -Exclude switches of Get-ChildItem or use more complex matches with -like, -eq, and -match.

 

Unfortunately, your question is too vague for me to decide which. If you decide not to share more details, here are the relevant help files:

• Get-ChildItem
• ForEach-Object
• Comparison operators
• System.IO.FileInfo class (The output of Get-ChildItem is an array, each element of which is this class)

[Cyber Akuma]

Never had any experience using Powershell at all to be honest.

 

I basically want to move a few thousand files that are on a MicroSD card and buried within subfolders, but I want to keep the subfolder structure intact in case I need to move them back. Which specific details were you looking for?

[Fleet Command]

Huh! I didn't expect this. Let me see if I have understood correctly.

• You have a MicroSD  card and you have plugged it into an appropriate card reader
• It has either of the following:
  • A lot of subfolders containing files
  • A folder containing the above
• You want to copy all of them (no exceptions) to a "C:\something\something\targetfolder" folder while preserving the folder hierarchy

Well, this is too simple. You don't need PowerShell. Do this:

1. Open File Explorer
2. Go to This PC
3. Go to your MicroSD's volume
4. (Optional) Go inside the subfolder who content you want to copy. (If you want copy the whole MicroSD, you don't need this step. You are already there.)
5. Press the Ctrl+A key combination on your keyboard
6. Press the Ctrl+C key combination on your keyboard
7. Go to "C:\something\something\targetfolder"
8. Press the Ctrl+V key combination

On health computer, this sequence copies the files over,  preserving their hierarchy.

[Cyber Akuma]

On 26/03/2022 at 01:11, Fleet Command said:

You want to copy all of them (no exceptions) to a "C:\something\something\targetfolder" folder while preserving the folder hierarchy

 

No, that part is wrong. I want to move (not copy) only files that match a specific text string, and I want whatever series of subfolders they are in to be re-created at the target destination.

[Fleet Command]

A series of PowerShell commands like this help you:

 

# Assuming D:\ is your MicroSD volume
Set-Location "D:\"

Get-ChildItem -Recurse -Force | ForEach-Object {
  $NewLocation = $_.FullName -replace ("D:\", "C:\something\something\targetfolder\")


  <#
  The following line is the core of your script. Customize it to return a boolean.
  In this example, it returns "true" if the file name starts with "TopSecret" (case insensitive)
  See "about_Comparison_Operators" for details
  #>
  $MatchCondition = $_.Name -like 'TopSecret*'


  if ($MatchCondition) {
    Copy-Item -Path $_.FullName -Destination $NewLocation
  }
}

Items between # and end of the line, or between <# and #>, are comments. PowerShell ignores these.

 

Edit:

The script doesn't have to be this complex, though. For simple cases, this suffices:

Set-Location "D:\"
Get-ChildItem -Recurse -Force | Where-Object Name -Like "TopSecret*" | ForEach-Object {
  $NewLocation = $_.FullName -replace ("D:\", "C:\something\something\targetfolder\")
  Copy-Item -Path $_.FullName -Destination $NewLocation
}

 

Or even this:

Set-Location "D:\"
Get-ChildItem -Include "TopSecret*" -Recurse -Force | ForEach-Object {
  $NewLocation = $_.FullName -replace ("D:\", "C:\something\something\targetfolder\")
  Copy-Item -Path $_.FullName -Destination $NewLocation
}

 

Edited March 26, 2022 by Fleet Command

[Fleet Command]

In the examples above, I used Copy-Item for safety, but you can use Move-Item once you are sure the script copies exactly what you want.

 

Another safety measure (for testing the script) is to add " -WhatIf" (without quotation marks, don't forget the preceding space) at the end of Move-Item or Copy-Item commands. This forces the command to tell you what it would do without doing it.

[Cyber Akuma]

Is this only something that works in Powershell? I was seeing what I could do with just a common .bat file, but I am still pretty new to that and was going to create it to just bruteforce attempt every possible folder. (The subfolders are based on dates so it's not like I am brute-forcing the whole ASCII characterset).

 

Actually, if it helps, the folders are basically nested dates it seems. In the root folder of what I want to do there are years, and within each year there are month subfolders, and within each of those is a day subfolder... and in those are the files. Not all dates are populated since there isn't a file for every day of every year.

 

E:G: \folder\2018\05\12 for March 12, 2018.

 

I was trying to look over the code you wrote for PowerShell and to be honest, I don't really undertstand how it works, or even what you want me to put in for "topsecret". None of the files are anything secret by the way.

[Fleet Command]

On 26/03/2022 at 12:36, Cyber Akuma said:

Is this only something that works in Powershell? I was seeing what I could do with just a common .bat file

Batch files (.bat) are too hard and too finicky. My brain cannot interpret something so ugly as "for /f %%i in ('dir "%temp%\test*.log" /o:-d /t:w /b') do".

 

Very sorry.

 

PowerShell has several advantages:

1. It handles files and folders well, even if the names are non-English, e.g., Japanese, Arabic, Hebrew, or Hindi
2. Its syntax is lovely. It's almost plain English.
3. It runs on Linux and Mac too.

PowerShell scripts use a .ps1 filename extension, instead of .bat. Still, you type out everything I wrote for you. Windows comes with PowerShell ISE, which colorizes PowerShell scripts, so that you understand them better. You can copy and paste the script into PowerShell ISE, edit it there to meet your requirements, save it, and run it from there.

 

I use Visual Studio Code, which is even better and open-source.

 

On 26/03/2022 at 12:36, Cyber Akuma said:

if it helps, the folders are basically nested dates it seems.

You said "files that match a specific text string." So, no, this bit of information doesn't help. My example script works OK regardless of the folder name.

On 26/03/2022 at 12:36, Cyber Akuma said:

I was trying to look over the code you wrote for PowerShell and to be honest, I don't really undertstand how it works, or even what you want me to put in for "topsecret". None of the files are anything secret by the way.

You said "files that match a specific text string". But you didn't give me your actual pattern. (Perhaps it is something confidential.) So, for the sake of example, I used -like 'TopSecret*', which means "any file that starts with "TopSecret" regardless of file name extension.

 

Here are more examples:

• -like 'Me and my second boyfriend *.jpg'
  Matches all .JPG files that start with "Me and my second boyfriend"
• -match '^Funny picture \d\d\d\d'
  Matches all files that start with "Funny picture ", followed by four digits, followed by anything. Example: "Funny picture 1243 (black and white).jpg"
• -match 'Funny picture \d\d\d\d'
  Matches all files that contain "Funny picture " followed by four digits. Example: "Damaged Funny picture 1243 (black and white).jpg"

As I told you before, the about_Comparison_Operators page helps you understand how to compose it.

 

If your pattern is really confidential, perhaps you can send it to me in a personal message. I'll write a custom script for you and send it back privately. Another possibility is that you don't have a specific pattern in mind.

[Cyber Akuma]

On 26/03/2022 at 03:42, Fleet Command said:

You said "files that match a specific text string". But you didn't give me your actual pattern. (Perhaps it is something confidential.) So, for the sake of example, I used -like 'TopSecret*', which means "any file that starts with "TopSecret" regardless of file name extension.

The files have a date/time (I assume this is unix time), a hyphen, and then a dozen or so character HEX string that works as an identifying ID. I am identifying which ones I want to move and which ones I want to keep by that ID.

 

It's nothing confidential, one example is: 2018051318554400-57B4628D2267231D57E0FC1078C0596D

 

And this would be in The folders \2018\05\13

[Fleet Command]

On 26/03/2022 at 13:47, Cyber Akuma said:

I am identifying which ones I want to move and which ones I want to keep by that ID.

Solution one:

If the ID is fixed, this might work: -match '-57B4628D2267231D57E0FC1078C0596D'
This catches (and copies) any file whose name contains "-57B4628D2267231D57E0FC1078C0596D". For example:

• Cat-57B4628D2267231D57E0FC1078C0596D-bad
• Dog-57B4628D2267231D57E0FC1078C0596D

Solution two:

If the ID is random, but has a fixed length of 32, and always appears at the end of the (i.e., there is nothing after it, not even extension), this might work: -match '-[0123456789ABCDEF]{32}$'

It matches the following sequence: a hyphen, the 32 characters (a sequence consisting of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F), and end of the filename. For example:

• Dog-57B4628D2267231D57E0FC1078C0596D
• Cat-68C5739E3178342E68F0AC2089D1569F

Remove the $ sign at the end of the pattern code to make it match a mid-filename string, e.g.

• Cat-68C5739E3178342E68F0AC2089D1569F-bad

Other solutions:

If the file name is exactly as you've shown in the example, try these:

• -match '^\d{16}-57B4628D2267231D57E0FC1078C0596D$'
  This matches file names that have this exact composition: 16 digits, hyphen, "57B4628D2267231D57E0FC1078C0596D". This is a derivative of solution one.
• -match '^\d{14}-57B4628D2267231D57E0FC1078C0596D$'
  This matches file names that have this exact composition: "20", 14 digits, hyphen, "57B4628D2267231D57E0FC1078C0596D". This is a derivative of solution one.
• -match '^\d{16}-[0123456789ABCDEF]{32}$'
  This matches file names that have this exact composition: 16 digits, hyphen, 32 characters consisting of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F. This is a derivative of solution two.
• -match '^\d{14}-[0123456789ABCDEF]{32}$'
  This matches file names that have this exact composition: "20", 14 digits, hyphen, 32 characters consisting of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F. This is a derivative of solution two.

How to use the solutions above:

There a is part of my code (see my earlier posts above) that reads: $MatchCondition = $_.Name -like 'TopSecret*'

You have replace anything after "$_.Name " with one of my solutions. For example, if I wanted to use solution one, the code would become:

$MatchCondition = $_.Name -match '-57B4628D2267231D57E0FC1078C0596D'

[Cyber Akuma]

On 26/03/2022 at 04:54, Fleet Command said:

If the ID is fixed, this might work: -match '-57B4628D2267231D57E0FC1078C0596D'

If by fixed you mean everything that I want to move contains the same ID then yes. AFAIK none of the filenames have anything after the ID, but even if they did, the ones I want to move all would have the same ID at the end of their filename. (Extensions can differ though, but a .* should fix that). I do want to move multiple groups of IDs, but separately, so for something like that I can just run a script twice. (e.g. Move id -AAAAAAA to a folder, then run the script again and move ID -BBBBB to another folder).

 

It's not clear if the end folders or IDs in the script you are proposing would be hard-coded or not, but the whole point is that I would need to tell it what files with that IDs to move to what destination folder while keeping all the subfolder structure intact from the root folder I search in. I am not trying to constantly move the same ID to the same folder if that's what you think I am trying to do.

 

On 26/03/2022 at 04:54, Fleet Command said:

If the ID is random, but has a fixed length of 32, and always appears at the end of the (i.e., there is nothing after it, not even extension), this might work: -match '-[0123456789ABCDEF]{32}$'

 

If by random you mean the ID itself is not sequential or anything, no, they are not. If by random you mean all the files have their own IDs then no, the IDs for specific sets I want to move are all the same, so there could be hundreds to thousands of files all in different subfolders with the same ID that I would want to move while keeping the subfolder structure they are in intact in the new location... but without deleting the original subfolders or touching any files with different IDs.

 

The IDs do not appear to be fixed length, I have seen some be bigger or smaller by a few characters.

 

They do seem to always appear at the end of a file, so far I haven't noticed any files with anything after the IDs but the ID would remain the same regardless if there was something after it in the filename.

 

 

[Fleet Command]

On 26/03/2022 at 21:40, Cyber Akuma said:

If by fixed you mean everything that I want to move contains the same ID then yes.

Good! You can go with the first solution.

 

On 26/03/2022 at 21:40, Cyber Akuma said:

Extensions can differ though, but a .* should fix that

Please don't do that. My solutions employ Regex not Wildcards. Just use solution one.

 

A little definition tutorial: File name = Base name + Extension. The extension always begins with a dot.

 

In case you found yourself in dire need of comparing against the base name only, use the BaseName property instead of Name, like this:

$MatchCondition = $_.BaseName -match '-57B4628D2267231D57E0FC1078C0596D'

 

On 26/03/2022 at 21:40, Cyber Akuma said:

It's not clear if the end folders or IDs in the script you are proposing would be hard-coded or not

It is. In $MatchCondition = $_.Name -match '-57B4628D2267231D57E0FC1078C0596D', replace 57B4628D2267231D57E0FC1078C0596D with the actual ID.

 

[ZakO]

If you just need a simple wildcard ID match robocopy (built into windows) can do this too:

 

robocopy C:\folder\from C:\folder\to *57B4628D2267231D57E0FC1078C0596D* /s

 

Edited March 26, 2022 by ZakO

[Cyber Akuma]

On 26/03/2022 at 13:41, Fleet Command said:

Please don't do that. My solutions employ Regex not Wildcards. Just use solution one.

 

A little definition tutorial: File name = Base name + Extension. The extension always begins with a dot.

 

Would your solution copy all the files containing that ID regardless of their extension though?

 

And yes, I am aware of the way Windows/DOS based filenames work, and the old 8.3 limit of the DOS days.

 

On 26/03/2022 at 13:41, Fleet Command said:

t is. In $MatchCondition = $_.Name -match '-57B4628D2267231D57E0FC1078C0596D', replace 57B4628D2267231D57E0FC1078C0596D with the actual ID.

Well, if I was forced to use a script since no application would be able to do it, I trying to see how I can make the ID and source/target root folders variables I would enter rather than hard code them into the script, since they would keep changing whenever I need to run the script again.

 

On 26/03/2022 at 13:54, ZakO said:

robocopy C:\folder\from C:\folder\to *57B4628D2267231D57E0FC1078C0596D* /s

Would that clone the directory structure the files are in though? That's the main reason I am not sure how to do this. If that just simply copies all the files with that ID, I could just navigate to the root folder on Windows Explorer and type *-57B4628D2267231D57E0FC1078C0596* in the search bar, but from what I understand there is no way for the Windows GUI to replicate the folder structure this way, and I believe that command in rocobopy would not do that either?

[ZakO]

On 26/03/2022 at 19:57, Cyber Akuma said:

Would that clone the directory structure the files are in though? That's the main reason I am not sure how to do this. If that just simply copies all the files with that ID, I could just navigate to the root folder on Windows Explorer and type *-57B4628D2267231D57E0FC1078C0596* in the search bar, but from what I understand there is no way for the Windows GUI to replicate the folder structure this way, and I believe that command in rocobopy would not do that either?

Robocopy copies the directory structure

[Cyber Akuma]

Wait, it does? So that one robocopy command can do everything I wanted to do in one shot? 

 

Would it go through all the subfolders if I just point it at the root folder and copy/move all the files while duplicating the directory structure in the copied location?

[ZakO]

On 26/03/2022 at 20:39, Cyber Akuma said:

Wait, it does? So that one robocopy command can do everything I wanted to do in one shot? 

 

Would it go through all the subfolders if I just point it at the root folder and copy/move all the files while duplicating the directory structure in the copied location?

Assuming I've correctly understood what you're trying to do... yes. As an example, if I have a directory structure:

 

C:\AMD2\Chipset_Software
C:\AMD2\Chipset_Software\Binaries
C:\AMD2\Chipset_Software\Binaries\GPIO2 Driver
C:\AMD2\Chipset_Software\Binaries\PCI Driver
C:\AMD2\Chipset_Software\Binaries\GPIO2 Driver\WTx64
C:\AMD2\Chipset_Software\Binaries\GPIO2 Driver\WTx64\amdgpio2.cat
C:\AMD2\Chipset_Software\Binaries\GPIO2 Driver\WTx64\README.rtf
C:\AMD2\Chipset_Software\Binaries\PCI Driver\WTx64
C:\AMD2\Chipset_Software\Binaries\PCI Driver\WTx64\AMDPCIDev.cat
C:\AMD2\Chipset_Software\Binaries\PCI Driver\WTx64\ReadMe.rtf

 

and run the command: 

 

robocopy C:\AMD2 C:\AMD3 *README* /s

 

It will generate:

 

C:\AMD3\Chipset_Software
C:\AMD3\Chipset_Software\Binaries
C:\AMD3\Chipset_Software\Binaries\GPIO2 Driver
C:\AMD3\Chipset_Software\Binaries\PCI Driver
C:\AMD3\Chipset_Software\Binaries\GPIO2 Driver\WTx64
C:\AMD3\Chipset_Software\Binaries\GPIO2 Driver\WTx64\README.rtf
C:\AMD3\Chipset_Software\Binaries\PCI Driver\WTx64
C:\AMD3\Chipset_Software\Binaries\PCI Driver\WTx64\ReadMe.rtf

 

[Cyber Akuma]

Huh, this does appear to be exactly what I am looking for. So if I wanted to move the files instead of copy, I would just need to add "/mov" at the end? There seemed to be a mis-match of number of moved files when I attempted it with the move operation.

 

I tried testing this out by making a backup copy of the source root folder to try it out on files with an ID that had 734 files attached to it, ID 257FD939428E4BFE6BF9E2F559D5037A

 

I tried "robocopy e:\SourceBackup e:\TargetCopy *-257FD939428E4BFE6BF9E2F559D5037A* /s"

 

And it resulted in TargetCopy having 734 files in it, with the directory structure of just the copied files intact. Exactly what I was looking to do. 

 

So then I tried "robocopy e:\SourceBackup e:\TargetMove *-257FD939428E4BFE6BF9E2F559D5037A* /s /mov"

 

And it resulted in the same 734 files in TargetMove, however, I noticed that SourceBackup now had 759 less files, not 734. Any idea why that is?

 

 

[ZakO]

Not sure why that would be, I've used robocopy /mov before and never noticed any discrepancies like that.

 

Did the backup source before running the command definitely have the same amount as the original source, it didn't miss any hidden ones like desktop.ini or thumbs.db? It would be interesting to copy the files from e:\targetMove back into e:\sourceBackup (via explorer) and then run a recursive folder diff (WinMerge) to tell you what the missing 25 files are, that would give a better picture of what has happened

Edited March 26, 2022 by ZakO

[Cyber Akuma]

On 26/03/2022 at 17:09, ZakO said:

Did the backup source before running the command definitely have the same amount as the original source, it didn't miss any hidden ones like desktop.ini or thumbs.db?

Well I feel stupid, I forgot there was an additional folder that was not in the backup since it was an extra folder that didn't have any files I wanted to move, it has exactly 25 files in it now that I checked.

[ZakO]

On 26/03/2022 at 22:22, Cyber Akuma said:

Well I feel stupid, I forgot there was an additional folder that was not in the backup since it was an extra folder that didn't have any files I wanted to move, it has exactly 25 files in it now that I checked.

😁The most obvious things are the things we always forget. Good to hear it wasn't robocopy messing up, had me worried for a minute there, I use robocopy often and never normally check the resulting file count!

[Cyber Akuma]

Yeah, just to be safe I tried cloning the original source folder again with those additional files intact. I verified they both had the same number of files and folders, then did the move again and this time there was exactly 734 files missing. Seems like I can do exactly what I wanted with just one command from Robocopy. Thanks.