未加星标

Powershell Duplicate file Cleanup for Plex Camera Uploads

字体大小 | |
[系统(windows) 所属分类 系统(windows) | 发布者 店小二03 | 时间 2018 | 作者 红领巾 ] 0人收藏点击收藏

Some time ago, my wife had her phone stolen and we had not setup any sort of backup for the pictures, so a good number of photos and videos of our kids were lost that day. We now both use Google devices, so we have automatic backup to Drive for free, but I didn’t want to rely only on that.

Enter Plex Camera Upload

We use Plex a lot in our house one of the major features we use is the automatic upload of photos to a library this library is shared with parents and in-laws and it also provides a simple way to backup photos or videos taken by our phones. Initially I had some issues with the feature since I was storing files on an SMB share mounted to linux and when that happened, I found I needed to restart the upload to get things working.

Since then, things have just worked. Even switching to new devices, once the Plex app is installed and configured for upload, it just works. Until it doesn’t. A recent update broke the upload feature.

It has since been fixed, but in the meantime, I went back and disabled\reset the camera upload capability to try to get it working. Needless to say, I have some duplicate files in my Camera Uploads directory. I think there is some validation in the Plex app for previously updated files, but not much. I had racked up about 4000+ duplicate files and these are not small files, so there was several GB of space savings to be reclaimed. If only there were an automated way to do this…

Powershell to the Rescue

The following script will do the following:

Prompt the user for a path to the Camera Uploads directory Parse all of the image and video files in the directory and group them by file size where each group has more than 1 file Generate a SHA1 hash of each of the files to guarantee that they are actually duplicates Delete hyphenated or malformed date files

A few notes:

I found -1, -2 and -3 duplicate files, there could potentially be more in your case, but the script would need to be modified to accommodate for this Nothing is deleted unless you pass the -delete parameter to the script and confirm The directory selection window likes to pop up behind the ISE window…making it look like the script is hung check the log file for activity in your MyDocs <# .SYNOPSIS Script to remove duplicate photos from the Plex Mobile Uploads directory .DESCRIPTION This script will enumerate all files in the Mobile Uploads directory, then find files that are the same size. A SHA1 hash is generated for each of these files to verify that they are the same image prior to being deleted. Files that have incorrect date formats (1970-01-01) or are hyphenated take precedence to be deleted, unless all duplicate files are formatted incorrectly, then the first file found to be duplicate will be deleted. .NOTES Script only looks for -1, -2 or -3 JPG files or -1 and -2 MP4 files More file types can be added to the Get-ChileItem -Include section if needed .PARAMETER delete If the 'delete' parameter is used, the script will delete files once hashed duplicates have been found #> param( [parameter(Mandatory=$false)] [switch]$delete=$false ) $logFile = "$([Environment]::GetFolderPath("mydocuments"))\DupeDeleteLog.txt" # Write to a log file in the current users MyDocs directory Function Write-Log { Param([string]$logStr) Add-Content $logFile -Value $logStr } # User select location to find duplicate files Function Get-FolderName { Add-Type -AssemblyName System.windows.Forms $FolderBrowser = New-Object System.Windows.Forms.FolderBrowserDialog [void]$FolderBrowser.ShowDialog() $FolderBrowser.SelectedPath } Write-Log "$(Get-Date)" Write-Log "Waiting for input from user to select working directory..." $mypath = Get-FolderName # Initial log file notes If($delete){Write-Log "WARNING: File deletion will occur!"} Write-Log "Checking for duplicates in $($mypath)" Write-Log "Finding same size files and hashing them. This will take some time...." $dupeHash = foreach ($i in (Get-ChildItem -path $mypath -Recurse -Include "*.jpg","*.jpeg","*.mp4" | ? {( ! $_.ISPScontainer)} | Group Length | ? {$_.Count -gt 1} | Select -Expand Group | Select FullName, Length)){Get-FileHash -Path $i.Fullname -Algorithm SHA1} $dupeHashGrouped = $dupeHash | Group Hash | ? {$_.Count -gt 1} # Set confirm, if $delete param is not set, no delete will occur If(!($delete)){$delConfirm = "Y"} If($delete) { Write-Log "Waiting for input from user to confirm delete..." $delConfirm = Read-Host "WARNING: This script will now DELETE FILES. Press 'Y' to confirm you want to DELETE, or any other key to cancel" } If (($delConfirm -eq 'y') -or ($delConfirm -eq 'Y')) { foreach ($dhGroup in $dupeHashGrouped) { Write-Log "Current file hash: $($dhGroup.Group[0].Hash)" Write-Log "Matching files: $($dhGroup.Count)" $goodFile = $false $i = 1 foreach($matchFile in $dhGroup.Group.Path) { If($i -eq $dhGroup.Count -and (!($goodFile))) { #Last file, no good found...keeping Write-Log "Last file in group. Keeping: $($matchFile)" } ElseIf($matchFile -like "*1970*") { Write-Log "File in group contains invalid date 1970: $($matchFile) - will be deleted." If($delete){Remove-Item -Path $matchFile -Confirm:$false} } #modify if more duplicated hyphens are found ElseIf($matchFile -like "*-1.jpg" -or $matchFile -like "*-2.jpg" -or $matchFile -like "*-3.jpg" -or $matchFile -like "*-1.mp4" -or $matchFile -like "*-2.mp4") { Write-Log "File in group is hyphenated: $($matchFile) - will be deleted." If($delete){Remove-Item -Path $matchFile -Confirm:$false} } Else { If($goodFile) { #Already found valid file in group Write-Log "Deleting this file: $($matchFile) already have good file." If($delete){Remove-Item -Path $matchFile -Confirm:$false} } Else { #Don't have good file in group; keep Write-Log "Keeping: $($matchFile)" $goodFile = $true } } $i++ } Write-Log " " } } Else { Write-Log "Deletion cancelled by user" }

The script will also be posted in my PowerShell library here.

本文系统(windows)相关术语:三级网络技术 计算机三级网络技术 网络技术基础 计算机网络技术

tags: files,Log,file,matchFile,Write,will
分页:12
转载请注明
本文标题:Powershell Duplicate file Cleanup for Plex Camera Uploads
本站链接:https://www.codesec.net/view/604429.html


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(windows) | 评论(0) | 阅读(28)