1. Home
  2. Desktop
  3. How to Transfer Millions of Files Using PowerShell

How to Transfer Millions of Files Using PowerShell


I was tasked with moving and archiving approximately 16.5 million files from one location, while ingesting around 4,000 new files per hour. This job was for a single client, and my company has multiple customers with similar needs. Keeping the data organized and available has been a challenge for over a year, and we have been working toward a solution for some time now.


My company takes raw data from the medical industry and converts those metrics into various types of measures, allowing the customer to gauge their performance and setup strategies for improvement. It also allows for several other options, but these core operations are what cause us to take in so many records.

Each record is typically tiny – no more than 100kb, and typically under 50kb in size. Individually, the files are not an issue, but in aggregate, the files cannot be listed before the default timeout is hit for such processes. This is regardless of OS, as the ingest point is a Linux server, and the archive server is Windows-based. In both cases, the files could not be listed using normal methods.

The Problem

This lack of listing made it difficult to determine a logical way to break down the data into usable chunks. There was no consistent naming convention to pull from, and even if there was, there was no way to determine what that might be. It was estimated that there were tens of thousands of files per day, but prior to attacking the problem, that was just an educated guess – there was no way to know beforehand.

Read the entire article here, How to Transfer Millions of Files Using PowerShell

via Devolutions, the Remote Desktop Manager guys

Devolutions Established in 2004, Devolutions is a Canadian-based company located near Montreal, Quebec. The firm currently has over 200,000 users in over 100 countries, and is on a mission to develop innovative enterprise software that helps users cost-effectively, simply and effectively achieve their network management, password management, credentials management and security goals. The firm is also committed to providing exceptional technical support, ensuring an excellent user experience that exceeds expectations, and delivering high performance with superior quality. The firm’s current roster of solutions includes Remote Desktop Manager, Password Vault Manager, Devolutions Server (formerly Remote Desktop Manager Server), and Devolutions Cloud (formerly Remote Desktop Manager Online).

Featured Resources:

Related Articles:


White Papers

‘All You Need to Know About Microsoft Windows Nano Server’ Veeam White Paper

Now updated for Windows Server 2016 GA release! You probably heard about Windows Nano Server already … but what is it exactly, and how do you get started with it? What value will it bring to your environment? Nano Server is a headless, 64-bit only deployment option for Windows Server 2016. Microsoft created this component specifically with […]


Download Commvault VM Backup and Recovery: end-to-end VM backup, recovery and cloud management

Commvault’s ability to provide end-to-end VM backup, recovery and cloud management creates a significantly better way to build, protect and optimize VMs throughout their lifecycle. Our best-in-class software for VM backup, recovery and cloud management delivers a number of significant benefits, including: VM recovery with live recovery options; backup to and in the cloud; custom-fit […]

On-Demand Webinars

Architecting for today’s desktop environments – FSLogix On-Demand Webinar

October 19, 2017 Webinar with David Young, Solutions Architect and Product Champion, and Brandon Lee, Solutions Marketer. Video Recording of a live demo of FSLogix and an overview of the latest release of FSLogix Apps featuring Roaming XenApp Email Search and OneDrive App along with Skype for Business Global Address List and Device Based Licensing. […]

Latest Videos

Current State of EUC – E2EVC Video

Session from @E2EVC 2017 Orlando. For event information please visit www.e2evc.com/home. For slides, additional info etc please contact the presenter directly on Twitter. For best video and sound quality do visit the event! This video is from the fine folks at E2EVC Conference

Views All IT News on DABCC.com
Views All IT Videos on DABCC.com
Win a Tesla P100D

Visit Our Sponsors