In the last post we looked at key use cases for server-to-server file transfer, where existing solutions tend to fall down, and the evolving requirements that support IT’s need to both maintain a reliable data infrastructure and respond quickly to new business initiatives.

In this post, we’ll look briefly at the modern technology in managed file transfer solutions that drive the ability to meet those evolving requirements.

Underlying the MFT solutions that support reliable and automated point-to-point transfers between servers are a core set of capabilities that are missing in older file transfer technologies.  These are:

• Success Notification: The ability to know definitively that a file has successfully completed a transfer, and when.
• Failure alerts with details: If a problem occurs that interrupts the transfer, when it happens, and what the root cause is likely to be.
• Declarative Automation:  A configuration-driven method (versus complex scripting) for defining and executing file transfers and the associated steps of initiating a transfer, error recovery, and post-processing of the files after they’re transferred.
• Post-processing Actions: The ability to identify commands, scripts, and batches that need to be run from the context of the target system (or the source) following a successful transfer.
• Event-driven initiation: The ability to monitor directories and initiate pre-processing activities and file transfers on the basis of a file showing up in those directories.

Some of these capabilities are often achieved by enhanced (and yes, proprietary) protocols that perform extra steps at the beginning and end of a file transfer.  These allow the transfer endpoints to negotiate the elements of the transfer up front (valid security credentials, target storage definition requirements, post-processing actions that will be run at the end of the transfer, etc.) and perform a handshake at the end of the transfer (Sender, “I sent XXXX bytes.” – Receiver, “Great because I received XXXX bytes.”  or Receiver, “Uh-oh, I only received XXXW bytes, better try again.”).

These capabilities are the building blocks then that are combined to drive real value in to mission-critical server-to-server file transfer scenarios.  Advanced MFT solutions (like Attachmate FileXpress) hit the key areas:

Reliability:

• Native platform support – Windows, UNIX, Linux , IBM i, IBM z/OS
• Error alerting – Send an email or an SNMP trap as soon as a problem occurs
• Error details – Information on the source and location of the error and whether it’s recoverable or not (i.e. insufficient disk space – recoverable, failed user authentication – unrecoverable)
• Automatic retries – Have the MFT software automatically reattempt a recoverable failed transfer after a prescribed waiting period.
• Recoverable work queues – Don’t lose file transfers in a system crash. After system recovery, automatically rebuild the list of file transfers that are queued up to execute.

 Simplified Automation:

• Initiation – The MFT software will start the file transfer process as soon as a user, or script, or application drops a file into a monitored directory
• After the Transfer – The MFT software on the end points will kick off the next step of the process after the transfer is finished.  No scripting or kludgy flag file techniques needed to determine that the transfer is finished.
• Reusability – Configuration-driven file transfer definitions that can be used as templates for future file transfers.

Putting it all together, with MFT solutions server-to-server file transfers become more dependable and automation becomes simpler and quicker to implement.  As such, MFT is helping IT organizations maintain a reliable data infrastructure and respond quickly to new business initiatives.

5 Capabilities Enhancing Server-to-Server File Transfer is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post 5 Capabilities Enhancing Server-to-Server File Transfer appeared first on Data In Motion - a Managed File Transfer blog.

Routine, scheduled and automated file transfers between servers occur in a wide range of use cases in just about every organization. In this post, we will begin a two part series and look at those use cases and the top five factors compelling enhanced server-to-server file transfer.

In the next blog post we will look at the technology requirements to improve server-to server file transfer.

The use cases for automated server-to-server file transfers include: 

• The movement of bulk/batch data from one application to another
• The deployment of web application updates from a staging system to production systems
• The transfer of end-of-day financial and/or inventory data from branch locations into systems hosted in the corporate data center

There are of course many additional uses in any organization.  These are simply some of the more common and generic examples.

5 Factors Impacting Server-to-Server File Transfer

Critical business functions are dependent on these file transfers working consistently, so a reliable and reusable framework for the movement of data between servers within an organization supports the key IT objectives of maintaining a reliable data infrastructure and responding quickly to new business initiatives. 

When server-to-server file transfers fail to support these objectives, it’s typically due to one or more of the 5 following factors: 

• Size/Volume Growth – File sizes and file transfer volumes have grown too a level that’s challenging for existing systems to handle reliably and efficiently.
• Too many solutions – There are a variety of disparate file transfer technologies implemented across the organization’s server-to-server file transfer use cases; there are too many different ways to do the same thing.
• Lack of file transfer governance– Aggregated auditing and reporting needs to be established for all sensitive or regulated data moving within the organization.  See this blog post on file transfer governance.
• High manual effort – Certain file transfers are too critical to leave unmonitored.  Existing technologies can’t provide the reliability and visibility to the critical transfers, requiring IT staff to babysit key transfers.
• Expensive Automation/Integration – The lack of ROI from the effort to automate critical server-to-server file transfers because existing technologies fail to do enough of the work needed for reliability, error recovery and post-transfer activity.  This means that you’re required to code this logic into your scripts and applications, increasing their complexity and making sustainability of those scripts/applications more challenging. 

3 Requirements for Enhanced Server-to-Server File Transfers

Managed File Transfer systems that are optimized for server-to-server file transfer use cases (not all are) support the following requirements that overcome these challenges. 

• Reliably accommodate large files and scale to large volumes.
• Provide a powerful and easy-to-consume interface for file transfer and administrative functions to system administrators and application developers.
• Deliver the visibility necessary to quickly resolve outages, audit the movement of sensitive files and generate reports that document compliance in your handling of regulated data. 

Supporting these requirements is essential in successful and secure server-to-server file transfers that maximize the value of automating routine but vital data transfer.

The next blog post delves into the technical capabilities of advanced MFT solutions that support the above requirements.

5 factors compelling enhanced server-to-server file transfer is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post 5 factors compelling enhanced server-to-server file transfer appeared first on Data In Motion - a Managed File Transfer blog.

“File Transfer Governance” isn’t a term you hear very often.  But it relates to the kinds of rules you deal with in everyday life.

Rules are guiding principles for everything. There are rules to govern the way you conduct yourself at your job, at the gym, as you participate in hobbies and as a citizen. As you go about life, you ask yourself questions to determine if you’re complying with the rules.

The same is true in business. In respect to transferring files, it is critical that organizations are in compliance with the standards – or rules – of their company and any industry requirements. File transfer governance is about monitoring compliance with the policies your organization establishes that enforce those standards. Of course those policies and standards need to be in place first.  Is there a policy establishing the use of a standard for a secure file transfer protocol? Has that secure file transfer protocol standard first been established?

File Transfer Governance Questions to Ask Yourself

We thought it would be helpful to provide you with some quick questions to ask yourself to see where file transfer governance is relevant to your organization and if a managed file transfer (MFT) solution could be an option to help.

1. Is there a corporate policy around encrypting sensitive data during transfer? Do you have procedures in place to ensure compliance with this policy?
• MFT solutions support the secure file transfer protocols that encrypt files in transit. Additionally, they can bridge between insecure and secure protocols making a phased transition possible. MFT solutions often serve as a concentration point for external file transfers, providing file transfer governance by ensuring that no transfer is allowed to occur over an insecure protocol.
2. Is your organization and its employees using multiple solutions for file transfers?
• If yes, then it’s likely each solution has its own approach to configuring security and file transfer governance capabilities, and your team has to learn each one. Consider streamlining your organization’s file transfers with one MFT solution. This can help reduce cost, simplify file transfer governance, and keep the process consistent.
3. Do you leverage “home-grown” applications built on FTP to transfer files?
• This is not uncommon. If yes, look at the potential costs and problems associated with these solutions. Homegrown solutions are typically quite tactical and lack the file transfer governance capabilities built into commercial solutions. Using the insecure FTP protocol, your information could be at risk. This can open the company up to regulatory fines, damage to its image and an increase in business costs when trying to frantically remedy these vulnerabilities as part of a “fire drill” project.
4. What regulatory requirements apply to your business as it relates to protecting data and breach notification? Any international regulations? Does your current solution support compliance with these requirements?
• If your current solution does not support your security policies, then it will be hard-pressed to facilitate compliance with regulatory requirements faced by your organization. Take auditing for instance.  Keeping a record of the secure movement of sensitive files and being able to report in aggregate on file transfer activity over a period of time is a key security control prescribed by file transfer governance and delivered in MFT solutions.

Simplify File Transfer Governance

By leveraging a robust file transfer solution, such as FileXpress, you simplify file transfer governance through such things as consistency, secure protocols and centralized auditing. To help you search for an MFT solution, check out our previous blog post on research tools.

File Transfer Governance? What’s that? is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post File Transfer Governance? What’s that? appeared first on Data In Motion - a Managed File Transfer blog.

I can hear the movie trailer right now, “In a world where files are growing larger, one hero sets out on the quest for reliability.”

Ok, pretty goofy I know.  It’s just large file transfer.

The reality is some electronic files are large.  Sometimes these large files need to be transferred.  Very often, the larger the file, the more challenging it is to get that large file transfer to happen over a network in a way that’s complete, accurate and on time.

When this is the case, many times individuals find themselves facilitating the large file transfer by copying the file to a physical medium (DVD, USB drive, etc.) and physically shipping that medium to the intended recipient. This approach has its own set of reliability and security challenges.

What determines a large file transfer?  It really is subject to the use case.  File transfers between individuals by means of email attachments will often become “large” when the files attached are larger than 10Mb.  Batch data coming off of a mainframe can be in the hundreds of megabytes.  Media files and virtual machine images are often in the multiple gigabyte range.

In a Managed File Transfer approach to large file transfer, certain technologies that drive up the reliability of file transfers of all sizes come into play.  This collective set of capabilities is often referred to as “guaranteed delivery.” These technologies are focused on responding to problems when they arise, and include:

• Automatic Retries – If the file transfer fails, the client knows that an error is one that can be recovered from, it can reattempt a file transfer after a pre-defined waiting period.
• Alerts – Communication over a variety of channels, such as emails, text messages and SNMP traps designed to bring attention to the fact that a problem has arisen and provide detail on the nature of the large file transfer issue.
• Checkpoint/Restart – Checkpoint control packets tell the sending client and receiving server how much of the file has been transferred so far.  In the event of a failed transfer, the large file transfer can resume at the last checkpoint, rather than restart at the beginning of the file.
• Email integration – Tools for leveraging reliable large file transfer capabilities into a user’s email experience, without making the user significantly change the way they currently send smaller files through email.

Comprehensive managed file transfer solutions such as Attachmate FileXpress support large file transfers across a wide variety of use cases, from user-driven file transfers through email, to bulk data movement between IBM mainframes and distributed systems inside of a data center. 

Image courtesy of clipartheaven.com

If you’re on a quest for reliability for your large file transfers, investigate managed file transfer solutions and the capabilities they bring to your particular use case. Who knows, you might just  become your organization’s hero.

Large File Transfer, the Quest for Reliability is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post Large File Transfer, the Quest for Reliability appeared first on Data In Motion - a Managed File Transfer blog.

Acronym Alphabet Soup (AAS)

Don’t you get tired of acronyms? 

Naw, me neither.  I love them.  

I remember the first time I was able to rattle of “ODBC API” really fast, without tripping over my own tongue (yes, it was a long time ago).  So satisfying.  

Anyway… 

Acronyms can be quite useful…once you know what they stand for.  In the world of managed file transfer (MFT) we use a lot of acronyms.  Terms like U2U, B2B, A2A, and so on pop up in our conversations and literature all of the time.  You’d think we’d get a clue and figure out that not everyone has been speaking in file-transfer-ese all their life. 

And now we have!  (got a clue, that is).  

The video below describes at a high level the common business file transfer use cases you’ll find out there, and what is challenging their ability to operate today in the same way they were able to yesterday.  The video also spells out (bad pun) some of those pesky acronyms and gives them context.  You’ll also find it provides a framework for you to reason out where new requirements may be impacting how your organization is moving files; between users, between businesses and between applications. 

Now you too can be an acronym-slinging MFT insider.   Pretty cool, huh?

U2U? B2B? A2A? Won’t someone start using actual words? is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post U2U? B2B? A2A? Won’t someone start using actual words? appeared first on Data In Motion - a Managed File Transfer blog.

I’ve always heard stories about restaurants that throw challenges at their patrons claiming, “if you eat the whole thing, you don’t have to pay for it.” 

I’ve most recently heard about the “World Famous” 72-ounce (about 2 kilograms) steak dinner that one can get for free (if eaten in 1 hour).  This feast is offered by the Big Texan Steak Ranch in Amarillo, Texas.

To quote their web site, “Many have tried.  Many have failed.”

Just as 72-ounce steaks will provide a significant challenge to the gastro-intestinal systems of the average human being, so too do large files when one attempts to transfer them via legacy file transfer systems.

One of the most pervasive file transfer methods in use is good old FTP.  FTP, which stands for “file transfer protocol” has its roots in specifications that go back as far as early 1971.  While some may argue that the FTP protocol itself has nothing inherently in it that prohibits the transfer of very large files, these large files often present a real challenge when transferred between FTP clients and servers.  Just search on the Internet for “FTP 4Gb”.  The problems frequently lay instead with design limitations of the software implementing the FTP protocol (often old clients or servers, written long ago), or the platforms on which the software is running.

The SFTP transfer protocol, an extension to the secure shell (SSH) protocol provides for more reliability while also adding security.  It does this by means of a message authentication code (MAC) that is computed for each TCP/IP packet.  This provides for data integrity which helps ensure that the file is transferred with complete accuracy and isn’t corrupted (or intentionally altered) in transit.

As managed file transfer solutions have evolved, more proprietary protocols have been developed in the interest of driving additional reliability into large file transfers.  Some of these protocols implement key reliability capabilities:

• Notification that a file transfer has been accurately completed
• Detailed feedback on errors and their causes
• Checkpoint control packets

Out of these capabilities, implemented in the proprietary protocol, file transfer recovery features have been designed that really drive reliability into the file transfer process.  These include:

• Automatic Retries – if the file transfer client knows that an error is one that can be recovered from, it can reattempt a file transfer after a pre-defined waiting period.
• Checkpoint/Restart – Checkpoint control packets tell the sender and receiver how much of the file has been transferred so far.  In the event of a failed transfer, the file transfer can resume at the last checkpoint, rather than restart at the beginning of the file.

In addition to the reliability benefits gained through proprietary protocols, some managed file transfer solutions implement the concept of a work queue in which file transfers are lined up, waiting for their chance to execute.  Work queue recovery is the process by which those jobs are retained in the event of a system crash and loaded back into the queue upon system restart.

Maybe you only push out a 4Gb file with the same frequency that you would eat a 72-ounce steak.  In other cases, you may be experiencing reliability issues on much smaller files.  Regardless, having the added reliability of a managed file transfer solution that supports both open and proprietary protocols will make a big difference when the file has to get there, complete and unaltered.

“I can’t believe I ate the whole thing!” is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post “I can’t believe I ate the whole thing!” appeared first on Data In Motion - a Managed File Transfer blog.