Picture this scenario: Your organization has a signed contractual obligation to provide a set of services  to a business partner.  As a part of those services, you must deliver a file to your business partner no later than  4 a.m. on Monday morning. The file is produced by a batch process that is generated over the weekend and then copied to your partner using a file transfer script. Monday morning arrives and you receive an angry phone call that the file never made it to your business partner, and now they are threatening to apply the penalties section of your service level agreement.

Nobody wants that phone call. Unfortunately, that situation is possible when using file transfer protocol (FTP).

Organizations using FTP are often drawn in by the allure of an easy, familiar and inexpensive solution. But there are FTP risks.  As we prepare to close down another year and ring in 2013, it is a good time to evaluate the past year and look for organizational improvements.

We’ve identified the top four FTP risks of 2012 below:

• Reliability. FTP has been seen to have challenges with large files.  Additionally, there is no provision in FTP for determining that an error has occurred in transmission, and alerting an operator that something needs to be done.   A more risk-averse approach to mission-critical file transfers is to use a Managed File Transfer (MFT) solution, which can trigger an alert if a problem arises and even retry the file transfer automatically.
• Security. FTP has no encryption. This means that the data in the transferred files is put at risk and open to network sniffing.  Beyond that, your login credentials (username and password) are also visible to a network sniffer. Imagine the risk you incur if administrative level access to your FTP server becomes available to malicious operator. By contrast, MFT solutions encrypt files as they are sent and keep login information protected.
• Automation. FTP users run the risk of compromising login information when trying to automate the file transfer process. Automating file transfers with FTP is script-based. The system administrator has to write a script that talks to the FTP server, specifies what files to send, and where they should go on the server; however for simplicity, systems administrators will frequently write an FTP username and password into a script. This creates a notable FTP risk. People sharing scripts are also sharing user credentials.  This leaves an organization vulnerable. MFT solutions significantly reduce the need for scripts and offer built-in capabilities for automation. For example, the initiation of the file transfer, the confirmation of the file transfer, the alerting if there are problems with the file transfer, are all done without scripting.
• Auditing and Logging. Another FTP risk comes through in auditing and logging. The challenge with FTP is that it does not provide the level of auditing that organizations need to document compliance with data security regulations. It also does not allow administrators to easily identify when a problem has occurred and help resolve that problem. MFT will help you confirm that a transfer went through and log the details necessary to identify the problem when it doesn’t. This will help you avoid situations like the one described above.

Investing in a more robust solution in the new year could save you and your organization time, money and headaches by avoiding these FTP risks.

What other FTP risks have you identified?

Top FTP Risks of 2012 is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post Top FTP Risks of 2012 appeared first on Data In Motion - a Managed File Transfer blog.

In the last post we looked at key use cases for server-to-server file transfer, where existing solutions tend to fall down, and the evolving requirements that support IT’s need to both maintain a reliable data infrastructure and respond quickly to new business initiatives.

In this post, we’ll look briefly at the modern technology in managed file transfer solutions that drive the ability to meet those evolving requirements.

Underlying the MFT solutions that support reliable and automated point-to-point transfers between servers are a core set of capabilities that are missing in older file transfer technologies.  These are:

• Success Notification: The ability to know definitively that a file has successfully completed a transfer, and when.
• Failure alerts with details: If a problem occurs that interrupts the transfer, when it happens, and what the root cause is likely to be.
• Declarative Automation:  A configuration-driven method (versus complex scripting) for defining and executing file transfers and the associated steps of initiating a transfer, error recovery, and post-processing of the files after they’re transferred.
• Post-processing Actions: The ability to identify commands, scripts, and batches that need to be run from the context of the target system (or the source) following a successful transfer.
• Event-driven initiation: The ability to monitor directories and initiate pre-processing activities and file transfers on the basis of a file showing up in those directories.

Some of these capabilities are often achieved by enhanced (and yes, proprietary) protocols that perform extra steps at the beginning and end of a file transfer.  These allow the transfer endpoints to negotiate the elements of the transfer up front (valid security credentials, target storage definition requirements, post-processing actions that will be run at the end of the transfer, etc.) and perform a handshake at the end of the transfer (Sender, “I sent XXXX bytes.” – Receiver, “Great because I received XXXX bytes.”  or Receiver, “Uh-oh, I only received XXXW bytes, better try again.”).

These capabilities are the building blocks then that are combined to drive real value in to mission-critical server-to-server file transfer scenarios.  Advanced MFT solutions (like Attachmate FileXpress) hit the key areas:

Reliability:

• Native platform support – Windows, UNIX, Linux , IBM i, IBM z/OS
• Error alerting – Send an email or an SNMP trap as soon as a problem occurs
• Error details – Information on the source and location of the error and whether it’s recoverable or not (i.e. insufficient disk space – recoverable, failed user authentication – unrecoverable)
• Automatic retries – Have the MFT software automatically reattempt a recoverable failed transfer after a prescribed waiting period.
• Recoverable work queues – Don’t lose file transfers in a system crash. After system recovery, automatically rebuild the list of file transfers that are queued up to execute.

 Simplified Automation:

• Initiation – The MFT software will start the file transfer process as soon as a user, or script, or application drops a file into a monitored directory
• After the Transfer – The MFT software on the end points will kick off the next step of the process after the transfer is finished.  No scripting or kludgy flag file techniques needed to determine that the transfer is finished.
• Reusability – Configuration-driven file transfer definitions that can be used as templates for future file transfers.

Putting it all together, with MFT solutions server-to-server file transfers become more dependable and automation becomes simpler and quicker to implement.  As such, MFT is helping IT organizations maintain a reliable data infrastructure and respond quickly to new business initiatives.

5 Capabilities Enhancing Server-to-Server File Transfer is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post 5 Capabilities Enhancing Server-to-Server File Transfer appeared first on Data In Motion - a Managed File Transfer blog.

Routine, scheduled and automated file transfers between servers occur in a wide range of use cases in just about every organization. In this post, we will begin a two part series and look at those use cases and the top five factors compelling enhanced server-to-server file transfer.

In the next blog post we will look at the technology requirements to improve server-to server file transfer.

The use cases for automated server-to-server file transfers include: 

• The movement of bulk/batch data from one application to another
• The deployment of web application updates from a staging system to production systems
• The transfer of end-of-day financial and/or inventory data from branch locations into systems hosted in the corporate data center

There are of course many additional uses in any organization.  These are simply some of the more common and generic examples.

5 Factors Impacting Server-to-Server File Transfer

Critical business functions are dependent on these file transfers working consistently, so a reliable and reusable framework for the movement of data between servers within an organization supports the key IT objectives of maintaining a reliable data infrastructure and responding quickly to new business initiatives. 

When server-to-server file transfers fail to support these objectives, it’s typically due to one or more of the 5 following factors: 

• Size/Volume Growth – File sizes and file transfer volumes have grown too a level that’s challenging for existing systems to handle reliably and efficiently.
• Too many solutions – There are a variety of disparate file transfer technologies implemented across the organization’s server-to-server file transfer use cases; there are too many different ways to do the same thing.
• Lack of file transfer governance– Aggregated auditing and reporting needs to be established for all sensitive or regulated data moving within the organization.  See this blog post on file transfer governance.
• High manual effort – Certain file transfers are too critical to leave unmonitored.  Existing technologies can’t provide the reliability and visibility to the critical transfers, requiring IT staff to babysit key transfers.
• Expensive Automation/Integration – The lack of ROI from the effort to automate critical server-to-server file transfers because existing technologies fail to do enough of the work needed for reliability, error recovery and post-transfer activity.  This means that you’re required to code this logic into your scripts and applications, increasing their complexity and making sustainability of those scripts/applications more challenging. 

3 Requirements for Enhanced Server-to-Server File Transfers

Managed File Transfer systems that are optimized for server-to-server file transfer use cases (not all are) support the following requirements that overcome these challenges. 

• Reliably accommodate large files and scale to large volumes.
• Provide a powerful and easy-to-consume interface for file transfer and administrative functions to system administrators and application developers.
• Deliver the visibility necessary to quickly resolve outages, audit the movement of sensitive files and generate reports that document compliance in your handling of regulated data. 

Supporting these requirements is essential in successful and secure server-to-server file transfers that maximize the value of automating routine but vital data transfer.

The next blog post delves into the technical capabilities of advanced MFT solutions that support the above requirements.

5 factors compelling enhanced server-to-server file transfer is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post 5 factors compelling enhanced server-to-server file transfer appeared first on Data In Motion - a Managed File Transfer blog.

Failure is not a term sought after by most. People do not start a project with the intention to fail, but rather to succeed. The same is true for the key file transfers that keep your business moving. Nobody wants to be set up for failure; however, with the wrong tools in place, failure is quite possible.

We’ve come up with the top three classic file transfer “fails” to show what you could be risking.

1. My customer knew the file transfer failed before I did. In interactions and transactions with clients, there is an expectation that promised files will be delivered in a timely manner.  When a customer notices that a file did not make it before the delivering organization does, it can create tension and questions about reliability. Additionally, with FTP (the basis for many home-grown file transfer solutions) there is no alert that a file did not reach its intended destination. So while FTP may be quick and easy to implement, it doesn’t protect you from the potential embarrassment and doesn’t provide a failsafe system that alerts you of an error.
2. My email attachment is too big. The number one file transfer fail that we hear time and time again is an  email attachment that has been blocked from being sent or received because it is too large. This is the way that the majority of people send documents to other people; inside and outside of their organization. Not only is the file transmission not secure, but there is no way to know that the file made it through your email and into your intended target’s inbox. Email attachments also take up inbox space.  “Your inbox is full” messages are so annoying.
3. My IT staff is spending too much time involved manually managing file transfers. When there are multiple file transfer solutions within an organization, it requires a great deal of maintenance. Custom scripts for each solution are hard to maintain and hard to support. This is increasingly difficult as an organization grows in size and the demand increases. Provisioning tasks are often manual, requiring a great deal of time and effort. Wasting resources is a big fail, when resources could be allocated to more pressing tasks.

The good news is these classic fails don’t have to happen to you! Your organization can avoid failure and be prepared for success by implementing a managed file transfer solution, like those offered by Attachmate. Read our previous blog post on tools for evaluating your solution.

Classic File Transfer Fails is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post Classic File Transfer Fails appeared first on Data In Motion - a Managed File Transfer blog.

Is your washing machine giving you signs that it's ready to be replaced? How about your processes for transferring files?

Before you decide it’s time for a new home appliance, there are usually signs that indicate it’s time to bring in the new and recycle the old. Your washer and dryer aren’t what they used to be. From funny noises, to clothes left stained and damp, the signs are hard to ignore. The same is true for developing a file transfer strategy. There are many indicators that illustrate the need for a new approach. We have highlighted three examples that showcase this need.  

1. The most classic symptom that indicates the need for a file transfer strategy is when you have a problem sending a large email attachment. If you are sending a large file to someone at another organization, you want to ensure that the document was received. But what if the attachment is too large for an email system? Even if the important file clears your email, it may not make it through the intended recipient’s email system and there is no way to know if the file was stalled. You need a reliable file transfer method that allows you to easily send even large files to other users.
2. Another indication that it is time for a file transfer strategy is when there is no standard for file transfers in the organization. If everyone is sending files in a different way, it is confusing and makes it difficult to integrate solutions.  A single transfer strategy could streamline the process across your organization, especially if you select a solution that is scalable and flexible to your growing business needs. Also, by investing in a single file transfer solution, you’ll help cut costs by dealing with fewer vendors and reducing the number of file transfer servers you use to move files into and out of the organization. 
3. Finally, if you are using non-secured file transfer protocols, your data is at risk. The information you are transferring could be compromised and there is no way to ensure compliance with corporate policy. This could have devastating consequences on an organization and illustrates the need for one, flexible file transfer strategy that can be used across an organization and ensures files are received and information is protected.

If you notice the need for a file transfer strategy across your organization, we would be happy to discuss the benefits of FileXpress. Please contact us here.

What led you to explore a file transfer strategy?

How You Know it is Time for a File Transfer Strategy is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post How You Know it is Time for a File Transfer Strategy appeared first on Data In Motion - a Managed File Transfer blog.

I can hear the movie trailer right now, “In a world where files are growing larger, one hero sets out on the quest for reliability.”

Ok, pretty goofy I know.  It’s just large file transfer.

The reality is some electronic files are large.  Sometimes these large files need to be transferred.  Very often, the larger the file, the more challenging it is to get that large file transfer to happen over a network in a way that’s complete, accurate and on time.

When this is the case, many times individuals find themselves facilitating the large file transfer by copying the file to a physical medium (DVD, USB drive, etc.) and physically shipping that medium to the intended recipient. This approach has its own set of reliability and security challenges.

What determines a large file transfer?  It really is subject to the use case.  File transfers between individuals by means of email attachments will often become “large” when the files attached are larger than 10Mb.  Batch data coming off of a mainframe can be in the hundreds of megabytes.  Media files and virtual machine images are often in the multiple gigabyte range.

In a Managed File Transfer approach to large file transfer, certain technologies that drive up the reliability of file transfers of all sizes come into play.  This collective set of capabilities is often referred to as “guaranteed delivery.” These technologies are focused on responding to problems when they arise, and include:

• Automatic Retries – If the file transfer fails, the client knows that an error is one that can be recovered from, it can reattempt a file transfer after a pre-defined waiting period.
• Alerts – Communication over a variety of channels, such as emails, text messages and SNMP traps designed to bring attention to the fact that a problem has arisen and provide detail on the nature of the large file transfer issue.
• Checkpoint/Restart – Checkpoint control packets tell the sending client and receiving server how much of the file has been transferred so far.  In the event of a failed transfer, the large file transfer can resume at the last checkpoint, rather than restart at the beginning of the file.
• Email integration – Tools for leveraging reliable large file transfer capabilities into a user’s email experience, without making the user significantly change the way they currently send smaller files through email.

Comprehensive managed file transfer solutions such as Attachmate FileXpress support large file transfers across a wide variety of use cases, from user-driven file transfers through email, to bulk data movement between IBM mainframes and distributed systems inside of a data center. 

Image courtesy of clipartheaven.com

If you’re on a quest for reliability for your large file transfers, investigate managed file transfer solutions and the capabilities they bring to your particular use case. Who knows, you might just  become your organization’s hero.

Large File Transfer, the Quest for Reliability is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post Large File Transfer, the Quest for Reliability appeared first on Data In Motion - a Managed File Transfer blog.

“Please get that file to me.” I said. “Sure,” he said, “I’ll put it up on my ftp server. You get a client that does ftp to a server, and you’ll be able to pull it down.” Simple and quick? Sure. Secure and scalable?  Definitely not.

It’s kind of like shopping for something big, like a car. As a consumer, when you’re looking to make a large purchase, you make decisions by evaluating your need, ability and objectives. Shopping around is a critical aspect of the process. You wouldn’t buy the first sports car you saw on the lot when you have five children and two dogs, right?

The same is true for selecting the right solution for transferring files within your organization. Many companies recognize the importance file transfer technology plays in their enterprise, and have adopted strategic solutions that meet a broad set of requirements around reliability, scalability and security. In other companies, file transfer is synonymous with “ftp to a server.” The latter may work well for ad hoc transfers within the corporate network between somewhat technical users, but it doesn’t take into account the broader scope of use cases and requirements.

Where does “ftp to a server” play well?  How does that contrast with the capabilities delivered in a more comprehensive managed file transfer (MFT) solution?

Consider the following contrast between FTP and MFT:

Ultimately each organization must prioritize their needs. But I caution each business to carefully evaluate the type of information they are transferring and what makes the most sense. Nobody wants to be left hauling a family of five in a BMW Z3.

Is FTP the Right Solution for Your Organization? is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post Is FTP the Right Solution for Your Organization? appeared first on Data In Motion - a Managed File Transfer blog.

We spend a lot of time in the Data in Motion blog talking about file transfer.  In the context of business, an electronic file can be thought of as a package of data, and a file transfer is that data package in motion between people or systems.  So almost by definition file transfer is data in motion.

Now Attachmate offers a family of file transfer solutions, branded FileXpress, and today Attachmate announced the availability of FileXpress 7.0, our latest release.

As always, this FileXpress 7.0 release is designed to support the core goals of the FileXpress product family, which include helping organizations do the following:

• Exchange files securely
• Transfer files of any size
• Enable file transfer automation
• Simplify file transfer administration
• Improve visibility into file movement activity for support and auditing

These core goals for FileXpress help organizations solve some very important business problems, including:

• How do we exchange sensitive information with our business partners and customers in a manner that is compliant with the data security regulations that govern our industry and the industries of those partners and customers?
• How can we help our resource-constrained IT operations staff work smarter, not harder, when it comes to administering external file transfers and troubleshooting problems with file transfers that happen anywhere within the enterprise?
• How can we help the business run more efficiently and smoothly by automating file transfers that support key business processes, even when the files may be very large or across disparate platforms?

How does the FileXpress 7.0 release support these goals and help solve these business problems?  Well, to quote from the press release:

FileXpress 7.0 helps information security, IT operations and IT architecture teams effectively manage file transfers within organizations by:

• Making Internet File Transfer Secure – FileXpress 7.0 includes critical secure file transfer capabilities including support for a broad range of secure file transfer protocols, detailed file transfer auditing and an architecture that eliminates the need to store sensitive information outside of the corporate firewall in the DMZ.
• Reducing the Burden of an Overtaxed IT Operations Team – FileXpress 7.0 helps manage and troubleshoot both internal and external file transfers through centralized logging with improved search capabilities, new server status alerts that notify when outages occur and more detailed SNMP messages for faster error determination and issue resolution.
• Easily Integrating into Existing IT Infrastructure – FileXpress 7.0 integrates more easily with existing user registry architectures through new capabilities that support the use of a local user account database for external contacts while simultaneously leveraging an existing LDAP user registry for internal users.

If you would like to learn more about FileXpress, you can go to the FileXpress product family web page.

FileXpress 7.0 is out the door is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post FileXpress 7.0 is out the door appeared first on Data In Motion - a Managed File Transfer blog.

I’ve always heard stories about restaurants that throw challenges at their patrons claiming, “if you eat the whole thing, you don’t have to pay for it.” 

I’ve most recently heard about the “World Famous” 72-ounce (about 2 kilograms) steak dinner that one can get for free (if eaten in 1 hour).  This feast is offered by the Big Texan Steak Ranch in Amarillo, Texas.

To quote their web site, “Many have tried.  Many have failed.”

Just as 72-ounce steaks will provide a significant challenge to the gastro-intestinal systems of the average human being, so too do large files when one attempts to transfer them via legacy file transfer systems.

One of the most pervasive file transfer methods in use is good old FTP.  FTP, which stands for “file transfer protocol” has its roots in specifications that go back as far as early 1971.  While some may argue that the FTP protocol itself has nothing inherently in it that prohibits the transfer of very large files, these large files often present a real challenge when transferred between FTP clients and servers.  Just search on the Internet for “FTP 4Gb”.  The problems frequently lay instead with design limitations of the software implementing the FTP protocol (often old clients or servers, written long ago), or the platforms on which the software is running.

The SFTP transfer protocol, an extension to the secure shell (SSH) protocol provides for more reliability while also adding security.  It does this by means of a message authentication code (MAC) that is computed for each TCP/IP packet.  This provides for data integrity which helps ensure that the file is transferred with complete accuracy and isn’t corrupted (or intentionally altered) in transit.

As managed file transfer solutions have evolved, more proprietary protocols have been developed in the interest of driving additional reliability into large file transfers.  Some of these protocols implement key reliability capabilities:

• Notification that a file transfer has been accurately completed
• Detailed feedback on errors and their causes
• Checkpoint control packets

Out of these capabilities, implemented in the proprietary protocol, file transfer recovery features have been designed that really drive reliability into the file transfer process.  These include:

• Automatic Retries – if the file transfer client knows that an error is one that can be recovered from, it can reattempt a file transfer after a pre-defined waiting period.
• Checkpoint/Restart – Checkpoint control packets tell the sender and receiver how much of the file has been transferred so far.  In the event of a failed transfer, the file transfer can resume at the last checkpoint, rather than restart at the beginning of the file.

In addition to the reliability benefits gained through proprietary protocols, some managed file transfer solutions implement the concept of a work queue in which file transfers are lined up, waiting for their chance to execute.  Work queue recovery is the process by which those jobs are retained in the event of a system crash and loaded back into the queue upon system restart.

Maybe you only push out a 4Gb file with the same frequency that you would eat a 72-ounce steak.  In other cases, you may be experiencing reliability issues on much smaller files.  Regardless, having the added reliability of a managed file transfer solution that supports both open and proprietary protocols will make a big difference when the file has to get there, complete and unaltered.

“I can’t believe I ate the whole thing!” is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post “I can’t believe I ate the whole thing!” appeared first on Data In Motion - a Managed File Transfer blog.

This past weekend in the United States, Mother’s Day was celebrated. This special day affords all the opportunity to reflect on, among many other things, the wonderful wisdom imparted over the years from our mothers. In the words of Dr. Benjamin Spock, “I really learned it all from mothers.”

It’s no surprise then, when dealing with the alphabet soup of acronyms that permeate all things IT, one acronym that can’t but help conjure up the sentiment of motherly wisdom is “MOM”.

That acronym has many different meanings, and one such meaning is “Message-Oriented Middleware”.

Countless organizations have recognized the wisdom of implementing a service-oriented architecture with its ability to integrate applications via easily-consumed services. Unfortunately, many SOA strategies are focused on message-oriented systems, and don’t take into consideration another key component of integration, namely file transfer.

In this case, MOM doesn’t always know best.

Message-oriented middleware cannot effectively solve all business integration scenarios. Three common issues related to file transfer that aren’t addressed by message-oriented systems are:

• Support for legacy applications
• Support for large volumes of data and batch oriented systems
• Application integration with partners and customers

Legacy applications know how to generate a file as output for another application to pull in. Then these files are transferred between the servers hosting those applications. The cost and risk of modifying these legacy applications to integrate with a MOM-based solution is often considered too great.

Likewise, many traditional applications, batch up individual transactions and transfer them in bulk to other systems. This can generate very large files. Messaging systems prefer to work at the transactional level with messages of a relatively small size. Batch processes involving large files don’t typically integrate well into a message-oriented system.

Finally, when a business application spans corporate boundaries, most companies will exchange data with their partners or customers by means of electronic files. It is not typically realistic to ask business partners or customers to change the way they conduct business to integrate with your specific messaging implementation.

MOM solutions have their strengths, and those aren’t to be minimized. But today’s SOA needs a way to add file transfer into its infrastructure. That’s what Managed File Transfer (MFT) is designed to do.

Unlike message-oriented systems, human Moms like flowers

MFT drives reliability and security into file movement, and offers services that allow file transfer to participate in a services-oriented architecture.

We love our MOMs, but even they can’t do everything.

Plus you probably shouldn’t send them flowers.

MOM Doesn’t Always Know Best is a post from: Data In Motion - a Managed File Transfer blog

Share and Enjoy

The post MOM Doesn’t Always Know Best appeared first on Data In Motion - a Managed File Transfer blog.