File System Management

An important aspect of any development is the management of your File System. Without careful management of your File System, you run the risk not being able to locate your files and directories in a consistent manner, running out of space and, ultimately, losing data.

The Talend Framework assists you with your File System management by ensuring that all of your files are stored in predictable locations, assisting with file naming conventions, archiving used files, and pruning aged files.

File System Context Group

The Framework manages the file system through the File System Context Group. This Context Group defines all of the locations that you are likely to need, including locations for you input files, output files, control files, library files, log files, and many other common locations.

For an introduction in to configuring the File System Context Group, read this article on the Base Directory. This is, often, the only value that you will need to change.

Initialisation

Each time you run a framework Job, it will initialise the File System; that is, it will ensure that the minimum required directories and files exist. If they do not exist, they will be created.

Archiving

On completion of your Job, you may wish to archive certain files and directories. The framework will assist you with this by helping you to move these files to an archive directory.

For more information on archiving, read this article on Job: LibArchive.

File System Pruning

As part of archiving, the framework is also able to automatically prune aged files and directories. Pruning is the physical deletion of files and directories that are older than a specified number of days. You may specify this age, individually, at Job level, for each of your archived directories.

For more information, read this article on File System Pruning.