Navigation :

Syntax

The Transfer instructions field of a transfer script can contain two different things:

A transfer instruction list, in the format described below. When executed, the script will follow those instructions.
The path of a local file that contains a transfer instruction list, in the format described below. When executed, the script will load the file and follow the instructions present inside.

The transfer instruction list is a CSV file with the following mandatory columns: Action, Source, Destination and Flags.

Path types

To understand the Source and Destination, here are the three types of values that they can take:

A file path always identifies a single file (but not a folder). It can be absolute (begins with / and traces the location of the file or folder from the root of the file tree) or relative (does not begin with / and traces the location from the local or remote root specified in the transfer script configuration). It must not end with a /.

A file path pattern is formatted like a file path, but contains at least one * character. It matches all files such that the complete path of the file fits the pattern, where * can be replaced by zero, one or more characters other than /. Note that all glob patterns are not supported: no ** or ? are allowed. For example, if a file /root/folder/file.csv exists:

Pattern	Matches?
/root/folder/*.csv	yes
/root/*/file.csv	yes
/root/fol*/file.csv	yes
/root/*.csv	no
/*/file.csv	no
/**/file.csv	invalid

A folder path is formatted like a file path, but ends with /. It always identifies a single folder (but not a file).

Note that folder path patterns (containing * but ending with /) are not supported.

The Destination must be a file path or a folder path. The Source is a file path, file path pattern or recursive path.

The Action column determines the meaning of each line, and how the Source and Destination are interpreted.

GET

GET will retrieve all files that match Source on the remote server, and download them to the Destination in the local file tree. The exact behavior depends on the nature of Source and Destination. The allowed combinations are:

Source file path to Destination file path, copies one single file.
```
Action, Source,       Destination,  Flags
GET,    /lokad/a.csv, /input/b.csv, 
```
This downloads the remote file /lokad/a.csv to local file /input/b.csv.
Source file path to Destination folder path, copies one single file and keeps the same file name.
```
Action, Source,       Destination,  Flags
GET,    /lokad/a.csv, /input/,  
```
This downloads the remote file /lokad/a.csv to local file /input/a.csv.
Source file path pattern to Destination file path, fails if the path pattern matches more than one file, and copies that file to the destination.
```
Action, Source,       Destination,  Flags
GET,    /lokad/*.csv, /input/a.csv,
```
This expects to find a single file with the .csv extension in the remote /lokad/ folder, and downloads its contents to local file /input/a.csv
Source file path pattern to Destination folder path, fails if several files matching the source pattern have the same name. Copies all matching files to the destination folder, keeping their names.
```
Action, Source,       Destination, Flags
GET,    /lokad/*.csv, /input/,
```
This finds all files with the .csv extension directly inside the remote /lokad/ folder, and downloads them to the local /input/ folder
Source folder path to Destination folder path, recursively downloads all the contents of the source folder into the destination folder, keeping the tree structure intact.
```
Action, Source,  Destination, Flags
GET,    /lokad/, /input/,
```
This traverses the remote /lokad/ folder and all its sub-folders, and downloads all files and folders to the equivalent location within the local /input/ folder.

Note that if there are no files or folders on the remote server that match Source, then this instruction will fail and report an error, unless the optional flag is provided.

PUT

PUT will select all files that match Source in the local file tree, and upload them to the Destination on the remote server. The exact behavior depends on the nature of Source and Destination. The allowed combinations are:

Source file path to Destination file path, copies one single file.
```
Action, Source,        Destination,  Flags
PUT,    /output/a.csv, /lokad/b.csv, 
```
This uploads the local file /output/a.csv to remote file /lokad/b.csv.
Source file path to Destination folder path, copies one single file and keeps the same file name.
```
Action, Source,        Destination,  Flags
PUT,    /output/a.csv, /lokad/,  
```
This uploads the local file /output/a.csv to remote file /lokad/a.csv.
Source file path pattern to Destination file path, fails if the path pattern matches more than one file, and copies that file to the destination.
```
Action, Source,        Destination,  Flags
PUT,    /output/*.csv, /lokad/a.csv,
```
This expects to find a single file with the .csv extension in the local /output/ folder, and uploads its contents to remote file /lokad/a.csv
Source file path pattern to Destination folder path, fails if several files matching the source pattern have the same name. Copies all matching files to the destination folder, keeping their names.
```
Action, Source,        Destination, Flags
PUT,    /output/*.csv, /lokad/,
```
This finds all files with the .csv extension directly inside the local /output/ folder, and uploads them to the remote /lokad/ folder
Source folder path to Destination folder path, recursively uploads all the contents of the source folder into the destination folder, keeping the tree structure intact.
```
Action, Source,   Destination, Flags
PUT,    /output/, /lokad/,
```
This traverses the local /output/ folder and all its sub-folders, and uploads all files and folders to the equivalent location within the remote /lokad/ folder.

Note that if there are no files or folders in the local file tree that match Source, then this instruction will fail, unless the optional flag is provided.

DELETE

DELETE ignores the Destination, and will remove all files matched by Source on the remote server.

Action, Source,           Destination, Flags
DELETE, /single/file.csv, ,
DELETE, /multi/*.csv,     ,
DELETE, /recurse/folder/  , 
This will delete the remote file /single/file.csv, then all files with a .csv extension in remote folder /multi/, then the remote folder /recurse/folder/ and all its contents.

Note that if there are no files or folders on the remote server that match Source, then this instruction will fail, unless the optional flag is provided.

LIST

LIST expects Source to be a folder path the remote server, and will list the names of all the files (but not sub-folders) directly within that folder. The list is formatted as a CSV file and is then written to the path Destination in the local file tree.

Action, Source,           Destination,     Flags
LIST,   /path/to/folder/, /local/list.csv,
This lists all the files (but not sub-folders) of remote folder /path/to/folder/ into a CSV file, written to the local file /local/list.csv

Note that if the Source folder does not exist on the remote, then this instruction will fail, unless the optional flag is provided. If the optional flag is provided, then an empty CSV file is produced.

MOVE

MOVE will move a file or folder identifed by Source on the remote server so that its new path is Destination, still on the remote server. The main purpose of this instruction is to allow the files to be uploaded (slowly) to a temporary location, and then moved (instantly) to their final location. The exact behavior depends on the nature of Source and Destination. The allowed combinations are:

Source file path to Destination file path, moves/renames one single file.
```
Action, Source,            Destination,   Flags
MOVE,   /inbox/a.csv.temp, /inbox/a.csv, 
```
This renames the remote file /inbox/a.csv.temp to /inbox/b.csv.
Source file path to Destination folder path, moves one single file and keeps the same file name.
```
Action, Source,         Destination,  Flags
MOVE,   /staging/a.csv, /inbox/,  
```
This moves the remote file from /staging/a.csv to /inbox/a.csv.
Source file path pattern to Destination file path, fails if the path pattern matches more than one file, and moves that file to the destination.
```
Action, Source,         Destination,  Flags
MOVE,   /staging/*.csv, /inbox/a.csv,
```
This expects to find a single file with the .csv extension in the remote /staging/ folder, and moves to /inbox/a.csv
Source file path pattern to Destination folder path, fails if several files matching the source pattern have the same name. Moves all matching files to the destination folder, keeping their names.
```
Action, Source,         Destination, Flags
MOVE,   /staging/*.csv, /inbox/,
```
This finds all files with the .csv extension directly inside the remote /staging/ folder, and moves them to the remote /inbox/ folder
Source folder path to Destination folder path, moves the folder and all its contents to the new path. Will fail if the destination folder already exists, or if the destination path is a descendant of the source path (moving /a/ to /a/b/).
```
Action, Source,    Destination, Flags
DELETE, /inbox/,   ,            optional
PUT,    /staging/, /inbox/,
```
This deletes the remote folder /inbox/ and all its contents (if it exists), then renames the remote folder /staging/ to /inbox.

Note that if there are no files or folders on the remote server that match Source, then this instruction will fail, unless the optional flag is provided.

Flags

The Flags column contains a space-separated set of case-insensitive flags with the following meanings:

optional: without this flag, if an instruction performs no action because there were no matching source files or folders, then it fails with an error message. By specifying optional, the instruction instead does nothing.
newer: applies only to GET and PUT. Before uploading or downloading an individual file, the instruction will check that the destination file is either absent, or older than the source file. If not, the upload or download of that individual file will be skipped. Note that if all the files are skipped in this way, the entire instruction will fail with an error message unless the optional flag is also specified.
nonfatal: without this flag, if an instruction encounters an error, the execution of the entire script is immediately stopped. If this flag is provided, an error during the execution of this instruction does not prevent subsequent instructions from being executed, although the entire run of the script will still be flagged as failed (and so, any retrieved remote files will not be written to the local store).
retry and noretry: enable or disable automatic retrying upon failure. If retrying is enabled, the first few occurrences of an error will be completely ignored and additional attempts will be made. By default, retrying is enabled by default for GET and LIST, and disabled by default for PUT, MOVE and DELETE. Note that when retrying, the entire operation starts over from scratch (there is no attempt to resume a transfer interrupted in the middle).
afterget: changes the behavior of a MOVE or DELETE instruction so that it will leave alone any files which have not been the source of a successful GET operation during the current execution of the script. In other words, any files that match the source pattern of MOVE or DELETE but were not successfully retrieved during the current execution will be treated as if they did not exist. The use of this flag is for “retrieve and delete” procedures which get the files and then delete them (or move them to another folder), so that if a file appears between the retrieval and deletion stage, it will not be deleted.

User Contributed Notes

0 notes + add a note