File Reader

This source connector reads files from a local or remote directory on a specified interval/time schedule. Several protocols are supported, including regular local file mode, FTP, SFTP, SMB, WebDAV, and Amazon S3. Files may be read in and converted to Base64, or converted to a message string using a specific character set encoding. After reading in files, the connector has options to either delete the original files, rename them, or move them to a separate directory. Additional options (like FTPS) are available with the SSL Manager extension.

Supported property groups:

Item	Name	Default Value	Description
A	Method	file	The basic method used to access files to be read in. Options include File (local filesystem or NFS / mapped share), FTP, SFTP, SMB, WebDAV, or Amazon S3. Once all necessary connection/directory information has been filled in before, use the Test Read button to attempt to actually connect and test the ability to read from the directory.
B	Advanced Options		If the file method supports advanced options, this button will be enabled. Advanced options are summarized in the Advanced Options label. For additional information, see: Advanced FTP Options Advanced SFTP Options Advanced SMB Options Advanced Amazon S3 Options
C	Directory		Only applicable to the File method. The directory (folder) in which the files to be read can be found.
D	URL		Applicable to all methods except File. The domain name or IP address of the host (computer) on which the files to be read can be found. If this setting is enabled, the second text field specifies the directory (folder) to read from. When using the Amazon S3 method, the first text field will be the bucket name, and the second text field can be used for a directory prefix.
E	Filename Filter Pattern	*	Files with names that do not match this pattern will be ignored. If Regular Expression is disabled, regular wildcard (*) matching is supported.
F	Include All Subdirectories	No	Select Yes to traverse directories recursively and search for files in each one.
G	Ignore . files	Yes	Select Yes to ignore all files starting with a period.
H	Anonymous	Yes	Only applicable to the FTP / WebDAV / Amazon S3 methods. If enabled, connects to the remote server anonymously instead of using a username and password.
I	Username	anonymous	Applicable to all methods except File. The username used to connect to the remote server with. When using the Amazon S3 mode, this will be your AWS Access Key ID.
J	Password		Applicable to all methods except File. The password used to connect to the remote server with. When using the Amazon S3 mode, this will be your AWS Secret Access Key.
K	Timeout (ms)	10000	Applicable to the FTP / SFTP / SMB / Amazon S3 methods. The socket timeout (in ms) to use when connecting to the remote server.
L	Secure Mode	Yes	Only applicable to the WebDAV method. If enabled, HTTPS will be used instead of HTTP.
M	Passive Mode	Yes	Only applicable to the FTP method. If enabled, the server decides what port the client should connect to for the data channel. Passive mode sometimes allows a connection through a firewall that normal mode does not, because the client is initiating the data connection rather than the server.
N	Validate Connection	Yes	Only applicable to the FTP method. If enabled, the connection will be tested for validity before each operation.
O	After Processing Action	None	Select Move to move and/or rename the file after successful processing. Select Delete to delete the file after successful processing.
P	Move-to Directory		If successfully processed files should be moved to a different directory (folder), enter that directory here. The directory name specified may include template substitutions from the list to the right. If this field is left empty, successfully processed files will not be moved to a different directory.
Q	Move-to File Name		If successfully processed files should be renamed, enter the new name here. The filename specified may include template substitutions from the list to the right. If this field is left empty, successfully processed files will not be renamed.
R	Error Reading Action	None	Select Move to move and/or rename files that have failed to be read in (for example, if an out-of-memory error occurs, or the network connection drops). Select Delete to delete files that have failed to be read in.
S	Error in Response Action	After Processing Action	Select Move to move and/or rename the file if an ERROR response is returned. This action is triggered when the Response selected in the Source Settings has a status of ERROR. If After Processing Action is selected, the After Processing Action will apply. This action is only available if Process Batch is disabled in the Source Settings.
T	Error Move-to Directory		If files which cause processing errors should be moved to a different directory (folder), enter that directory here. This action is triggered when the Response selected in the Source Settings has a status of ERROR. The directory name specified may include template substitutions from the list to the right. If this field is left empty, files which cause processing errors will not be moved to a different directory.
U	Error Move-to File Name		If files which cause processing errors should be renamed, enter that directory here. This action is triggered when the Response selected in the Source Settings has a status of ERROR. The filename specified may include template substitutions from the list to the right. If this field is left empty, files which cause processing errors will not be renamed.
V	Move-to Variables		The variables listed here can be dragged-and-dropped into the Move-to fields to the left. channelName: The name of the current channel. channelId: The unique ID of the current channel. DATE: The current date, formatted as a human-readable string. COUNT: A numeric count that increases for each file read in, from the point when the channel was last deployed. UUID: An auto-generated universally unique identifier. SYSTIME: The current epoch time in milliseconds. originalFilename: The name of the file that was read in. Use this to easily add on an extra file extension to the original name.
W	Check File Age	Yes	Select Yes to skip files that are created within the specified age below.
X	File Age (ms)	1000	If Check File Age is enabled, only the files with creation dates older than the specified value in milliseconds will be processed.
Y	File Size (bytes)	0, Ignore Maximum	The minimum and maximum size (in bytes) of files to be accepted. If Ignore Maximum is checked, the file size will only be bound by the minimum value.
Z	Sort Files By	Date	Selects the order in which files should be processed, if there are multiple files available. Files can be processed by Date (oldest last-modification date first), Size (smallest first), or Name (a before z, etc.).
AA	File Type	Text	Select Binary if files contain binary data; the contents will be Base64 encoded before processing. Select Text if files contain textual data; the contents will be encoded using the specified character set encoding.
BB	Encoding	Default	If Text is chosen for the File Type, select the character set encoding ( ASCII, UTF-8, etc.) to be used in reading the contents of each file.

File Reader

Recommendations

HTTP Listener

Database Reader

File Writer

Batch Processing

Raw Data Type