Error handling in ETL

1. info

2. warning

3. error

4. secure

1. debug

2, trace

3. fatal

1. record level

2. file level

3. global level

1. File level, but needs to stop other files

3. File level, but actions differ what that error means, for example don't move the file and consider rerunning it, etc.

how to send alerts based on these errors

1. archive processed files

2. when to remove files for backup

3. how long to retain

4. log files retention and removal

How to reduce the amount of logging?

Search for: How to reduce the amount of logging?

1. validateParams

2. copy to local variables

3. validate local variables

1. validateParams

2. validate necessary environment variables

3. copy to local variables from params

4. copy from env variables to local

3. validate local variables

Simulation can help a lot...to debug and test

1. A better name desirable

2. flowlog is a log statement which when printed and nothing else shows the flow of an EtL without overwhelming the logs.