- There are things like pv(1) which allow you to monitor pipes. Things like systemd open other interesting possibilities for implementing, grouping and monitoring your processes.
- Recovery could be implemented by keeping a logfile of completed steps like a list of completely processed files or moving processed files to elsewhere in the file system (could be done in memory only using ramfs or tmpfs). Of course, it depends on the case whether it's feasible or not.
- Extensibility: Scripts and configurations can be done in shell syntax. Hook systems and frameworks of varying complexity exist. I agree that doing extensibility in shell code is going to turn out to be hazardous when done without proper concept and understanding of the tool at hand.
- There are things like pv(1) which allow you to monitor pipes. Things like systemd open other interesting possibilities for implementing, grouping and monitoring your processes.
- Recovery could be implemented by keeping a logfile of completed steps like a list of completely processed files or moving processed files to elsewhere in the file system (could be done in memory only using ramfs or tmpfs). Of course, it depends on the case whether it's feasible or not.
- Extensibility: Scripts and configurations can be done in shell syntax. Hook systems and frameworks of varying complexity exist. I agree that doing extensibility in shell code is going to turn out to be hazardous when done without proper concept and understanding of the tool at hand.