In my opinion, what works in the small also works in the large and even though there may be caveats with this rule (e.g. synchronous communications vs asynchronous ones that may introduce non-determinism), if you stick to the rule of building simple components that do one thing and do it well and them combine them to build bigger components that are still simple (in the non-entangled sense), then you'll definitely do a good job. But that's easier said than done because ...
There are two types of problems here:
1. doing design mistakes because you make a decision that you never took before and then you get burned by it - as an example you might introduce unwanted cyclic dependencies between your components and if you're talking about different address spaces, then things can get really problematic, or you might introduce concurrency concerns (like in the communications between components) that could be avoided - problems that you won't avoid until you get burned by them.
2. some projects have an incredibly complex business logic - I'm working on such a project right now and the business logic is like a freaking fractal. Being so complex and the architecture being based on micro-services, the downside is that people are only willing to understand only as much as they need to for the components they are responsible for. On one hand that's good because that's how you parallelize the development effort. But then the people can't view the big picture anymore and this is usually a failure of business analysis, because truth be told most systems are built with a primary purpose (for example the primary purpose of an email platform is to send emails in bulk) and then the "stakeholders" lose sight of that and come up with often useless and often conflicting features that detract the developers from focusing on the primary purpose (e.g. what use has an email platform that's shitty at sending emails). The end result in our team is that a single man knows all the business logic details and dealing with that takes so much of his time that he does not have time for software development at all.
I do not agree with you on studying large systems. Large systems that have to be studied because they are large are in my opinion a failure of design that shouldn't be copied. IMHO, much better is to develop a nose for simplicity and always strive to build simple things, while also pushing back on the amount of work needed by simplifying the business logic.
While I agree with you that the people that can deal with the above are incredibly rare, I'll stick to my initial claim ... people good at it are software developers that (1) are always studying new ways of thinking and learning from others and (2) that try to do things and get burned a lot.
And nr. 2, the experience you've got, is incredibly important. There's no amount of studying that you can do for successfully recognizing and fixing or avoiding concurrency problems (and btw, I'm not talking about multi-threading here, but about the general issue of concurrency that can happen in all sorts of communications). There's no amount of studying you can do that can help you recognize entanglement that shouldn't exist in the architecture. Sure, books help, but books don't fix the superficial attitude that rookies have towards such problems. In order for one to learn, one has to build stuff and suffer from such issues and then improve.
Good news is that if one follows the above advice and gets out of his conform zone, then one can short-circuit the learning process to about 5 years instead of the usual 10.
There are two types of problems here:
1. doing design mistakes because you make a decision that you never took before and then you get burned by it - as an example you might introduce unwanted cyclic dependencies between your components and if you're talking about different address spaces, then things can get really problematic, or you might introduce concurrency concerns (like in the communications between components) that could be avoided - problems that you won't avoid until you get burned by them.
2. some projects have an incredibly complex business logic - I'm working on such a project right now and the business logic is like a freaking fractal. Being so complex and the architecture being based on micro-services, the downside is that people are only willing to understand only as much as they need to for the components they are responsible for. On one hand that's good because that's how you parallelize the development effort. But then the people can't view the big picture anymore and this is usually a failure of business analysis, because truth be told most systems are built with a primary purpose (for example the primary purpose of an email platform is to send emails in bulk) and then the "stakeholders" lose sight of that and come up with often useless and often conflicting features that detract the developers from focusing on the primary purpose (e.g. what use has an email platform that's shitty at sending emails). The end result in our team is that a single man knows all the business logic details and dealing with that takes so much of his time that he does not have time for software development at all.
I do not agree with you on studying large systems. Large systems that have to be studied because they are large are in my opinion a failure of design that shouldn't be copied. IMHO, much better is to develop a nose for simplicity and always strive to build simple things, while also pushing back on the amount of work needed by simplifying the business logic.
While I agree with you that the people that can deal with the above are incredibly rare, I'll stick to my initial claim ... people good at it are software developers that (1) are always studying new ways of thinking and learning from others and (2) that try to do things and get burned a lot.
And nr. 2, the experience you've got, is incredibly important. There's no amount of studying that you can do for successfully recognizing and fixing or avoiding concurrency problems (and btw, I'm not talking about multi-threading here, but about the general issue of concurrency that can happen in all sorts of communications). There's no amount of studying you can do that can help you recognize entanglement that shouldn't exist in the architecture. Sure, books help, but books don't fix the superficial attitude that rookies have towards such problems. In order for one to learn, one has to build stuff and suffer from such issues and then improve.
Good news is that if one follows the above advice and gets out of his conform zone, then one can short-circuit the learning process to about 5 years instead of the usual 10.