Big +1 to “ Never give up on local testing”. The current code base I’m dealing with has many tests that require interaction with a dev environment on the cloud, and occasionally these tests fail due to a timeout or some other thing not related to actually testing the code (and instead reveals that I forgot to refresh my MFA).
Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.
Yeah, that advice was bad and looked like it was written by someone that's done too many deployments and not enough development.
He's certainly right that deployment validation is the best test at some level. But it completely leaves off the most important aspect. That deployment validation, and the mistakes that you inevitably find, should directly guide your test development to prevent regression.
If you do this, future component upgrades and deployments get easier. Its well worth the extra effort.
He's not discouraging automated testing; he's just saying that we should prefer integration tests (using services running locally) to unit tests of individual functions. Look up "testing trophy" for more about this philosophy.
Totally with you on the value of integration tests. I'm familiar with the trophy and the triangle, but he's saying this:
'..with unit tests notably coming in last place – i.e. “only if you have some time”.'
The "only if you have time" part is what I disagree with. If we're doing TDD then I don't see how you can avoid writing unit tests, or even deprioritize them.
> The "only if you have time" part is what I disagree with. If we're doing TDD then I don't see how you can avoid writing unit tests, or even deprioritize them.
Perhaps the author doesn't recommend TDD? They don't suggest anywhere that they do. It seems to me that the industry these days sees TDD as a tool for occasional use, not the dogma once proposed.
+1 unit tests are essential for software evolution like refactors, updating business logic, and updating dependencies. More generally, good testing requires a defense-in-depth strategy and unit testing, integration testing, and canary testing all have a role to play. All other things being equal, it's much better to catch a bug before you even push to the the source repo versus when it has one foot out the door to your customers.
Yeah as some of the responses to your reply said, agreed, and I've always found value in unit tests, and the "too many deployments and not enough development" comment from jsight rings loud in my head. Deploying to a real non-prod environment as a "test" can be fine for a team, but to do so while not testing for how you expect the code to behave... maybe that works for some folks and if you're moving super fast, but I wouldn't be super comfortable with it myself.
I totally disagree with the author on testing. Integration tests are more trouble than they are worth in my experience, and unit tests are really useful.
Most people building stuff quickly in serverless seem to eschew the concept of local entirely, and it's something I've recently started doing too. I make a code change locally and it gets picked up by the deployment of my stack associated with my feature branch the second I save the file.
It's an uncomfortable concept at first but I find it's helped me build things faster and has ultimately led to less rabbit hole chasing across multiple categories of issues.
thanks for saying that! people are pretty skeptical about localstack when they hear about it for the first time, and don't understand how we could ever emulate something remotely resembling AWS. tbh sometimes i'm baffled myself, it's pretty crazy what it can already do (i work there).
With AWS SAM you can invoke functions locally, run api gateway and use step functions. There's also a 3rd party project to run various AWS services locally, can't remember the project name.
There's also SAM sync that syncs changes you make quickly, so if you have your own dev copy in AWS you can quickly test changes.
There's also a fair amount of lambdas where you can just invoke the lambda handler yourself without anything special other than a compatible version of python/node/etc. Variations of things like:
For golang, but would work in any language. What's really worked is the idea that your main loads configuration and injects all of the configured interfaces into your handler.
You then can build all of the testing on mocks/stubs to test the behavior. If you access a database you access in through the interface which can mimic the appropriate behavior for your code in test vs production. If you need to you can do local integration testing of the db access layer.
Honestly, my answer is: don't use serverless, or at least have a non-serverless way of running your code locally, preferably outside of a container/VM. Otherwise you have no way of running or testing your code locally, much less attaching a debugger; IMHO this can slow down dev cycles so much that it cancels out the time saved by adopting a serverless architecture.
Serverless deployments aren't painfully slow like they are with Kubernetes or other container orchestration platforms. At least with AWS lambda you can change and deploy your code in <10 seconds and have it taking traffic (In an infinite number of "environments"). So developing locally is kind of pointless.
> Serverless deployments aren't painfully slow like they are with Kubernetes
What are you doing with your K8s cluster/containers that is making your deployments slow? My cluster at work pulls down containers faster than my local, and deployments are usually swapping within ~1 minute of being committed in git/the container build finishing…
Additionally the dependency thing can be huge; we ran into a weird bug for installing a particular dependency on our CI system so our test there keeps failing, but being able to run it locally let’s us know that the changes did not break our actual code tests.