You still have to defend against MitM attacks, that could steal your data. A recent example is Samsung smart TV that could be controlled with voice. If someone can route TV connection to his server posing as Samsung's server, she can eavesdrop you in your living room.
Another problem is that during the lifespan of product, several critical exploits could be found that would compromise the security if not patched. There already efforts to defeat SSL and we may need to upgrade to more advanced protocol in near future.
One last problem is that in your case the device needs to have server address hardcoded. The company could go out of business and you would have no way to redirect it to alternate unofficial server. Therefore, firmware upgrades are pretty nice to have in IoT.
Another problem is that during the lifespan of product, several critical exploits could be found that would compromise the security if not patched. There already efforts to defeat SSL and we may need to upgrade to more advanced protocol in near future.
One last problem is that in your case the device needs to have server address hardcoded. The company could go out of business and you would have no way to redirect it to alternate unofficial server. Therefore, firmware upgrades are pretty nice to have in IoT.