I've been struggling with this for years now; after 4 boring jobs I finally gave up even the expectation that what I'll be asked to do will be even remotely interesting (or socially useful). The chance of landing an interesting job is absolutely minuscule - the market rewards whatever it rewards at the moment, and that's where jobs flock. These days, one of the bigger thing the market rewards is writing a web-based CRUD that's 99% chrome, 1% of actual functionality. Extra points if you can get some user data to sell to adtech industry.

(My own criteria for "interesting" these days are a bit different. I'm not looking for "interesting technical problems", as in advanced CS stuff, but "interesting outcomes" - pushing the humanity forward, or alleviating a social problem for some group of people. I find it hard to even find companies that apparently work on something like that, and even then it seems they have to do lots of boring work as well.)

--

EDIT: Then again there's this topic that has been casting shadow on my happiness for a long time, that I finally seem to be able to define. Maybe our civilization has finally crushed my soul, and I noticed it just now. But it feels to me that the revealed goal of our market-driven world is to abstract everything away with money. The end-game is that as an individual, your responsibility in the society is to pick a skill - whatever skill - that makes you money, and then to use money to solve everything else with your life. It doesn't matter what you do, there is only one meaningful meta-skill - making money fall out of the system, into your bank account.

Maybe I'm just realizing something obvious that parents should have taught me when I was a kid. I don't know. Maybe it was wrong of me to seek meaning in the things one does? Maybe that approach is no longer supported by our civilization?

Not all of us are raising families, but I think it highlights a bigger points. Work is not what builds societies (certain jobs do harm society I think, mostly by destroying it's moral virtues). People relationships are what build societies and build happiness.

And you're right - there are a lucky, select few who get jobs that can also influence society.

I'm not worried about that though (anymore - I was for a time though). With my job, I can provide and care for those around me (provide for my family and generally have positive, uplifting interactions with those around me).

I have resigned myself that it's not my job to fix society, it's our job to fix society, and I'm doing my part where I am. From my perspective I have more than enough to do locally anyway: https://news.ycombinator.com/item?id=17176036

The reason I ended up in tech is in part my fascination about tech, especially as applied to uplifting ourselves and solving our problems (yes, I watched way too much Star Trek: The Next Generation in my formative years). I never thought much about family - in fact, for the most of my life, I felt it's something of a side endeavour. I still feel it shouldn't be the sole focus of a person, though I definitely believe it's more important than I believed in the past.

Same for relationships. And small communities. I feel that maybe we overdid this globalization thing. Maybe global competition and centralization of various kinds of innovative work is more efficient, but it also doesn't seem healthy. Living in a city, there's no such thing as "local community" anymore. Not even neighbours.

> I have resigned myself that it's not my job to fix society, it's our job to fix society, and I'm doing my part where I am.

Still, I'd like to feel like I'm actually doing my part beyond just participating in the economy as consumer. I have specialized skills, I'm not dumb. But I have hard time finding opportunities to do things I'd know could help meaningfully and effectively. There must be something. Maybe I'm just not looking hard enough.

Your comment you linked to is golden. I'm favouriting it. Those are definitely words I'll try to remember from now on.

A few years ago I applied for a really interesting jobs within Telematics - one of my favorite areas. So I called them, got the CTO on the phone and he had almost no time for me. Then I wrote a mail with my formal application to them. It was appealing and they replied with some coding riddles. I guess I did well enough and was one of 2 or 3 people to get invited. Of course before the in-person interview I studied some Algorithms & Data Structures book. At the same time I did Scala programming although it wasn't expected to know that but people worked with it.

Anyway, during the interview they made me sweat for at least 3 hours. Every now and then also reminding me how good the other people working there are - the one guy having worked 8 years at a company writing DBMS in London. So yeah, after leaving the communication was difficult but the CTO said the people said they liked me and they would hire me despite my shortcomings - I would have to unlearn stuff. The offer was really bad, in fact even the most senior people got below average salary.

At the end I started working at a company that worked on a moonshoot project involving a CAD program in the browser. Even some mathematician who worked in Japan on Robotics was there. Salary was good enough but the people were borderline insane. I quit after 3 months and landed a below average paid job, but with a lot of nice people. I don't regret that. Work was mostly boring but the environment was nice.

Anyways, not living in the Bay Area, I guess things are they different again.

I still enjoy trying to match a user's expectations with the software I write. But, it's balanced by other aspects like coaching and being in a family.

Sure there is. I live in a condo with 20 other units, and it makes a great small neighborhood.

In reading your post, I had to stop and think for a bit.

I agree with you but want to give the thought that family can be one's top priority without looking for it fill every aspect of one's life.

no one is above great tragedy and loss.

If my family was taken from me unexpected, I don't I'd say that I wished for more time at the office, or working on personal hobbies. I do believe in balance, but the point I'm trying to make is that family can be the number one priority.

The mindset of "I can be happy because I have x" (in this case where x = family), is indeed dangerous. And while the mindset of "I have chosen to be happy, and my top priority is my family" will bring sadness and pain if they are suddenly taken, I feel like that pain and sadness will have been worth it. Well, it is for me. That for others it might not is understandable.

Please note as well that I'm not advocating self-deprecation or letting your family take advantage or you, or throwing yourself under the bus for an uncaring family - it's a group effort and there a lot of ways to appear to put your family first in a way that's detrimental to one's self.

For me when I think of dedication in family life, I think of the story, "the gift of the magi". Nothing of true importance was sacrificed, and yet the others needs were prioritized.

According to the book, the transition from being war criminals by day and family men by night, was possible through some rationalisation acrobatics or detachment from responsibility. As an example of detachment, the Nazi concentration camps didn't have an atypical ratio of psychopath to average joe. The predominant "average joes" would offload responsibility by telling themselves that this is not their decision, but that this is the _current policy of the organistation_, which is an interestingly complex statement, in the subconscious sense. It detaches the organisation participant from the organisation, and it detaches the organisation from their policy.

To come back to business v family, I think the old koan holds: separation isn't good. The rule of thumb in social situations is if detachment or separation is the easiest route, you're doing something wrong. Business is social action by necessity. So should we separate action from responsibility? What do we think of someone who is selfish and avoids responsibility?

Yes, your job as the "average Joe" is to be loyal to the organization. That statement doesn't sound surprising to me, and it took me a few seconds to realize why. I've heard it before. I grew up in a religion where there was an organization, and your job was to be loyal to God by being loyal to the Organization. Only by being on the outside now I see how this is a neat way to release responsibility. See also: "Ours not to reason why, ours but to do and die.".

> The rule of thumb in social situations is if detachment or separation is the easiest route, you're doing something wrong. Business is social action by necessity. So should we separate action from responsibility?

It was easier back when you were naturally limited by geography to do business with people you also knew "after hours". These days, business and personal relationships are almost entirely non-overlapping. Hell, it's a common advice to avoid that overlap.

It's as if childhood primes us to think that life is all about figuring out what the heck is going on around us, and as a consequence some of us go into something science-y, but then we reach a certain stage in adulthood where we feel like we have a pretty good rough closed form solution to the problem, and now the journey all of a sudden seems empty and without purpose and we just keep going with the lives 16-year old selves planned out for us, with too much cost sunk to pivot. I almost suspect the end-state of science is not some sort of blissful state where we feel like we understand everything, but an utter dread and feeling of emptiness at no longer having a purpose.

For some time in my dream I was lost, not sure what my value was as a human being without my skills that I've spent decades refining.

Then I remembered that humans have inherent value, that I am not my work, and went to a park to meditate and teach karate. It made me think, in my dream, how even if a machine could teach karate better, I was not doing it for society -- I was doing it for myself.

Realizing that you are not your work, and are not defined by your work; that humans have inherent value and you have every right to simply be happy when the opportunity arises; that the Puritan virtues of work are an illusion, and that humans should follow their passion; that the illusion of the necessity of work often prevents people from ever discovering their passion in the first place.

It was probably my best dream, and I approach life quite differently now. I think when I feel like I have a closed form solution, it is much the same spirit as an AI that will solve your problems for you.

Personally, understanding the "big picture dynamics" of systems is kind of an intellectual porn for me. I loved doing that for as long as I remembered. It always came with the sense of control - understanding the systems means you could tweak it to your liking, if you so desired.

Finally realizing that the system in which we live is basically uncontrollable by an individual within it has blindsided me. So many interesting things happening, so many interesting things that need to be done still, and yet all you and I get to be are observers and consumers of whatever the system happens to spit out.

https://wiki.snowdrift.coop is probably the best introduction right now; the main site is in the process of being updated.

I agree. Maybe there is a word for that?

'Attachment to expectation'. See buddhism.

https://en.m.wikipedia.org/wiki/Marx%27s_theory_of_alienatio...

I don’t think “existential angst” fits. As the name implies, you get that just by existing and being trapped by the limitations of being a mortal human.

You’re talking about the specific meaninglessness of modern work. Marx suggests that capitalism affords us some real freedoms, but at the cost of our feeling of purpose and solidarity. And, Marxists would argue, this makes us easier to exploit.

Marx and his inheritors failed to produce systems that solved this problem, and maybe that indicates that the problem is deeper than just forms of ownership and control.

But as a problem description, I think he was onto something.

Koyaanisqatsi - life out of balance - see also the film

You guys talk about boxes. Well there are huge ones for the mind, carefully constructed to be believable for the masses. Too complicated to understand, they require you to put beliefs in people with credentials, who will repeat to you what they found in the books they were given, or the math they got shown.

If you want to understand your purpose here, you have to step outside of scientific thinking to start with, and that's a huge problem for almost everybody. You have to look at origins of the matrix we are living in, the illuminati, the Egyptians, book of the dead, the occult.

> ...also get away from [the people in] the box...

This may be the harder part. I consider some of the people in the box to be friends. But then again, if I think about it... maybe they're outside the box too, but we're on opposite faces, and so close to it that we are still looking at each other _through_ the box.

So here I am, with the map of the boundaries of the box finally taking shape. Now the question is, how do I exit the box? I'd like to walk around on the outside and check it out.

Also, is leaving this box even a good idea? What if everyone left the box at the same time? Would the world collapse then? (I can't help myself, I tend to be interested in the global view as well.)

It's a very nice idea.

It does have the potential to create localized unimpressedness though; the question is how powerful the temporarily-put-out groups would be, and whether the backlash would remove your power as well.

Money is power, laterally too.

In my mid twenties this was hard for me to grasp. I was working as an electronics design engineer for a company that made low-end industrial controls. We had a customer who had what seemed like the dream company to me: he made robots for animatronics and lab applications. One day I was talking to him about how exciting the work was and his answer surprised me: it used to be fun in the beginning but after doing it for a few years, it was just another job.

I work with wireless sensors now. Yes, it's technically interesting, but it's pretty boring.

Why? I've discovered over the years that I am far more excited by making customers happy than by doing technically interesting work. I've come to see the technology as a means to an end, not the end in itself.

I worked with an software developer at my last job who was incredible. He seemed to have a strong interest for the technology, no matter what technology that was. At the time, it was iOS, and he was an absolute master of iOS. I was constantly impressed by his attention to detail and emphasis on quality, well written, technically challenging code.

I've loosely followed him since, and he seems to be absolutely crushing it wherever he goes. He's spanned many different roles and seems to do enjoy taking on new challenges in the form of different programming languages / platforms.

When I think back to him, I remember someone who seemed to just love programming. In a way, he was like a Robot. It doesn't appear to matter much to him what he's working on, or if it's "fulfilling" in the ethical sense.... He just enjoys the process of coding, and being damn good at it.

Question - is his management crushing it as well or are they crushing him? Is he good at standing up for himself and not letting management crush him?

I think a lot of enthusiastic and brilliant people get destroyed by their managers exploiting them. So the individual goes from crushing it to getting crushed. Management really has to enable that kind of person to succeed.

Yes. This. He was very, very good at standing his ground and asserting that his time and effort was being spent well.

Multiple times I witnessed management clashing with him over something. Usually he felt like more time and effort needed to go into a certain part of code. Rather than just "get it done" he took pride in his work and wanted it done RIGHT.

In general he was so good / fast that he could get it done "RIGHT" within management's expectations. But occasionally he would take on a big chunk of work, and management would try and shut him down. 9/10 they failed, he would do the work, and a damn good job of it.

A good manager would realize that his quality of work was top notch but that he was at times difficult to get along with. He was a perfectionist who held those around him to the same standards, and that often clashed with the business needs and even sometimes other teammates. A business generally isn't concerned with the product being perfect, they're more concerned with it getting to market on time.

So my suggestion to any manager that has an employee like this: Value him for what he is. Put him on MISSION CRITICAL tasks, where you need a rock solid implementation. Let him work for 2 weeks without interruption.

For other tasks that are perhaps not as "critical", throw other team mates at them.

Also, not everybody has the personality or energy to constantly stand up for themselves. A lot of people will eventually crumble or leave. Don't make it so the only way the employee can do their best (e.g. what rubicon33 describes) is if they put their foot down and scream.

100% agree. I'm a Data Scientist and I'm supposed to have one of the most exciting and sexiest jobs in tech. But after many years of doing this I've come to realize that working with good customers and seeing them happy is much more rewarding than playing with the latest machine learning thing. New tech is just the thing that makes it easier to tackle a customer use case.

There's a strong relationship between the quality of the people I'm working for and my sense of the work being interesting. If the people I'm working for are unrealistic a-holes no amount of technical novelty can make up for that.

At one of my previous jobs, "my customers" were the managers of some company who asked managers of the company I worked in to deliver some software that will be sold as a part of a software package, to be used by someone downstream. I don't know who ended up using my work and how, because we were isolated by several abstraction layers from that. I did care about the happiness of those people, little I knew about them. But I couldn't care less about the happiness of our customers - a company I didn't hold in high regard, with managers not being particularly helpful. I suppose I did get some small sense of satisfaction when their managers told my managers that the work is good. But that was it.

So, are we talking about happiness of customers, or happiness of people actually using the product? Those are very often different groups. Or, in other words, are we talking about finding happiness in generalized servitude, or making people's lives better?

I've also worked in situations you've described. My only interaction with the end user is through layers of middle management. I found that environment to be extremely demotivating and certainly no direct sense of satisfaction. One of the small benefits of being a Data Scientist right now is that a lot of managers don't really understand what it is I do so I get some interaction with the end-user from a requirements and solution validation standpoint. But I suspect in a few years Data Scientists will become invisible cogs (like everybody else) in the Enterprise IT machine.

I don't think you can get that kind of satisfaction in B2B software as a developer (only a salesperson/exec).

Two kinds of "work": work that gives you money, and work that gives you value. They are not necessarily the same one.

Most people have to work for money. The best we can do is to gather more "money" (or resources). So that you can dedicate more of your time to generate, as you said, "interesting outcomes".

I recommend the book "Your Money or Your Life" by Vicki Robin. I learnt a lot relevant ideas from it.

Ouch, this resonated. I've been walking this path with you.

Sometimes I make my own food. Or grow it. Or pickle it. It would be more "efficient" to buy such food in many cases. But if it gives me pleasure, I trade my time to do it.

My opinion: the point of life is not to acquire so much money that you can avoid doing any difficult things and wrap yourself in a cocoon of comfort. The point of life is to find your own good life and live it. That will entail some difficulties (at a minimum the difficulty of seeking your own definition of the good life, which is hard!). You may find that in your work (lucky you!) or you may have to work to support yourself and find this good life in other areas.

When I wrote my edit to the comment, I didn't expect much reaction to my confused ranting. I was surprised to discover it resonates so strongly with people here. I guess I'm nowhere near alone in thinking those thoughts, and going through this process.

This is the lens through which von Neumann and Morgenstern saw all of human behavior, and their "rational" economics has been hugely influential in public policy. I use scare quotes because I disagree that the approach is rational -- many things in life can't be measured in dollars, not even dollars! (E.g. you can't compare a dollar today with a dollar ten years from now. Regardless of your discounting strategy, that money is going to mean something different to you in ten years.)

Your position is a great example of this phenomenon -- there's not really any amount of money that can compensate for a lack of inspiration at work. You might be willing to give up on interesting work for enough dollars, and the rational economist would say that this is the measure of how valuable interesting work is. But there's a long history of studies, notably by Kahneman and Tversky among many others, showing that framing human motivation this way leads to all sorts of paradoxes.

So yes, we have a strong tendency to measure everything with money -- this outlook is deeply embedded in our economic and political systems -- and it makes people miserable because it's a terrible way to account for human happiness.

Agreed. For the most part, going to church is not economically rational. For the most part, having children is not economically rational. For the most part, being monogamous to one's spouse is not economically rational. For the most part, patriotism to one's country is not economically rational.

In a slightly different sense, going to space, or colonizing Mars, would all be economically irrational.

I'm wary of a definition of "rational" that tends take all of the things that humans hold dear, and declares much of it "irrational".

https://en.wikipedia.org/wiki/The_Righteous_Mind

These days, one of the bigger thing the market rewards is writing a web-based CRUD that's 99% chrome, 1% of actual functionality. Extra points if you can get some user data to sell to adtech industry.

I don't think they are rewarded on the grand scheme of things. Software has always showed a winner-takes-all structure and most software disappear. Take for example the game industry. 15 years ago, you would have talk to anybody, they would have told you the market is mature and they are making money on making movie-based games. Today, most players (publishers, studios, etc) of that era have disappeared (Im too lazy to get the long list of publishers that are gone).

Also innovation can act in a subtle way. Take the iphone for example. People are more productive because of it. That means that hospital, universities, research labs, non-profits, etc, all are doing better jobs which means we cure more disease, teach, help and study better, etc. because of the iphone.

But yeah, if you dont work in innovation, then you work for the current "government", the current group-thinked direction made by some smart "politicians" (by "politician" I mean that guy who lick asses all day long, who goes always in the direction that's well accepted and who use words to seduce people and get what he wants). It's either innovate (rare) or maintain.

Maybe our civilisation has finally crushed my soul, and I noticed it just now. But it feels to me that the revealed goal of our market-driven world is to abstract everything away with money.

There is a passage in the movie "1492: Conquest of Paradise" by Ridley Scott that I love:

"Look outside, what do you see?" ask Colombus to some royal acolyte

"I see towers, I see palaces, I see steeples, I see civilisation. And I see spires that reach to the sky!" answer the politician contrasting with the wild america of that time.

"All of them created by people like me" say Colombus

I handicapped myself by going into CS instead of graduating some hard engineering degree (or biology). See also matte_black's comment elsewhere in this discussion. Programming alone is a tool in search of problems to solve, and the interesting problems are elsewhere.

Isn't this what humans like to strive for? Making an impact, being remembered after death, thus living forever. That's one of the things a family (with kids) helps you get over, you know, the whole "living on through your kids" thing. If they don't end up as homeless drug addicts, it probably kinda works.

I've taken a job at a university two years ago. Pay is much lower than before. Haven't felt anywhere close to overworked or burnt out ever since, even though I've stayed until late at night occasionally when we fucked something up by deploying without any proper testing again, because hey, it's such a tiny change. I'm not doing frontier science or anything, we're just working on a custom netbooted Linux distro that also runs windows vms via qemu-kvm, for teaching environments. A lot of what we do could be dismissed as NiH-syndrome or creating solutions to problems that have already been solved by big companies, but as long as someone is paying for it I'm good with this job. Because there pop up a lot of problems or requirements that are just interesting to solve by yourself, actually not just for fun but for actual use cases. Nothing game changing for humanity, not for millions of people, but for some. And I will interact with them directly, and sometimes they tell you that they like the system you helped develop. That, and the fact that you're not in competition with your coworkers or another team or anything really makes for a great work atmosphere that I'm currently really enjoying. I don't know how long I can/want to stay, but for now I can say it greatly improved my overall happiness.

I don't know. Everyone has kids. Having kids is technically making an impact, but I think that when most people say they want to make an impact, or to accomplish something great, or to change the world, they don't mean by doing something that almost everyone else does.

I guess that's what the "help getting over it" part was referring to. In your teens/early twenties you dream of changing the world, and after reality sank in, you do the next best thing and go for kids because you shape them and so live on through them, mission accomplished, impact has been made. Humans are great at bullshitting themselves. (Not trying to sound harsh or implying kids are worthless or anything, but it really seems to be a fallback plan for a lot of people)

No, it's hoping that you plus your kid(s) will have more impact than you alone. Or that your kid will magnify your impact just by existing (which there is some evidence for if your chosen venue of impact is electoral politics, at least.)

That's one mental model but it doesn't fit exactly with reality and there are others. See the Love & Money graph here for example. Not quite sure how that fits in with the job hunting. https://www.theatlantic.com/magazine/archive/2003/01/does-mo...

Not sure what you mean by "social" problems specifically, but when you alleviate a problem for a group of people, they pay you, and that's what a business is.

I realized recently what a useful abstraction money is, because when money came about we didn't have the ability to create a massive blockchain keeping track of who did favors for who, and when. So it's a reasonable proxy for reputation and social credits.

- Make it so that people in your community have access to clean water. Surprisingly, it's a problem even in the most advanced countries.

- Contribute to something in healthcare that reduces costs, improves availability, or makes more people survive a procedure, an illness, etc.

- Make firefighting safer, or more efficient. Make it less needed in the first place.

- Make the air clearer. Make the energy cheaper. Make the roads better and safer. Etc.

- (Meta) make the market less ennui-generating.

I found that "what pays" is not a good proxy, because often what pays best is solving problems for a group of people, which also generate further problems for other groups of people. Usually, the benefits for the first group are purely financial.

To use a simple example - advertising. There's lots of people who would pay you for your work in this space, because your work will earn them more money. What your work will do downstream, however, is most likely one of those:

- It'll create more work for someone else to cancel the effects of your work; a lot of advertising is playing a zero-sum game, i.e. simply wasting natural resources and human time. (Conversely, often your work is only needed to cancel out the effect of someone else's efforts.)

- It'll disturb the market, promoting a worse solution over a better solution, leading customers to make wrong choices, and thus lose some amount of money and/or happiness.

- It'll add to the noise we all have to endure, also slowly eroding happiness.

Occasionally, you might end up promoting the right thing to the right crowd for the right reasons, but I doubt that happens very often ;).

Conversely to all of that, I feel that solving what I see as "social problems" pays less, if at all, again making "what pays" a bad proxy towards finding ways to address those problems.

It's a somewhat serious, somewhat tongue-in-cheek question. But if you need to clean your room, maybe you should do that first. Instead of having grandiose visions of improving things at large scale, prove to yourself you can improve things at a local scale and keep them that way.

If that sort of thinking is interesting, there's a whole book of it in https://jordanbpeterson.com/12-rules-for-life/ Though I'll say I got out of the existential and nihilistic mindsets differently, I don't feel like I have to wrestle with meaningness like I used to, before I ever heard of the author, I've just read good-sounding things from people who have read that book or have otherwise gotten themselves invested in the author's doings.

I suspect you've read this before but if not, point 5 might resonate with you: http://coding.derkeiler.com/Archive/Lisp/comp.lang.lisp/2006... If you fit the mold and can't change your mindset then you'll forever be unhappy unless you manage to get one of those "interesting" jobs.

For some reason, I still think they have value and that their problems are worth solving, regardless of what the market thinks of them.

Also, getting paid by the government to do something is decisively a non-market solution, given that the government gets its money through taxes.

Panera Bread gross revenue $2.8B, McDonalds gross revenue $22.8B

Nordstrom gross revenue $14.4B, Walmart gross revenue $500.3B

Payday loan revenue in the rich burbs roughly-zero, Payday loan revenue nation wide mysteriously between $6B and $46B depending on data source

Now can you get a cool and trendy very high social status hipster job to brag about on social media in the coolest office with only the best "personality fit" coworkers helping poor people, oh heck no you'd be savaged here if you admitted to working for McD, but the market of poor people products, at least in some areas of business, is about ten times as large as rich people products.

It’s also a hell of a lot easier to take advantage of them, given they generally lack the education or resources to protect themselves from predatory businesses. I struggle to think of McDonald’s as “helping” poor people. It’s putting food in their mouths, yes, but if McDonald’s were more concerned with the health and financial wellbeing of their customers as opposed to exploiting psychology to get as many people to impulse buy as much junk food as possible it would be an incredibly different company.

The goal of a company is not “help” its customers, the goal of a company is to exploit any and every untapped source of wealth. If it can do that by hurting someone who doesn’t know better, it is going to.

Payday loans may "solve a problem" but they do it in a predatory and exploitative way.

But, I'm not trying to comment on either of those, just that you can create utility for a poor person and still have a viable business. If it relies on government for funding, because society has decided it's worth funding that sort of work, so what? We have an entire defense industry in the US based on that idea. Most of our education is too. Other countries have their entire healthcare systems based on that idea. You can decide you don't want to be paid directly by taxed money, but that's on you, not on the viability of such a model.

My point is, you have to compare them entirely. At least Gates seems to have moved into the "good" quadrant later in life.

With the explosion of NGsequencing there's no shortage of data or data-magician positions, and there's many challenges with social impact

That said, I'm also having hard time separating academic work that helps things from that which just adds to the noise - grant-chasing, paper-generating studies. Any advice on that?

Why does this noise bother you? Besides, keep in mind that the grant-chasing doesn't happen because of greed, it happens because people want to continue doing science and that's the only way to maintain an active lab.

I could shit on the state of academic publishing for days, but at the end of the day, there is an overall positive direction in this 'academic endeavor': the creation of knowledge.

I find this direction much lighter on my consciousness than "generate value for the shareholders"

The noise bothers me for two reasons. One, because grant-chasing to keep the lights on means you end up having to prioritize busy work over important problems, which wastes lives of researchers and slows the scientific endeavour. Two, because high noise makes it more difficult to find interesting/important results of other people's research.

Maybe I'm just wishing for unachievable perfection in this space.

> I could shit on the state of academic publishing for days, but at the end of the day, there is an overall positive direction in this 'academic endeavor': the creation of knowledge.

That I agree with, and I definitely complain more about the private market than scientific research because of that :).

> I find this direction much lighter on my consciousness than "generate value for the shareholders"

True, I feel the same.

I'm very interesting in knowing what are your motivations to switch towards "something less altruistic". Could you share some more details? Maybe it really is that no matter what you do, no matter how meaningful, it'll always become a boring chore? Maybe I didn't have an meaningful job yet, so I'm not disillusioned?

I see in this thread lots of people who have their security level established and are now looking to fill their social level. I feel inverted on the subject and let's just say Maslow's order matters.

* top-tier peer-reviewed conference.

* paper abstract and introduction.

If I am not convinced that the problem is interesting by introduciton, I will skip it.

I mean if you're a programmer and want a job fighting cancer, maybe pick up some ML skills and find a way into that industry. But short of getting your PhD in medicine or a law degree fighting injustice, we're all just a different sort of coal miner providing the things society needs to keep chugging along. And there's no shame in simply being a productive member of society. As far as societies go, we're better than all that have come before.

Yes. The most rewarding thing one can do is to figure out a way to change that.

It's sad that our brightest minds don't want to understand this, and that they'd rather think about how to make people click ads.

The reality of the situation is that, if I'm not so drained from the mind-numbing soul-crushing CRUD work that I feel like doing more when I get home, I'm sure as hell not going to work on anything remotely related to my employer's line of work. I get quite enough of that at work. I'm going to work on problems that matter to me, and that seems to be stuff that really doesn't matter to my employer.

Boring problems are quite manageable when they are also important and meaningful. The general lack of purpose most of us find in our jobs (apart from 'earning money') is the real problem.

For instance, patient outcomes from a feature you built in an electronic medical record get published in major newspapers.

Now that sounds meaningful. Email me if you agree.

It's a good idea. There should be an easy way for a small team to capture the value that their open source project creates. There are a variety of crowdfunding platforms like Patreon that are doing something similar. CodeMill and Bountysource are paying developers directly for adding features.

Maybe a software repository site, where users must watch a 30 second commercial, before they can download the software. Then a percentage of the profits made from selling that commercial is shared with the programmer.

Sounds like sourceforge, but I don't know how that site is monetized, or if they share the revenues with the project creators.

Another recommendation is to have a look at 80000hours.org which is a great resource for finding the best work opportunities out there! I think they even provide some coaching :)

One might ask: what does the alternative look like? From what I can tell Marx is a bit wishy-washy on that point.

Hayek more helpfully explained that the market provides a mechanism for determining value, not virtue. That it works, but doesn't uplift. And that trying to force it to will leave with neither -- you will need to find your meaning in something else.

The market's notion of value is weighted by wealth: making coffee for the rich is "valuable", meeting the basic needs of the poor is not "valuable." This makes the market notion of "value" very different from just about any other notion of value in the world. If you muddy the distinction, or let someone else muddy the distinction, markets become almost tautologically awesome, because the market notion of "value" is something that markets are very good at optimizing.

There's a lot of privilege talking about what is and is not valuable, but its worth pointing out that its only a very recent, probably very temporary, phenomena that coffee is all about rich people massively overpaying in public as an act of conspicuous consumption at Starbucks, for some decades previous McDonalds and other fast food sellers made more total money selling coffee than hipster coffee shops.

There are some market bifurcation events going on, as relates to income inequality only increasing over time. For example the number of independent bicycle shops has been dropping for decades while average revenue and profit increase... thats because the market for $2000 adult bikes is extremely high social status but also very small, leading to the inaccurate observation that "there's no money selling bikes to poor people". However, sales and revenue for poor people bikes at department stores, walmart, etc, are really pretty good, its just that you can't buy a poor person bike at a cool trendy rich hipster independent bicycle store, only from extremely uncool mass retailers. There are absolute tons and boatloads of money in selling bicycles to poor people; the money is inaccessible to the "cool kids" is the only problem.

Food deserts are a similar thing; the point of talking about food deserts is to point out the individual noticing them is a really cool person.

Perhaps a HN car analogy where nothing could be cooler on social media and more valuable than bragging about selling Tesla cars to very rich people, but you won't make even a tiny fraction as much total revenue as the incredibly uncool act of selling commuter SUVs to suburbanites.

Being cool and trendy on twitter is important, but you can make many times as much money by being very uncool. To some extent the whole meta overall topic is conspicuous consumption at its finest, along the lines of I'm rich enough and privileged enough to pretend deeply uncool (as defined on social media) market opportunities literally don't exist.

Hayek understood that what we think about prices and what we think about ethics are distinct things. He didn't see rightness as flowing from markets. He saw it as a discovery process for allocations of resources.

> One might ask: what does the alternative look like? From what I can tell Marx is a bit wishy-washy on that point.

I have no clue either. But it feels to me as if we're turning humanity into a single superorganism, with each one of us being just a cell, except all those cells have brains capable of wishing they'd play on the same level as the superorganism.

Related: I've never had a libertarian bent. I'm known to argue for coordination over individual freedoms. I frequently wondered, just what is it that makes many people say that we're in chains, and (here in the West) we have so little freedoms left. I think I finally realized where this could be coming from.

I've been spending some time recently going, in my head, through a list of activities I'd find exciting. I quickly discovered that most of them are either impossible or illegal, especially when living in a city. Someone (I think pg) wrote that as we entered the industrialized age, we had to invent the whole fake, hollow thing called "exercise" to replace what we've lost in the change of lifestyles. I'm starting to see it as more general pattern - we're losing many more things we could do, and replacing them with structured activities available on the market. They're similar, but also feel fake to me.

> Hayek more helpfully explained that the market provides a mechanism for determining value, not virtue. That it works, but doesn't uplift.

That sounds about right (I suppose I should start reading the foundational texts of economic thought at some point). I do not deny that the scheme I described, in which your only role in society is to pick something profitable to do, doesn't work. It generally does (with all coordination problem caveats of the market in mind). But the meaningful work, the virtuous work, the uplifting work, seems to be distributed scarcely and randomly throughout the market space. Much like in the past, you got to be a king or a peasant depending on the lottery of birth, nowadays you can be involved in civilization-changing work, or forever toil on the margin, depending on random chance. Is there even a place for fining meaning in contributing to society? Is there any agency left as to how to contribute? Are we supposed to seek either of it?

I have no answers here. Only a topic I'm trying to think my way through, hoping to find some meaning on the other end.

I wish you luck.

I'll be sure to let HN know if I ever come to any lasting conclusion.

https://wiki.lesswrong.com/wiki/The_Fun_Theory_Sequence

What has always tied Marxism in knots has been the tension between waiting for the inexorable laws of history on the one hand (and sometimes trying to give them a hurry up) versus on the other hand agitating for reforms. The former see the latter as delaying the revolution. The latter see the former as something between dreamers and jerks.

I think that history, having been somewhat lawless after all, makes the reformers look better.

The thesis of capitalism is that acting only in self interest is good for the collective, i.e. your own capital is all, i.e. we can abstract everything away with money.

I think you're completely right about the problems with above dogma and hope that you don't stop looking for a way to do something you find meaningful.

Step 2. Push for creating more and more social infrastructure. So if people want to, they can choose to not be part of the box

I do that a lot on HN. I've been considering blogging on this topic, but I feel I suck at writing, and people may not want to listen to confused ramblings of a random programmer -.-

> Step 2. Push for creating more and more social infrastructure. So if people want to, they can choose to not be part of the box

I'm a noob at that. Need help.

This nice run has come to an end for me and my team and it's made me realize that getting to work on nifty, complex algorithms and do real CS is not the default. To say the least.

I'd like to try to figure out how to do it again: the "algorithms startup" is a fun thing to do. You aren't going to be raking it in as having the 'nice library to do task X' doesn't allow you to capture huge amounts of value relative to the people selling boxen/UI/SAAS/etc. that wraps up your library. But you can work on a decent scale.

I am thinking about blogging about this; generally I've been sticking to tech stuff on branchfree.org so far but having done this successfully once makes me wonder if it could be done again.

I'd probably read it.

Submitted to HN, and on my blog at https://branchfree.org/2018/06/12/some-opinions-about-algori...

Where would you like to work next?

US has it. They live and breathe tech. 5 largest companies by market cap are all Tech. They eat other tech companies for billions of dollars. Everyone and their dog in SV has a startup on the side. There are conferences on how to invest in startups. Some people just exclusively invest in startups. It’s crazy.

Sydney also has a tech brain drain. I lived in Sydney for 10 years and most of my friends moved out here too. Just better opportunities.

Sydney is such a nice place to build a startup though. Sucks that the ecosystem isn’t there. Vancouver has a similar problem too.

It is a nice place to live, though, and the Atlassian guys proved it's not impossible to build a pretty kick-ass startup here. There's also a pretty decent pipeline of smart people around. Part of my hope is that you can do smaller-scale stuff here without the Go Huge Or Go Home mentality of SV. What would be interesting is to work on things that interest me, make some money and influence the field. I don't think SV is mandatory for that.

Yes, please do.

I think it's all about the motivation and setting your expectations that you won't make it every time.

Do you have any updates on that water-filtering molecule you showed us then?

I stay away from webdev. I did a couple of small projects but the web was just an outer API, the main work was done by native code running on CPUs or GPUs.

I stay away from enterprise systems, or fin.tech, or ad.tech. I removed enterprise-targeted keywords from my resume, Java, J2EE, SOAP, SQL. I have some experience with them from my early career but I don’t want to use them anymore.

Also, for many years I’ve never worked in/for huge companies, i.e. google/amazon/microsoft scale. My employers/clients are/were mostly small to medium companies.

What do I do? Almost everything else. Native desktop, native mobile, embedded, GPGPU, robotics, GIS, videogames, CAD/CAM, streaming media. Many startups, and small to medium companies, have problems to solve that I find interesting to work on.

There’re still boring parts in my work. But because of my experience I usually find ways to minimize the time I’m spending on them. The most generic advice, pick the right tools.

I also like to abstract away the non-sense part of a problem, so that I can focus on the real intellectual puzzles.

Some boring problems need to be solved anyway, and they can’t be abstracted away. Just when you’re picking the right tools, it shouldn’t take too much time.

For example, that CAD/CAM app I’ve developed has a lot of very interesting algorithms inside. But the software still need to appear on the user’s PCs somehow, i.e. we needed an installer. I’ve picked http://wixtoolset.org/ It’s mature, well supported, first-party i.e. made by Microsoft, open source, integrates nicely into visual studio. Because of these things, I’ve spent less than a week in total over the course of 1.5 years working (=develop, test, and support combined) on the relatively boring installer part of that software.

The installer was just an example, many commonly encountered boring problems already have good solutions. E.g. need RPC, pick .NET and WCF. Need to store structured data, pick the highest level library that fits other requirements (despite I generally don’t like SQL, I still know it and I’ve used sqlite a couple of times when it fit well). Need to expose JSON HTTP API on MacOS without much performance constraints, pick Python + CherryPy, the Python is sufficiently high level and it’s already installed there. Without picking the right tools, I’d spent a lot of time solving these boring problems.

I think the issue here is order: solving which first. Algorithms motivate you, Handling interface demotivates. So start with the motivator.

The transition for me came when I started a new job and my first project was to fix a slow batch job. Then I managed to get a second performance improvement project and soon I was the performance guy focusing entirely on performance projects. I really loved it.

Then through side projects targeted at problems I saw at the company, I built valuable tools using new open source tech and I got made tech lead of a new platforms team, solely focused on resolving technical problems and improving the system. Since then I have been an R+D engineer and now starting as a big data developer at the same company.

I am no genius, failed my interviews to get into big tech companies. I am a pretty big fish in a pretty small pond, I am known as one of the best technical guys in the company, which opens many doors and gives me a lot of freedom to move about inside the company and get some of the most interesting technically focused projects.

I work as a software engineer for an ISP. We're a startup and so of course our mission is to disrupt the industry. The great part is we're actually doing it!

Fun stuff I've done so far:

- Wrote a library to find the best available subnet given our set of IP blocks, set of existing subnets, and the desired subnet size. This may not sound like a big deal, but we've shown our IP block utilization to other ISPs and they are blown away that we achieve a perfect 100% utilization of each block. (Typically at ISPs of our size this is done manually and so there tend to be a couple leftover subnets in each block which are too small to use).

- Wrote a monitoring system for OLT hardware which A) does not provide out-of-the-box monitoring and B) does not have an API. We actually managed to build a robust system on top of its CLI interface. This is huge for us. If there are hardware or fiber problems, or if a customer simply unplugs their modem, we know instantly. It can even pinpoint where the problem is.

- Wrote a system which is able to talk to our OLTs, core switches, core routers, and customer modems to do things like provisioning or upgrading/downgrading our customers' bandwidth. Again, there is nothing even close to an out-of-the-box solution for this. You have to build real software!

- We have some upcoming products I can't share, but one of the most exciting things right now is that we are focusing on helping our customers troubleshoot their networks. This is a surprisingly complicated thing to do while providing a top-notch customer experience, and we think software is part of the answer.

So, do I feel like I've been using "computer science" knowledge? Not really. Closest thing would be using binary trees to do subnet stuff.

But what I have been able to do is flex my software engineering muscles, go back to first principles, and design software from scratch. No CRUD, no huge frameworks and package managers, just plain old fashioned programming.

(Ok, we do have CRUD and framework stuff. The web browsers have to talk to something!)

What kind of equipment do you use mainly ? Cisco, Juniper... ?

Here is our web site: https://pilotfiber.com/

Here is our Who is Hiring post (contains my email):

https://news.ycombinator.com/item?id=17209386

I have been extremely fortunate in my career. I'd say I won the interesting lottery. I started very early by today's standards, taking my first programming class at the local university when I was in 5th grade, back in '76.

At 17, in '82, I had a game company selling nationwide in Sears & KMart a few Vic-20 games I wrote. That got me a beta testing role with the unreleased Macintosh. Which got me to move to Boston, where I worked for Mandelbrot & DeVenney on the initial publishing of Fractals.

Basically, I've been fortunate to be an early developer for a series of advanced technologies: - initial 3D graphics research community (back when the question was "how to do 3D graphics at all"), - initial GUI developments (before Windows 3.1, there were a lot of competitors), - early real time mapping / navigation, I worked on one as my first post graduation job - early streaming media, I was writing media players for CD-I/CDROM/Network back in '90-'92 - early 3D gaming consoles, I was an OS developer for the 3DO and the first PlayStation - early 3D games, lead developer for games for 15 years - VFX, left games and worked on 9 VFX heavy feature films at an academy award winning studio - startup, left VFX and spent 7 years doing a ML based 3D reconstruction of photos to realistic 3D avatar web service - Facial Recognition, closed my startup and now I develop facial recognition software for an industry leader

There is not time enough in a day to explore all the stuff I've done or am interesting in further pursuing. Luckily, I am one of those that sleep less: 5 hours leaves me groggy, and 4 is just right. So I spend my daze 10 hours coding for work, and another 4-6 pursuing "mad science".

Is it really possible to sit for that long everyday and not suffer from it. I know I'm probably focusing on the wrong thing but just the idea of being able to code for that long everyday without suffering any physical problems would be reward enough for me.

It can help a lot of people like me who are struggling to work even 5 hours a day at just 37.

Still thanks for the reply, I am now seriously thinking of getting a standing desk.

<back story> When I was 16 I was a fairly muscle bound half geek / half athlete. To give you an idea, I was a running back on my high school football team, and a tackle and two linemen and I would have to shampoo one anothers head's after weight lifting practice because our arms could not reach the top of our heads from being so buff. We were muscle bound jocks in Iowa, what can I say...

Well, I had a bad tackle when running with the ball during a game, and I crushed 3 vertebra, doubling over backwards with my hips pushed backwards till they touched my shoulders. The stadium had a collective gasp and then silence as I was seen on the field broken. Long story short, bolts in my temples and hanging vertically for 6 weeks healed my young spine with only weather changing aches. However, I went from 230 lbs to 130 lbs, and it took another 6 weeks of therapy before I could walk.

That was the start of a decade of no athletic activity and nihilistic punk rockerism mixed with quite the intellectual journey as I left Iowa and became a part of the Harvard/MIT/BU 3D graphics research community. (This was the early-mid 1980's)

It was not until I was 30 that my then girlfriend, now wife, got me to start morning walks, then light jogs, and finally an actual return to an athletic lifestyle. To give you an idea, at 30 I could not jog a block without heavy panting and sweats. After a year I was able to jog 1 block. It took whole year to do a block without dying! During that year, I'd get delirious and throw up if I pushed. But the persistent and incredible patience of my girlfriend slowly got me back into what is considered "normal" shape.

Once I'd refreshed my previously athletic muscle / body memory, all kinds of positives kicked in like a general optimism, a general active energy level, and greater interest in everything in general.

However, this was tempered with realization that if I slacked for even a few days, I'd get depressed, and if I went a week I'd start having bone/joint chiropractic misalignment and then I'd be injured and unable to be active at all.

So, now I have a strict 3 days (at least) per week hard cardio. I ride my bike about 8 miles, living near a bike path in the San Fernando Valley. Plus I have a yoga mat next to my standing desk. I need to get a newer computer, as a recompile of my work take 10-12 minutes. But I exploit that time working out, so maybe I don't want a new development computer...

I've never had any injuries but doing long hours siting all day without exercise has taken a huge toll on my back. I'm not overweight but my lifestyle is very sedintary :/ Long story short lately I've have my issues become so much that I've had trouble with simple things like putting on pants, getting in and out of the car, sometimes even sleeping. Have done lots of mris, xrays and months of physio and pain keeps coming back. Also religiously did the exercises prescribed by my doctor but nothing is helping.

I'm kind of like at that point in your 30s where even jogging for a block is a nightmare for me. But reading your story gives me hope. I'm 37 and just learning that taking health forgranted for work has been a big mistake.

I will try to research some cardio exercises that won't aggravate my pain and get started. I love your idea about doing things during code compilation. What a blessing in disguise that is. I already started that today doing some SI joint exercises that I can do while sitting. It's a start and I now to get involved with beginner cardio next.

I appreciate your thanks. I could write a book about my career and maintaining health in both physical and mental aspects - being a developer can be a hell of a mental game too. If I were not so busy writing code, I probably would. Developing my ability to live like this has taken decades, and now in my middle 50's I am peaking in multiple aspects: I know how to code an immense number of highly complex things, my mental game is such that I am coding like water pouring, and my work is offering me the freedom to use everything in the creation of a new flagship application for the company. I'm simply having too much fun and getting too much done to write about it at length. I do post on Quora.com quite a bit though.

Anyway yes it no ordinary thing you're in such good health and are putting 10 hours a day at 50 years while some of us are struggling lot younger.

I'm sure you already have done quite a bit of research that lays ahead of me. A book does sound like a big commitment but if you have a blog I'd surely hope you would put your story and what you did to fix yourself and how you're so productive.

A lot of people would be inspired and helped by it I'm sure.

P.S. just read this book called 12 rules for life (huge bestseller) and the author starts it by saying so this book started as a result of my quora post :)

BBSs?

What was your first computer?

I know this is a highly speculative question.

As a discipline, we're too young and our technology is too immature to support a true split between engineering/science and computer programming.

If you want to work on interesting problems you're passionate about, there's a few ways to do it.

First, you can do research. Years ago I joined a lab as a professional researcher to follow my passion for machine learning and sensor networks. It's not the easiest path, but labs need disciplined programmers to complement the skills of PhD students.

Second, you can consult. If you're passionate enough about a topic that you have encyclopedic knowledge, I guarantee there's someone out there who will need that expertise and will pay for it. It might not cover all your bills, but it gets you exposure (and potentially a job if they need your skills a lot).

Third, you can start a company. I followed my passion for the challenges of learning from sensor data at scale into entrepreneurship and eventually started a company, http://sentenai.com . As a technical founder, I chose carefully the technologies we use (Haskell) and the problem we were tackling (making it fast and natural to combine multiple sources of time series data and find temporal patterns). You still need to solve somebody's problem and build a business, but you can at least lay the foundation.

At my previous company we ran SSH servers on our gateways for server->client pushes and it ran into the fewest problems with enterprise firewalls. We also ran our own mesh wireless networks a lot using our own gear. I'm guessing that's probably not an option, but when there's no admin running around looking for unauthorized wireless signals it's quite efficient.

Now all of these require a local host machine (or several) behind the firewall that do the brunt of routing and orchestration for pushes. There's a good reason for it though: It's a lot easier to get several hardened systems authorized to cross the firewall than a bunch of IoT devices. And if you can't get that authorization, you can put a cellular connection in the box or boxes more cheaply than one in every deployed device.

Happy to chat more if you want specific advice about certain situations. I also spend a lot of time investing in and mentoring Boston-based IoT startups so I have seen a broad cross-section of situations that can happen. brendan@sentenai.com

I don't really approve use-cases like running a webserver to control lights on ESP8266 where simple TCP sockets would do.

Btw I've written an implementation of ZRE protocol which you might possibly find interesting - https://github.com/vpsfreecz/haskell-zre

The amplifiers, processing, and data storage used to take up most of the space of a small cargo plane. Then it got down to four half-racks. Now it is the size of a dishwasher.

We hope to get it drone-sized. It is a never-ending series of interesting technical problems that span the entire STEM range, including RF, network, system, electrical and mechanical engineering, HPC, data science, voodoo-mathematics I will never understand, system administration, and plain old software development.

Everyone here is expected to cross-train in various disciplines. My background is CS but since starting here I have done electrical and mechanical engineering work including diagnosing micro-fractures, visible only through a microscope, in connectors on a bunch of VPX chassis stuff, caused by bad soldering work by the manufacturer.

On top of all of that once or twice a year we all get to go on operational deployments of the system. This year we went out and imaged thousands of miles of coastline, 6 engineers crammed in a tiny airplane hopping from airstrip to airstrip in support of a coastal erosion monitoring program. Data scientists are expected to haul around amplifiers and radar engineers are expected to live patch code while in the air.

The only reason I got this job was because I was a bored programmer teaching a Linux course as a side-hustle and a bunch of engineers came through and started asking questions way beyond the scope of the class. They were radar guys struggling to learn Linux and a couple of post-class drinks later I had an interview.

I guess my only advice would be to have a side-gig and network as much as possible, peppering everyone you meet with questions about what they do.

If what they do sounds cool, ask for an interview.

My initial consideration is to develop and host a sort of public Wolfram Alpha, but specialized for climate data queries and visualization. I'm open to other suggestions, as I'm currently in the process of architecting the system and reaching out to climate scientists for feedback on what they need that isn't well-served yet.

Once all the infrastructure work is done, I'd be happy to contribute to climate modeling more theoretically. But for now my prerogative is to develop a central repository for as much climate data as possible, normalize it as reasonable, and store it in a queryable format.

There are interesting technical problems here due to the scale of the data, the variety of its native (raw) formats, the frequency with which each source updates and the types of information involved. But speaking directly to the title of this post: I don't think anything I'm doing is strictly an unsolved technical problem, it's just a complex technical undertaking. There are plenty of data serving and processing pipelines for this that have already been proven capable.

There's still a lot left to figure out, even storage formats aren't fully solved.

Feel free to reach out to me individually (email in profile), I'm working on operationalizing climate data like this and would like to look at this together!

High-level questions:

- At the risk of birthing a page-long subthread on ZFS-vs-everything-else... what storage solution are you using, and why?

- What sort of hardware are you using? (This is a non-catalyzing question, and is just out of curiosity)

- How did this get started, and how are you managing this?

- Will you ever be interested in accepting donations or funding? (Including on a voluntary basis; and including with clear stipulations/structure about management)

- What sorts of decisions/motivated this initiative?

More focused:

- Do you have any interest in creating this as a "community hub", with a central code repo that data scientists can push updates to that then get run on the cluster? If the visualization data (or prerendered bits of it) are openly cached/accessible, having the code that generated the data equivalently open/available could be interesting.

- What kind of availability/openness are you looking at with data? (TL;DR translation: rate limiting)

You may already be aware (very likely), but AFAIK archive.org is interested in this kind of thing. Interesting bunch of people, but the various projects do like climate data, and they have a few hundred PB of space, FWIW.

> - At the risk of birthing a page-long subthread on ZFS-vs-everything-else... what storage solution are you using, and why?

Right now, XFS and HDFS. This is primarily due to my prior experience with it; I have considered ZFS, and might move to that later on. I may also dispense with HDFS for shared storage and to reduce the redundant JVM memory footprint. I'm attracted to ZFS for compression and error correction features, I just don't have any actual experience using it.

> - What sort of hardware are you using? (This is a non-catalyzing question, and is just out of curiosity)

Four bare metal servers, three of which have dual Xeon E5-2630 V4s (10 core, 20 thread 2.2 Ghz), one of which has an i7-6900K (8 core, 16 thread 3.2 Ghz). The latter server has 10TB SATA SSD capacity and four GTX 1080 GPUs. The other three have 2TB of NVMe SSD capacity. Each of the four has 128GB DDR4 2400 Mhz RAM. Currently all storage is local to the hardware, but there is a SuperMicro SC847 for future expansion. Every server has dual 10Gb/s SFP+ connections in bonded 802.3ad LACP, and are connected via a 16 port 10G network switch in the same rack. I think that's everything at a high level, off the top of my head.

> - How did this get started, and how are you managing this?

I joined John Baez and a few other scientists/researchers working on the Azimuth Climate Project, which sought to preserve critical snapshots of climate data in the event of mass defunding. Later on I decided to take this a step further since those snapshots were becoming very out of date and the multifarious data repositories were never very well documented, organized or normalized. Not sure what you mean by "managing this" though.

> - Will you ever be interested in accepting donations or funding? (Including on a voluntary basis; and including with clear stipulations/structure about management)

I initiated the process of starting a formal nonprofit, but not for the purpose of soliciting donations; rather, just so that it would be clear it's a noncommercial activity. I might be interested in that kind of thing - in either way you mentioned - once I have exhausted my own reasonable resources for the task and have to significantly expand.

> - What sorts of decisions/motivated this initiative?

It began with reading worrydream's blog post, "What can a technologist do about climate change?": http://worrydream.com/ClimateChange/. Not much more to it than that.

> - Do you have any interest in creating this as a "community hub", with a central code repo that data scientists can push updates to that then get run on the cluster? If the visualization data (or prerendered bits of it) are openly cached/accessible, having the code that generated the data equivalently open/available could be interesting.

Yes, that's an interesting idea.

> - What kind of availability/openness are you looking at with data? (TL;DR translation: rate limiting)

Well everything is intended to be extremely transparent, so I'm going to open source all software (infrastructure, development, research, etc). I'm also going to keep all data open. In practice there will probably be a limit of 1000 or so queries per day per IP address, but I'll burn that bridge when I get to it. It really depends on how "real time" queries end up being, and how much abuse the system actually receives.

Everything is duly noted; I'll just expand on some bits.

> I joined John Baez and a few other scientists/researchers working on the Azimuth Climate Project, which sought to preserve critical snapshots of climate data in the event of mass defunding. Later on I decided to take this a step further since those snapshots were becoming very out of date and the multifarious data repositories were never very well documented, organized or normalized.

Archive.org is very probably interested in this sort of thing then, FWIW.

> Not sure what you mean by "managing this" though.

Heh :) I was curious what sort of work you do in order for this initiative, which appears(?) to be somewhat of a side project, to be viable in terms of budget and adequate spare time. I'm also interested in storing/working with somewhat large amounts of data too (one project I want to try at some point is implementing an infinite browser cache so I can "Google" the content of every version of every webpage I'd ever visited), so I definitely want to optimize for something that doesn't require much time :) (don't we all)

> It began with reading worrydream's blog post, "What can a technologist do about climate change?" ... Not much more to it than that.

After having read that page I now understand the sentiment of that statement. Major TIL :)

And if only all webpages were that well designed... (The interactivity was a bit of an information overload, but I really liked the layout.)

Writing complex code to solve a gnarly problem is often the wrong path, because it's harder for coworkers to understand the code, and harder for future maintainers to keep it correct. Thus it's often better to encapsulate the hard problem in a nice interface, and possibly release it as a separate library.

That's where most of those libraries you're gluing together come from. A tough bit of code that needed writing but also encapsulation.

A tougher problem than most CS problems you'll find while writing CRUD apps is getting architecture right, so that adding new features isn't an uphill struggle and doesn't involve piles of boilerplate. Getting the aesthetics right, making the code clear so that anyone can maintain it - this requires experience and the mastery that comes from it pays off in the longer term, because you can more easily leverage other people's efforts too.

If you can't find solace in mastering software design, you're better off moving away from CRUD apps and move to a developer of tools. Tooling, whether it's compilers, OSes, databases, whatever - has more interesting and deep technical challenges. Watch out though: tooling often doesn't pay particularly well because developers enjoy making tools themselves. They'll prefer to build their own shoddy hammer 9 times out of 10, precisely because they're in the same boat as you right now: they don't feel challenged.

Side projects are good for your brain and in the long-term good for your career (lest you stagnate with old technologies). I recommend having this outlet. On the plus side, having a somewhat interesting job + really interesting side projects gives you both good money and intellectual freedom.

If you want to solely focus on interesting technical problems, I'd recommend working in a research lab or going to academia. On the downside, you may lose that nice paycheck, but your curiosity will remain forever piqued!

If you join an industrial lab, rather than a lab in a university, you can keep the pay but still do the same thing as the academics.

I've been running DNSFilter for 3 years. We're (still) facing interesting texhnical challenges - ranging from BGP community strings being used to balance a global anycast network, to handling volumes of time series data [1], to designing a golang daemon to ssl sign incoming requests on the fly for domains we won't know ahead of time, to figuring out new ways to pass data in DNS, and support cutting edge standards like DNS over TLS, DNS over HTTPS.

I was just saying to my wife the other day how ridiculous it is that most things we do are technically challenging, and you have to think outside the box how to get it done -- while I was implementing a block page for user agents which requires a 19 step process to determine the user's DNS Agent ID, and display the right logo on the block page ;)

Some might dismiss these challenges due to our use (abuse) of DNS for purposes it was not intended for. Fair. Also, welcome to the real world, and we're solving problems are customers are asking for... and just figuring out interesting ways to get the job done.

1: https://blog.dnsfilter.com/3-billion-time-series-data-points...

I'm all for work-life balance and working on passion projects on the side.

But do that many people really think of their developer job as a salary and a daily grind?

Maybe because I remember all to well what jobs I used to have:

- farm hand (heavy work on smaller less automated farms)

- construction (noisy, wet, dusty etc)

- security

- help desk

Being paid 8 hours a day to work on borderline interesting tasks, getting free lunch, being paid to study etc feels great. Oh, and I can work from home if I need to.

I also make my work interesting by putting some effort into it.

Work with great people. Also code reviews make sure new dependencies don't usually in unless the team agrees (we are a reasonable bunch).

Standardized on Angular. It seems to me Angular has less churn. Yes: there is a life cycle and we have to follow the releases but it is mostly well documented.

Most developer jobs are for CRUD apps, and you'll rarely be doing greenfield development. There's not a whole lot of room for passion there. If you just treat it as a job, though, rather than something you have to force yourself to be passionate about, it doesn't need to be that interesting. Once you make a living, you can satisfy your creative urges elsewhere.

I think those people live the work life equivalent of hitting all green lights.

There is absolutely no reason to be miserable at your day job. You should be preparing for superpowers if not actively using them.

I recommend watching lectures/talks during work, skipping meetings and reading from thought leaders instead of working hard to reach deadlines. Everyone is connected which means you can read and watch what Ian Goodfellow has to say about unsupervised learning or see what the issues with the world computer are from Vitalik himself as easily as listening to your marketing department figure out how to grow 10% in the quarter.

For the first 7 years I worked as a "programmer", ie. writing code, a lot of C++, and of course web stuff. About 6 years ago I (accidentally) became a "data guy". I do a lot of data infra/engineering, a lot of stats and analytics, and lately ML. Also I do a lot of management stuff and hiring.

I'm lucky, this is a pretty sweet time to be working for me. My skills are a perfect match for the work, I'm able to do impactful work and can get paid for it.

I don't really care about interesting technical problems that much, in the sense that I don't want to solve an interesting technical problem just because it's interesting. What matters is having an impact on your users and the business.

I think if you're looking for those types of technical challenges, it's better to contribute to open source code, rather than doing it at the day job. At the day job, you should always choose the method which makes sense to move fast..

Business: here is a continuous stream of problems, most of them are boring, some of them are hard, but you have to solve them all regardless.

Beside that we're also building vpsAdminOS (https://vpsadminos.org/) for running LXC on top ZFS. At https://vpsfree.cz we currently run openVZ with the intention to migrate to vpsAdminOS which is currently tested by members.

In my experience you gotta sacrifice stuff (like well paying job) so you can do stuff you really want.

The most exciting thing at my old job was this DFS I had to do once.

That said, there are very interesting tech jobs out there - such as scaling boring old CRUD apps. The high frequency trading space also has many interesting problems. Imagine working on a system where you start to care about nanosecond optimizations. I interviewed (and failed at it, lol) at Optiver [0]. You should check them out.

[0] https://www.optiver.com/na/en/

Finding jobs like this isn't easy and mostly requires luck - my last job was very different, it was just maintaining a plain old CRUD app. Not the most mentally stimulating work, but it's probably 90% of the work that gets done on a day to day basis by software engineers.

Try looking for people who are solving a problem that is technical in nature. Business problems use technology as a tool, technology problems require development of brand new tools or the use of existing tools in novel ways.

Now I’m working at a startup making a collaborative VR modelling tool which applies all the same problems to a more ‘serious’ end.

Both still have a fair share of plumbing work as well. But that can be a welcome relief. I find solving hard problems (particularly under pressure) can become obsessive and make it hard to be present with my family and sleep!