we absolutely do, the library itself is designed to be extensible. we are currently working on adding webhooks as one of our sources. are there are any specific connectors/sources you'd be interested in?
I have lots of HTTP endpoints that we poll with a cursor but actually the underlying data is very large (we work with snapshots of it) and updates very frequently and eventually we'll move to something else (e.g. interact directly with the underlying services with capnproto) so really it would just be useful to be able to define these sources ourselves. I'm working doing full-stack engineering at an HFT currently and we were thinking of using DataFusion to allow users to join, query and aggregate the data in realtime but I haven't attempted this yet (and to do so means integrating with what currently exists as I don't have time to rewrite all of the services).
It's bittersweet. It seems likely to me that the US government didn't really want an open trial due to the possibility of scrutiny and that indefinite detention without trial followed by setting the legal precedent that aiding and abetting legal whistleblowers is a criminal conspiracy was their goal.
I'm surprised that an executive or lawyer didn't realise the reputational damage adding these clauses would eventually cause the leadership team.
Were they really stupid enough to think that the amount of money being offered would bend some of the most principled people in the world?
Whoever allowed those clauses to be added and let them remain has done more damage to the public face of OpenAI than any aggravated ex-employee ever could.
If I can sit down at any table in my house and get a multi-monitor setup without needing to buy multiple 4K screens then it'll win me over. However, in practice, I do not think the hardware will be high enough quality for me to want this yet.
These aren't "the rules of capitalism" they are just maximally self-interested hubristic behaviours by companies with monopolies. Capitalism doesn't have rules; states have rules. Capitalism has market incentives and stakeholders.
Capitalism is just a system based around the belief that capital appreciation is the ultimate end goal, and the ends justify the means. Everything is fine so long as the GDP keeps going up.
This is probably correct but I'd prefer that family don't read the conversations I've had, as even if I'm not saying anything too private, it feels too intrusive (it'd be a bit like reading my inner thoughts).
How could I look my wife in the eye, or expect my kids to grow up happy, if they knew their dad doesn't know how to use a regex to detect emojis in a string?
> Is it your intent to convince them that they don't need privacy?
Quite the opposite actually. My intent is to shed light on the fact that sharing information with OpenAI is not private. And you should not do that with information that you wouldn't even share with people you trust.
> Quite the opposite actually. My intent is to shed light on the fact that sharing information with OpenAI is not private. And you should not do that with information that you wouldn't even share with people you trust.
I'm not OP, but I think you're missing the point.
Privacy and trust isn't really a 1D gradient, it's probably planar or even spatial if anything.
Personally I'd be more willing to trust OpenAI with certain conversations because the blowback if it leaves their control is different than if I have that same conversation with my best friend and it leaves my best friend's control. The same premise underlies how patients can choose who to disclose their own health matters to, or choose who their providers can disclose to.
Same reason behind why someone may be willing to post a relationship situation to r/relationship_advice and yet not talk about the same thing with family and friends.
> Same reason behind why someone may be willing to post a relationship situation to r/relationship_advice and yet not talk about the same thing with family and friends.
I ask that you consider the people who use Reddit and the people who run Reddit independently. The people who use Reddit are not in a position of power over someone who asks for advice. The people who run Reddit on the other hand, are in a position of power to be able to emotionally manipulate the person who asked for advice. They can show you emotionally manipulative posts to keep your attention for longer. They can promote your post among people who are likely to respond in ways that keep you coming back.
OpenAI has a similar position of power. That's why you shouldn't trust people at either of those companies with your private thoughts.
You're assuming power comes with an assumed guarantee of use. OpenAI has vast amounts of power with the data they're collecting, but the likelihood of OpenAI using it against any individual is small enough that an individual could consider it to be outside their threat model (I'm speaking using security language, but I doubt most people go so far as to threat model these interactions; it's mostly intuitive at this point).
Your family has limited power in the grand scheme of things, but the likelihood that they may leverage what power you give them over you is much higher.
The IRS has vast power and is likely to use it against you, hence why tax fraud is usually a bad idea.
> OpenAI has vast amounts of power with the data they're collecting, but the likelihood of OpenAI using it against any individual is small enough that an individual could consider it to be outside their threat model
I think your use of the word "individual" is a bit weird here. I absolutely find it likely that OpenAI is doing individualized manipulation against everyone who uses their systems. Maybe this would be more obvious if you replace OpenAI with something like Facebook or Youtube in your head.
Just because they are using their power on many individuals doesn't mean that they are not using their power against you too.
> I think your use of the word "individual" is a bit weird here. I absolutely find it likely that OpenAI is doing individualized manipulation against everyone who uses their systems. Maybe this would be more obvious if you replace OpenAI with something like Facebook or Youtube in your head.
> Just because they are using their power on many individuals doesn't mean that they are not using their power against you too.
Yeah but at this point you're identifying individual risks and grasping at straws to justify manipulating* everyone's threat model. You can use that as your own justification, but everyone manages their own personal tolerance for different categories of risks differently.
*Also, considering the published definition of manipulation is "to control or play upon by artful, unfair, or insidious means especially to one's own advantage," I think saying that "OpenAI is doing individualized manipulation against everyone who uses their systems" is an overreach that requires strong evidence. It's one thing if companies use dark UX patterns to encourage product spend, but I don't believe (from what I know) that OpenAI is at a point where they can intake the necessary data both from past prompt history and from other sites to do the personalized, individualized manipulation across future prompts and responses that you're suggesting they're likely doing.
Considering your latest comment, I'm not sure this discussion is receiving the good faith it deserves anymore. We can part ways, it's fine.
Too much discussion about the Holy Roman Empire over dinner? People talk to get things of their mind sometimes, not the infinite pursuit of conversation.
My point was not that they should talk about the Holy Roman Empire with their family, but that they shouldn't share information with strangers that they wouldn't share with their family.
If you don't want your family to know something, you shouldn't tell it to OpenAI either.
The reason you wouldn't say something to someone is because you are afraid of the power that you give people along with that knowledge.
Your family is in a a position of power, which is why it can be scary to share information with them. People at OpenAI are also at a position of power, but people who use their services seem to forget that, since they're talking to them through a computer that automatically responds.
This is silly. It's not like OpenAI is going to find your family's contact info, then personally contact them and show them what you've been talking about with ChatGPT. It's just like another post here comparing this to writing a post on /r/relationshipadvice with very personal relationship details: the family members are extremely unlikely to ever see that post, the post is under a pseudonym (and probably a throwaway too), and the likelihood that someone is going to figure out the identity of the poster and seek out their family members to show the post to them is astronomical.
Is that truly interesting? OP does not have to care about what AI think of him. OP does notnhave to care about accidentally offending or hurting AI either. Open does nor have to care about whether AI finds him silly or whatever.
Normal humans care about all of those with their families.
Fine then. You don't want to find out your family the love you have for Roman empire. But you are a programmer, yes? So make an app that's just a wrapper for ChatGPT API's you're paying for and distribute that to your family phones. They'll use your OpenAI API key and each will have their own little ChatGPT 4 to query to. Have fun.
These issues are caused by (1) the buy-out by Elon Musk involving a large amount of debt, and (2) Elon Musk annoying "woke" advertisers who have subsequently deserted the platform.
The actual engineering hasn't been as affected or rather while it has been affected it hasn't been affected to the magnitude that you might expect given the size of the lay-offs.
I wouldn't expect too much change right away after a lay off that size. Any decent engineering team will have processes, workflows, CI/CD, etc... in place and if all the engineers went away today, most places would still run just fine for a while, maybe even a long time if the systems are set up correctly. The question becomes what happens next? How quickly will they be able to leverage new advantageous technologies? What happens when that rare thing breaks and you have no institutional knowledge? How do you solve difficult problems like content moderation? I'm curious to see where Twitter will be a couple years from now in terms of how it's engineered. I would expect a slow decline in expectations and results.
You are right in theory. But instead of trying to predict the future I'm trying to be descriptive about what has changed so far (very little) and why (mostly advertisers + debt driven by buy-out).
> The actual engineering hasn't been as affected or rather while it has been affected it hasn't been affected to the magnitude that you might expect given the size of the lay-offs.
Are new features being added? Honest question -- I don't use the product.
Keeping the lights on for a product with 10% of the workforce isn't shocking or new. We do it in this industry all the time. Can you iterate and ship with 10% of the workforce? That's much more impressive.
Features are both being added and removed. Actually, the main difference is that more feature are being added than were before, but this is being done in a more haphazard way. Sometimes things appear to break but are then fixed. It's not awful though, is what I'm saying.
Honestly, I could feel it. In 2023, I practiced ~200 leetcode problems and learned how to create an LLM but I felt like I was the least employable I'd ever been. It was very difficult to get interviews, and even though I managed to get multiple offers, many of these were at companies I'd not normally have accepted offers from. It took a long time to find a job offer I wanted to accept.
I do feel that there are signs of recovery now, so good luck to those that are still looking. I know it's very hard -- particularly if you're also supporting a family.
Yeah in past 20 years I would receive a job offer from nearly every company I interviewed with. Past few months have been really disappointing with rejections after interviews that I'd normally have said went great.
It's also true that you have now 20+ years of experience. The higher you go, the harder it gets - you might want a different role, etc etc.
If on the other hand you're competing with senior sw devs with 5+ years of experience, there could also be other factors to keep in mind (age, ability in very specific tech stacks, etc).
I think companies are still hiring, but more focused, unlike in 2021-2022 when they over-hired "just in case".
In-company recruiters are often quite helpful and accommodating, but, as soon as one single tech person gets involved, the temperature drops 30 degrees.
Standalone recruiters send me these breathless emails, extolling my qualifications, but, as soon as they find out my age, they ghost me. I have actually had recruiters hang up on me, as soon as I told them my age. I learned to just mention that up front, to get the ghosting out of the way.
Apparently, they aren't very good at math. I list 30 years+ experience, yet they seem to think that I'm under 35.
After a while, I just gave up, and accepted that I'm retired.
It's not the money; it's the "culture." Many folks, much younger, and much more inexperienced, are paid more than I ever made, in my entire career. I would have gladly accepted less money than I had made before. I don't really need it. The work is what interests me.
>I would have gladly accepted less money than I had made before. I don't really need it. The work is what interests me.
We need an acceptable way to express this, without desperately extolling you'll take less money. I think a lot of us "old guys" are in the same boat; I don't need 150k, I'll take 100k if the work is interesting, and I'm more likely to be loyal to boot.
The funny thing, is that I've been told that "Old people are just cruising to retirement," but it's OK to establish your entire business infrastructure on the idea that your young, energetic, engineers will not remain at the company for more than 18 months.
>but it's OK to establish your entire business infrastructure on the idea that your young, energetic, engineers will not remain at the company for more than 18 months.
Yeah, it's a tough nut.
It's understandable that SV companies want ambitious strivers that move on every 2 years, for the same reason many of these CEOs consider the "job done" as soon as they get "an exit". The life of the company is measured in months.
But if you're building a company for the long term, you need smart, loyal people who build institutional knowlededge within the company. This isn't something you can fast track, I don't care if you're the brightest MIT AI grad.
I was just struggling, this morning, with the Apple App Store Connect Web interface.
We're in the final stages of releasing an app, so we're spending a lot of time on that Web app.
It's...challenging. I know that they have big issues with security, privacy, and sheer scope (I'll bet they get millions of submissions, every day), but the site is dog-slow, the CDN breaks constantly, I need to refresh the page quite often, and they seem to forget where I was, the last time; necessitating that I follow the breadcrumbs back to where I was (I have admin accounts on several orgs).
I've released over 20 apps on the store, and have dealt with this, for a while. It's actually getting worse.
But Apple is a multi-trillion-dollar company. I think they could afford for this to work a bit more smoothly, and, quite frankly, I'm surprised, as they have some of the best, and most experienced engineers on the planet, working for them. These are the types of things that lots of sites seem to be doing quite well.
It seems that many startups don’t actually have a product, other than the startup, itself.
A “successful exit” means that the company is sold. The company is the product.
I have spent my entire career at companies that made actual products, for use by actual end-users. These corporations never had any “exit strategy,” because they were meant to be ongoing, perpetual, concerns. They had “future planning,” and “growth strategies,” that often looked a decade into the future.
Isn't age discrimination literally illegal pretty much everywhere? Proving it is always hard, but "they hang up on me as soon as I tell them my age" seems pretty clear-cut.
It's probably not even legal to ask in some jurisdictions.
I suspect that it's not so easy to prove with external recruiters, and that may explain the difference in demeanor between the two types of recruiter. The standalone ones aren't on the hook for $MEGACORP's brand and legal.
They just start saying "Hello?, Hello?, Are you there?," etc. It's a convenient way to hang up on people.
That's why I like the "make it illegal to even ask"-type of policy. At some of these American firms they asked me things like my sexual orientation and religion, aspects you often can't infer from appearance, and that always seemed rather odd to me – I'll just have to trust it won't be used to discriminate. If I don't tell this information I know they can't.
Things like age, gender, and ethnic group are harder because you can't always conceal that. Still, "don't ask, don't tell" during the hiring process seems like the best option here.
I don't lie; especially in my profession. It's a thing. I know that personal Integrity is considered a "quaint anachronism," in today's world, but I won't compromise on that.
You’re right not to compromise and even if you were less ethical, that’s something which is easily detected and could lead to being fired with no recourse. It’s very reasonable for a company to say someone who lies about simple facts can’t be trusted with anything else.
You certainly don't have to lie, but there's no reason you can't simply calculate your age differently. The common 365 days to a year for age isn't a universal given.
The higher you go, the more expensive you get. And the fewer places see the value you bring compared to a cheaper "senior sw dev" with 5 years of experience.
If you want to be paid more than someone with 5 years, you have to generate more value than they do, and you have to persuade the hiring manager that you can generate more value. The fraction of places that can see that is smaller than the fraction that can see the value of a senior over a junior. (On the other hand, for those places, the competition for the jobs is also less severe - there aren't tons of people with 20 years of experience on the street at any given time.)
So it takes longer than it did when you had 5 years of experience. But keep looking. There are places that will see the value in what you provide.
The job category you're looking for is "principal software engineer" or "staff software engineer".
The mere statement that a 5 year ‘Senior’ software engineer has more perceived value than a ‘Senior’ software engineer with 20 years of experience is just crazy. Let’s replace ‘software engineer’ with ‘electrician’ and then tell me this is sane.
A 5 year software engineer may have more perceived value per cost than a 20 year software engineer. Sure, if they have the same salary, you hire the 20 year engineer, but they don't. The 20 year engineer expects more pay, and (rightly) won't work for 5-years-of-experience wages. So you have to find an employer that perceives the additional value of the additional 15 years of experience.
Have you seen the salary requirements for these jobs? People are being paid way more than they were being paid 10 years ago. Nobody, and I mean nobody, is turning down jobs because of insufficient wages. The perceived ‘value’ that an employer sees is the ability of being able to exploit a younger engineer with ridiculous timelines and all night ‘coding sessions’ because they don’t have a family waiting for them when they get home.
> In 2023, I practiced ~200 leetcode problems and learned how to create an LLM but I felt like I was the least employable I'd ever been.
You did two superficial things and are surprised you didn't feel more qualified? Practicing / memorizing leetcode and having a rudimentary understanding of LLMs (if I were hiring, we're targeting advanced degrees with a focus in the space) -- yeah, probably not sufficient. This is a feature and not a bug as the employment market moves back to sanity.
I think this is very unfair. In the few months that I was looking, I put in more than 300 hours of study. It wasn't superficial and I didn't attempt to memorize anything — I wanted to understand fundamental techniques, and put a considerable amount of effort in to doing this, treating it as if it were my job.
Nobody owes me anything but that doesn't mean that I'm not allowed to point out that this is the weakest I've ever seen the market in my decade or so of experience of it. It's very tough for those that have been laid off. Particuarly those have to support families as I did -- I am a sole earner and have two children.
You could benefit from practicing a bit of empathy and not lying to yourself that the market has merely moved back to "sanity" without gaining a bit of recent experience in it.
> I put in more than 300 hours of study. It wasn't superficial and I didn't attempt to memorize anything — I wanted to understand fundamental techniques, and put a considerable amount of effort in to doing this, treating it as if it were my job.
If you landed a job, do you think this new ability to implement a CS 101 data structure would let you keep it? Help in your performance in a measurable way? This is what I mean by superficial. The same applies to the LLM work. You studied enough to where you could buzzword it in a 45 minute interview.
My response, which I'll be more clear with: Maybe it's not that bad that we're more careful and these superficial techniques are no longer adequate to get someone a job in this industry.
> I'm not allowed to point out that this is the weakest I've ever seen the market in my decade or so of experience of it.
Don't think anyone is preventing you from doing anything.
I did land a job making applications for an HFT firm and while I don't think that being able to implement data structures or write my own algorithms is necessarily what will allow me to keep this job, recently I somewhat regularly use the skills I pick up doing competitive programming exercises in my workplaces. Therefore, my answer to your question about whether it will help my performance in a measurable way is: yes.
Leetcode is not superficial when nearly all companies are using leetcode as a barrier to entry these days. Many of us with 10+ years of experience did not need to practice leetcode to get hired originally, but many would fail miserably today due to the time constraint and necessity to socialize/speak the problem while solving it in 45 min. Some practice problems took me all day at first, but after a while it became a skill of quickly identifying the type of data structure to use and solving it. It's great practice.
As for LLMs, it's probably the best thing to learn right now, especially by building something. I personally believe it will lead to an explosion in potential jobs. It's hard to describe but imagine a software engineer learning how to create a rest api for the first time after doing soap. And then integrating various apis to harness saas products, cloud deployments, etc back in the 2000s.
Then again I also talk to an ex-googler I met on Hacker News who built an LLM app and is now spending most of his time selling his saas product because he can't find a job either. That's a very positive go-getter attitude.
There's no enough information to extrapolate that fact. Were these offers median salaries? Bottom 25th percentile?
Lots of companies low-ball candidates because they are happy with Harbor Freight quality candidates, so long as they will work for Harbor Freight prices.
I'm the OP (of this thread). The main issue caused by the industry-wide hiring freezes wasn't necessarily income but work/team quality. I've been in the top-10% of incomes in my country for more than 5 years and the top-1% of incomes for a lesser period of time.
I was able to get offers in this percentile but the companies making these offers didn't usually hire at this level. If anything, I want to be around people that are more competitive than me, that put a lot of effort into what they do and that can easily walk into high paying roles. I definitely don't want to be the highest paid engineer in the room at a company that frankly doesn't need this.
I know this can be seen as entitled but I'm the sole provider of a 4-person family and willing to work hard and prove myself. I'm happy to say that I now have a higher income than I did before being laid off and at a company with more successful people than me and a higher ceiling, however, that doesn't mean I didn't find it significantly more difficult to get a job this time around than I have in the last decade.
There are people that are going to have a much harder time than I have.
This is a negative outcome of the Town and Country Planning Act. We've mandated by law that all planning is done upfront and due to the possibility of construction fraud provide detailed designs to building contractors that disallow them from substituting any materials or design choices without it being explicitly requested by the employer. In theory, there can be conversation between individual subcontractors and the project managers or architects, but in practice these are not always on site at the same time and there are significant costs involved in changes as well as issues with trust. Therefore, we effectively have a very expensive waterfall process that everybody is legally obligated to follow.
If it's your project, you can order the builders to reconfigure the ventilation ducts pretty much however you like.
Building control require that an architect sign off on things like the size of the beam holding the roof up. If you change that, the architect will have to sign off on the building as constructed before building control will approve it.
Planning permission is given on the basis of external appearance, so you can't change too much in that area - but go look at some planning applications, they're very light on details.
But other than that? If it's your project and you tell the electrician putting in a socket to put it somewhere else, he'll tell you if it's legal under the wiring regulations, and if it is, he'll move it.
The main reason that in large projects builders are required to build exactly to the plan is to stop them substituting cheaper materials. They're under great pressure to do that, because if you allow your builders to replace expensive zinc cladding with cheap ACM cladding, and you ask several builders to tender and choose the cheapest builder, you'll always choose the builder who quoted on the basis of the cheap ACM cladding.
In a small house you can move outlets. In a large building the space for every wire and pipe is accounted for. Moving the outlet means that the wires to it have to move and in turn that means a plumber later can't come in and put the pipe where it must go. In a house they solve this by having the plumber come in before the electrician (the order is much more complex), but in a large building you need them working at the same time or you will never get done. Even in a house you still need to plan the plumbing as often a pipe and a beam need to go through the same place - either the beam moves or the toilet moves.
Sometimes - but some large buildings have loads of space above suspended ceilings and in cable trays and suchlike. Office buildings know every new tenant might want to move internal walls, rearrange sockets, reconfigure the kitchen and so on, and they're built with that in mind.
You're right that making arbitrary ad-hoc changes during construction can be expensive, of course. Especially if it's wrong and has to be torn out and redone.
But it's not illegal - if you're in charge of a project and an installer tells you the specified vents are going to be a problem and they recommend a bigger one, nothing in law prevents you from telling them to go ahead and make the change, right there on the spot.
Of course, how often the person with in charge of a large project will be talking to the vent installers while they're working is a different matter...
Many buildings are designed that way. It is a useful thing to do, but there are trade offs - you pay for that extra space in other materials and extra time. For an office the office/cubes/open layout is installed by the renters after they lease the building and thus the space you are talking about isn't part of the initial construction effort. They still plan/reserve space for the pipes and large wires in initial construction. Generally the bathrooms and break rooms (kitchens/cafeteria) would be done in initial construction and so carefully planned out.
It of course isn't illegal to make changes - but it is expensive and thus not something you should do. (though some things are much easier than others to retrofit half way in.
I don't think you disagree with me. For larger projects, like infrastructure projects, very detailed designs are used and contracts that disallow substitutions or alterations to the design without contractual agreement.
I'm talking with some level of experience here. I did a residential project that was obviously much smaller than this but still very expensive (£100+k). Even for this architects recommended a detailed design and a contract that enforced this.
Building is not a waterfall process because of the Planning act. It's waterfall because of the cost structure of building. Every time you iterate physically, you are throwing a lot of money away in time and materials. So any iteration is pushed back to the paper part of the exercise, in which case it might just as well happen before planning approval.
Also, it's possible to request a variation, especially if something wasn't going to work. There is a category called 'non-material variation' which the planning officers are lenient with.
If you got rid of planning (and building control) I guarantee that the quality of buildings would go down. Before building control, many unhealthy buildings were built due to bad drainage, etc.
> It's waterfall because of the cost structure of building. Every time you iterate physically, you are throwing a lot of money away in time and materials.
Also it's because of the financial structure. People invest a lot of money in something with a delayed, sometimes for years. Delay increases everyone's risk, increases interest expense, and defers revenue.
Building is - or should be - iterative. However each building is a separate iteration. Once you are done you stop and look at what works and what doesn't, then adjust the next one. The more you make your building different/unique from what everyone else around you is doing the more expensive it will be as nobody knows what changes really work - and even if they are for the better the lack of experience means you are paying extra for the laborers to figure out how to do things.
