Since there wasn’t anything broken on our end, there shouldn’t be anyone else impacted by this. But as part of making sure that OP wasn’t visible by mistake, I had my team double-checked to make sure our previous fix from last year was properly retroactive. It was.

More generally, I don't think that tricking anyone is a viable way for us to run a business. We’re trying to create a marketplace that can open opportunities for engineers who wouldn’t otherwise have them, and we need the trust of engineers in order to do that.

As far as I can determine, the last deliberate interaction I had with Triplebyte was in May 2020, when I got an email with your name and address on it saying that my profile was about to be made public, and I replied the following day asking you to delete my account. (I assume you never saw that email, and in any case I never bothered to follow up on it through other channels.)

(EDIT: on closer inspection, the reply with my account deletion request went to candidate.support@triplebyte.com, not to you personally.)

I don't recall ever logging into my Triplebyte account between then and yesterday, and I couldn't find any evidence of doing so in my browser or search history. I guess that's your word against mine, but if you have reason to believe I'm mistaken, you're welcome to send it to me privately.

> The support response you received was incorrect

I'm very curious as to where the incorrect information came from, then.

> As mentioned on HackerNews, we cannot locate any emails from you before today. After getting in touch with the product team, I realized I had made a mistake. Upon doing a bit of digging on the back end, you set your profile set to be visible in 2019 prior to the 2020 events.

I guess I can't concretely disprove this story, but I have a really hard time buying it. I don't remember even being given the option to make my profile public when I tried out Triplebyte in 2019. To back up my recollection, the messages that they sent out about the "public profiles" feature in 2020 described it pretty unequivocally as a new feature that hadn't existed previously. Archived versions of Triplebyte's FAQ from 2019/2020 make no mention of it; they only talk about the ability to be matched with companies after completing a full interview with Triplebyte (which I never did).

And if the person who initially responded to me truly did just make a mistake, that would certainly be no big deal -- but it seems like an oddly specific and convenient mistake to make.

I responded about 24 hours ago saying why I found these explanations unconvincing, and haven't heard back. I'm posting here not to try to pressure Triplebyte into a response, but because (a) I don't know how much longer this thread will stay open to new comments, and (b) I don't really think it's likely that I'll get any further closure about this issue, so I don't plan to spend any more of my time and energy on it. People are welcome to read the discussion and judge for themselves.

> To back up my recollection, the messages that they sent out about the "public profiles" feature in 2020 described it pretty unequivocally as a new feature that hadn't existed previously.

The "public" profiles we announced (and rolled back) at the time were distinct from the visible-to-Triplebyte-subscriber-companies profiles we have today (and had in late 2019). "Visible" is doing double-duty here between public-to-the-internet (which we never ended up doing) and visible-on-our-platform (which predated the 2020 announcement).

I'm going to have our support team get back in touch with you via email for further details here. Can you reply here if you haven't been able to do at least one round of back-and-forth with them within the next day or so?

Nope. Just to show that I'm not yanking your chain, this is the email I'm referring to: https://imgur.com/2FpAiik

The redacted name and email address are exactly the same as when I contacted you yesterday.

To be clear, I'm not trying to criticize you for not taking action on this message, because for all I know it could have gotten dropped by a mail server along the way. I'm just using it to illustrate the fact that I wasn't even aware that I still had an active Triplebyte account, so I find it implausible that I logged in, set my profile to public, and then completely forgot about it.

> Can you reply here if you haven't been able to do at least one round of back-and-forth with them within the next day or so?

Sure, will do.

Yeah, of course. Totally fair.

(Further comms via email)

Gotta chase those dollars

Provided they're in the US. My experience as a European has been: more or less everything I apply to ignores my application. I haven't checked recently, is it changing?

I'm not implying you're wrong since you used the word trying to create. Moreover, you didn't specifically specify which group of engineers you're trying to create opportunities for. I don't even want to go in a right or wrong type of frame, because it doesn't matter, but my lack of eloquence might give that impression.

What I am implying is that the statement is a bit broad. On a more emotional (perhaps even non-rational) note: I feel spoken to yet left out, for years now.

At the very least, a couple of U.S. companies with remote positions that claimed to be U.S. only (or "work authorization required?"), reached out to me.

But I think some companies select U.S. only because Americans tend forget about other countries where people may be in the same time zone.

It's possible German companies are reluctant to engage with a Swiss person because German salaries are lower than what I understand Swiss salaries to be (German CoL is also much lower though)

We actually just (like, two days ago) shipped a change to be more restrictive about how we show jobs based on location. And we have a few things in the works (unfortunately I can't share details) that might result in significant increases in activity for engineers outside the US.

> Moreover, you didn't specifically specify which group of engineers you're trying to create opportunities for.

The simplest way to put this is "we want engineers who can do the job to be able to get hired". That is obviously a pie-in-the-sky goal that is more aspirational than anything else (you're never going to bat 1.000), but it's the guiding principle.

In a more on-the-ground sense, we think our assessments allow engineers who do well on them to get company attention that they might not otherwise get. And we have hard data to suggest that we are correct. Each assessment with a score of 3 or higher on an application roughly doubles the chance that that application is accepted, for example, and the overwhelming majority of outbound messages on Triplebyte go to engineers with at least one such score on their profile.

But if I had to guess? It's because we've been ramping up a feature that is increasing the amount of outbound on our site by a very large margin (currently almost an order of magnitude) from where it was a couple of months ago.

Why? My experience with them was pretty bad. I took their assessment for web development, I think I even did an assignment, and got put on a video call with someone from Triplebyte. He never cracked a smile. Suddenly I got asked a bunch of CS questions that really were not very relevant to web development, some of which were entirely inappropriate like sorting a binary search tree. I even told the guy that I thought I was getting those questions wrong and he just scowled and said "well you just don't know when you're going to use this stuff." "My point exactly," I thought.

Ultimately I got rejected.

The whole idea that you can boil down a candidate to some coding challenges and a video quiz is bad. I do like the idea of streamlining the hiring process for developers, but there's more to it than knowing a bunch of stuff, because that can be gamed. And quizzing me on irrelevant material was a bad move. A firm like Triplebyte won't be as good at interviewing a candidate as the employer itself, and may even keep perfectly qualified candidates out of view from all employers affiliated with them.

- The screening had a lot of false negatives. "I got rejected by Triplebyte, but got a FAANG offer" is quite common.

- Most companies used Triplebyte not as an interview replacement, but as an additional screening process, which means that as a candidate, you don't have any real incentive to use them.

The only real use case I heard recently about Triplebyte is to send candidate who normally you wouldn't even screen, so if they pass Triplebyte process, you know that you should consider the candidate, but if they fail is fine because you would have passed them anyways

I used them many years ago, this was my impression. When I got to company "on sites" they were just full-blown interview loops. I could have just applied to the companies directly.

TripleByte isn’t what they used to be, though. I don’t think they do anything close to what I experienced anymore.

I started using them about a year ago (first passively looking, then actively looking)

I really enjoyed the ability to be assessed on something besides Leetcode style questions.

I didn't take a job through their platform (though I did get one really strong offer), but even still, found the assessments incredibly useful, since they give you a percentile distribution of your performance for each topic-specific test.

After taking their assessments, when interviewers asked me how I am at, say, Python, I could tell them I have a hard time assessing my capabilities. "But hey, I took this standardized test that says I'm in the 85th percentile, not sure how good of a metric it is" (and not mentioning that I think I'm OK at best, at Python)

It's the only way I've found to get a measure of your talents compared to the rest of the field (even if it might not be reliable/useful)

A lot of the companies that interview through Triplebyte also skip LC mediums because they have a different signal about your potential suitability as a candidate.

Way too much of engineering is non-quantifiable. Putting a number to someone's skills is bound to be reductive at best.

Speaking as someone who doesn't like being reductive, I've had to make my peace with the fact that "flawed" can still be "way better than the status quo". And I really do think we do much, much better than the status quo of companies putting a bunch of top schools in as linkedin keywords.

If you're familiar with data science a bit - think of it as trying to project out the first couple principal components of your skills. It won't account for the whole data set, but you can go a long way with just those first couple components.

Like honestly I might think I'm a 3 at X, but if some test that thousands of other people took tells me I'm in the 90th percentile of X users, that information is still useful to me.

One complaint I do have is that (in addition to the percentile bucket) they give you a 1-5 rating, where 4 is "senior engineer level" and 5 is something "exceptional performance, a leader in the field"

But the ratings seem to fall at different percentile distributions for each test.

For example, I might get 80th percentile on one test, but get a 3 rating, and for another test, 80th percentile is a 5.

In general, different quizzes have vastly different populations of people attempting them. For example, our front-end quiz gets a lot of beginners and hobbyists, and thus has a very bottom-loaded score distribution. Our devops-related quizzes, on the other hand, have a population that skews skilled and senior, and has a very top-loaded score distribution.

Communicating this information to our users (particularly the less-quantitatively-oriented ones on the company side) has been a source of considerable UI challenges for us.

Personally speaking, I've used Python a handful of times over the years, but never as a primary language for any work I've done. I got a 4 on the Python test.

Compared to front-end, which I've been using professionally, and also dabbling in for ~20 years (still keeping up with developments in the years in which I wasn't primarily doing front-end dev professionally)

I got a 3.

I definitely know 100X as many random facts about front-end APIs, libraries, tooling, and technologies than I do about Python. So perhaps it just came down to luck (guessed unlucky for the front-end and lucky for Python). Or perhaps there's just so much more to know that falls in scope for the front-end quiz than there is for Python, to the point where you can spend 20 years learning the front-end technologies and still be "middle-of-the-road" in terms of "absolute level of skill".

But I think that makes your descriptions of the 1-5 rankings a bit disingenuous. If people who has (what most other companies would consider) senior-level knowledge is generally considered a 3 by your system, a more honest ranking of the descriptions would involve changing "4: level expected of seniors" to "4: knows roughly ~80% or more of all things there are to know about this subject".

They're set by experts in the area, the same as the ones who write our question content.

To give a little more detail: the tests run in a beta state for a while before being fully released. We gather a bunch of data and calibrate parameters for our IRT model based on that. So the ordinal ordering of performance is entirely mathematical and data driven. (When we were still doing in-house human interviews, those were part of the data set as well, and still are for the subjects that overlap them.) But that produces a continuous, hard-to-interpret, and population-dependent score distribution, and SMEs draw the lines with which we bucket those scores. (For those of you familiar with IRT as a framework, they set theta thresholds.)

But yes, there is some chance involved. It's a tradeoff between the standard error in our scores and the length of the quiz, and we try to optimize for a sweet spot there (since most people don't want to take two hours of quizzes). And we are absolutely going to get it wrong sometimes. That's both for in-model reasons (the statistical standard error is enough that we we'll be off by a level either way around 20-25% of the time or something like that) and for out-of-model ones (maybe some of our questions just test the wrong thing in ways that don't show up in the data). Assuming your self-assessment is correct (and I will say that many peoples' are not! confidence correlates with skill, but with a whooooole lot of noise.) then yeah, you probably had a bad roll of the dice on one and not on the other.

As I say a lot (in this thread and elsewhere), we can't reasonably bat 1.000: our goal is to bat better than the next guy. And I think we do do that, messy though the entire space can be in practice.

---

For the record, when we talk to companies, here's what we tell them about scores:

2 = knows something in this area, but we can't say with confidence that they know enough to handle things independently. OK for entry-level roles, but lower than you'd like for others. We don't show a 2 on profiles. The only place companies see it is if they're using our screens to screen their own candidates via our Screen product. The idea being that if you have the choice of whether to take an assessment or not in the first place it shouldn't really hurt you to try.

3 ("Proficient") = professional competence in that area, can work independently in it. A score we'd expect of a mid-level engineer within their area of expertise. A recommendation for most roles, maybe not very senior ones (but not a point against even for senior roles). A score of 3 or above counts as certified, meaning it earns a shareable certificate and makes you appear in search results for a particular quiz score.

4 ("Advanced") = significant expertise, something more typical of a senior eng who really knows their way around the subject. A recc for all levels, even very senior ones.

5 ("Expert") = exceptional, above and beyond even by the standards of senior roles

So it helped me in introducing me to companies I wasn't previously familiar with, but other job platforms work in similar manner.

I get the impression that Triplebyte has changed from what it used to be. I never even talked with anyone at Triplebyte. I did well enough on some skill test that I was put on a fast track and quickly approved without any interviews. I also got opportunities to take other tests to rate my skills in particular areas.

It seemed like a decent place for presenting possible candidates to employers with some pre-screening, but it wasn't anything particularly innovative. I imagine that as an employer, it helps filter out a lot of unskilled candidates with pretty resumes and reduces the number of interviews required.

With the money they raised, after spending so much on marketing, I assume they downsized, lost some talent, and pivoted mostly to a sales-driven recruiting business for their top clients.

I can't imagine the difficulty to accurately measure the success or failure of long-tail HR hiring processes like phone screens. The success or failure of a candidate post-hire has so many variables it must be very hard to attribute them to signals present in a screen. I imagine most of the data points are derived from signals found in successful candidates, and then trying to find them in an assessment or screen.

Its really hard, and I hope the negative tone of my comment does not suggest I don't respect the problemset and the people willing to throw themselves at it.

And even if they stayed to their original model it would have been too easy for niche competitors to erode their margins. Think Triplebyte for Android developers only times 20 different programming areas.

But it needs to be:

- In-depth. Not just a single exam or interview. You need to really know the employee’s strengths and weaknesses

- Detailed. You can’t just give someone pass / fail or a single score. Not only is it mean, but you end up getting misaligned candidates anyways, because some people are really bad at some aspects of software but good at others. In fact maybe the process should ditch scores entirely and just show the recruiters the actual employee interviews, and what he/she has and has not accomplished.

- Changing over time. Not a short period of time. But like, if I take the assessment, 6 months later I can take a smaller assessment and it will update my scores and log my progress.

Triplebyte is not 1 or 2. Idk but I think it’s 3 and you can retake the quiz. But then it’s only telling employees if you’re basically competent for some arbitrary statistic, which doesn’t even tell if you’re basically competent at the company.

I think it would be nice if i could take one thorough interview instead of several less-thorough company-specific interviews, but that’s not Triplebyte.

Companies wouldn't trust a third party to run binding technical assessment for them, and quality devs would probably avoid places that hire without having someone from the destination team show up

I think the opposite would be more beneficial: a light check to validate work claims and some high level foundational question about code just to make sure one has basic proficiency in what he claims he has

Then companies would need lot less hr pre-screening and could focus in technology and culture matching

If you can hire someone skilled, but who other companies might overlook, that's a huge benefit to your team.

Triplebyte is really just the first screen, and the importance each company wants to put on the signal is up to them.

You dodged a bullet, some of the most "interesting" interactions I've had with founders were from those I talked to through TripleByte. There was also a pair of them that were clearly digging around for business ideas and markets to enter.

yes, there are too many variables between the candidate, job, company, and work environment to determine long-term fit via a test, especially for "creative" jobs. the more regimented the job (e.g., fast food cook), the lower the variability, but it's still significant. plus, such tests only evaluate technical skills, not the more important non-technical ones (like punctuality, integrity, steadfastness, etc.--note that these are a function of the involved parties and the relationship between them, not just the candidate).

but also, the underlying problem of hiring is not one of trying to get the best fit, but of trying to avoid the pain of firing. that's the thing that needs to be reframed/solved, but that's a much harder and a much less technical problem (alternatively put, technical tests are marginal at best).

I spend weeks drumming up two or three days worth of coding work. The coding aspect is basically manual labor and pedantic arguments with other devs.

a 10-day contract is better, since it's real work, for pay, but the relatively short duration doesn't tell you much about the candidate's intrinsic motivation or how relationships develop past the honeymoon stage.

so really, it'd be best and easiest if we all explicitly assumed that jobs had 6-12 month trial periods, for both parties, and that after that time, either can walk away without hard feelings (or negative judgment), other than in the most egregious cases (i've seen a couple cases that'd fall in this category). again, this is primarily about jobs that have the most variability. less variable jobs don't need as long of an evaluation period (but do need more than a few weeks).

my (now failed) startup in this space attempted to answer it for less variable jobs (hourly work), where we could tease out more signal from the noise, but even that had lots of unaccountable variability.

1) Create an incremental game (start simple, see how far you can get) 2) Live debugging (can run tests, they fail, you need to figure out why and fix it) 3) Flash rounds (Do you know what Big O is? Can you explain linked lists?) 4) ... I forget

I thought it was one of the widest range of actual skills and their final assessment I agreed with. Stayed away from algorithms-y questions (which I hate)

I’d say I cost them a lot of money when I deleted my account, but they actually cost it themselves.

They could have asked me if I wanted a fancy jacket, because that ended up immediately getting recycled. The book was dece tho

Anyways, I ended up getting a job that I love the old fashioned way - through a friend.

Great concept, shit company.

Edit: Triplebyte employees reading this: you massively betrayed my trust. Had you asked me, and let me review on my own time, I might have even been proud to have a verified skill set badge page I could link to. Instead, I will probably look directly for your competitors on my next job search. You had a great idea, too bad that you seem to be irresponsible.

I wish more companies use this method.

I always thought getting jobs through friends was a decent vetted matchmaking process. That said, I've never had a lot of direct power to get friends hired, so I never felt it was nepotism. And the friends were usually from previous jobs.

https://en.wikipedia.org/wiki/Nepotism

Nepotism in government is restricted at the federal level, and most states have their own rules about it too. But in the private sector it is common and not illegal by itself. Think about it: if nepotism in businesses were illegal broadly, it would end the concept of family run businesses, and make small farms even harder to sustain. This comment is in reference to the U.S., where Triplebyte normally operates.

Being able to skip an hour or two out of a full interview across many companies is a great concept.

Having a 3rd party verified that allows you to be quickly matched with potential employers is a great concept.

Execution may be difficult, or impossible even. But that’s not an issue, when said company insists on shooting itself in the foot constantly.

We'll have a more complete answer shortly.

EDIT: This does not appear to be a widespread issue. Continuing to investigate.

EDIT2: Full response from Ammon, our CEO, at https://news.ycombinator.com/item?id=31771836

————-

At this point, if you (Triplebyte) haven’t set private the profiles of everyone who didn’t explicitly consent, then you are irredeemable. You’ve had, what, two years to work this out?

I would love to understand what the goals and numbers here are, and what the revenue implications of public profiles are? It seems to me that you’re trying to build a social network of some sort with verified-skill capability.

I went to your site, and your about me page, to see what your business model was. What you stated, the mission and the copy, does not align with a public profile and social network style feature.

If you truly are trying to make hiring better, how does public profiles achieve that?

If you truly are trying to make hiring better, people on all sides of process should be benefiting. And there should be nothing to be afraid of in terms of mission.

Your /manifesto document falls short. What, exactly, is your business model? You should have no shame in it. Unless you think it’s shameful, in which case, please stop.

Like your manifesto claims, transparency is good!

But to reply to your questions:

------

> I would love to understand what the goals and numbers here are

Well, zero, since nothing was broken in this case. The issue last year had affected a couple hundred people (out of a total pool in the hundreds of thousands) before it was caught and reverted; the worst-affected person was erroneously visible (to Triplebyte companies only) for about a week and a half (and was hidden again within hours of us knowing about it).

> It seems to me that you’re trying to build a social network of some sort with verified-skill capability.

More or less. Not a social network, necessarily, but a platform of engineers with skill data from our assessments.

In brief, our goal is to make the job search process lower-grind, higher-signal, and more fair for candidates by exposing more information about companies, targeting messages more intelligently, and relying on skills over resumes. And on the company side, our goal is to provide them better and more reliable information than they can get when trying to hire elsewhere.

Everyone hates the status quo of a thousand applicants or a thousand outbound messages per conversation started, it's just a hard status quo to disrupt as an individual candidate or recruiter.

> If you truly are trying to make hiring better, how does public profiles achieve that?

To be clear, we do *not* have public profiles in the sense of "visible to the internet as a whole". We have profiles that are or aren't visible to our (company-side) subscribers, according to each engineers' settings; a "visible" profile is visible to those subscribers and an invisible one is not. The only way a profile can currently be exposed to the world at large is if an engineer on Triplebyte explicitly enables a public link and posts it elsewhere. (Even then, I don't think they're indexable? I'm not 100% sure about that bit; this is an old feature I haven't touched during my tenure.)

> What, exactly, is your business model?

"Be a place where the best engineers want to look for jobs, because it's a better and fairer experience, and charge companies for access to those job seekers and for the tools we give them to surface the best matches for the role they're hiring for."

In a literal financial sense, we charge a subscription fee to employers for access to our candidate database ("visible" profiles in the previous section's sense only).

You're welcome to state your side of the story, but I take exception to you calling this a "fact". It's a theory that you've asserted, but not shown any evidence of (publicly or privately). And I've already explained why I think this theory doesn't make sense.

I have amended my post to refer to yours, for clarity.

But I hope you can understand why I'm skeptical: not just because of Triplebyte's track record, but also because the customer support representative who responded to me seemed to be under the impression that this hadn't been fixed retroactively. If I'd instead gotten a response that said "hang on, that shouldn't have happened, give us some time to figure out what's going on", I would have found it a lot less concerning.

Anyway, tl;dr - see Ammon's top-level reply.

You can get a shareable link to share on e.g. your personal website if you want to, but you have to manually enable it.

It's just so disrespectful, as you said.

Whether it's dick pills or job opportunities, it's spam.

I have no issues when I purposely choose to work with a recruiter, but invading aspects of my life when they never got permission to IS disrespectful.

+1

Next...

I took their generalist test and scored perfect. It was actually not so hard, but it tested some language things (a couple/few languages), some Unix things, SQL, etc. It was right up my alley.

After completing the 30? min test, I got an email saying they were enthusiastic about me and would like to continue the process.

I think the process was to become an interviewer; I forget now.

Time passed... weeks, and then months. I emailed them and got nothing back.

Then I checked Glassdoor (YMMV). There were a lot of negative stories about internal politics, gender issues, leadership problems, "insane" bad manager stories, etc. I don't know how Glassdoor works, but perhaps it was all lies; or perhaps it was very true; perhaps it has been erased (pay to play).

Nonetheless, I parked them in the corner of my mind behind Toptal.

I didn't even get the "privilege" of doing the 40 hour take home project (which culminated in a formal business presentation that some people reportely failed despite turning in a polished project".

Later I found complaints of Toptal members that they were pressured to provide low wage offers to get gigs.

Overall it felt like perhaps an ernest initial effort to make a better system and a quality consulting firm, but it got perverted into a heartless profit machine. I'm glad I didn't progress to the 40 hour take-home stage.

Are we looking at the same Triplebyte? Their reviews on Glassdoor are pretty good

edit: There are some nasty ones further back, the latest being Feb 2020

Has anyone here on HN?

I group them in the category of companies that has an attractive proposition but turns out to be much less lovely once known better.

The repeated fails on execution is pretty disheartening.

Candidates spend their time on stupid coding questions all day instead of actually coding something useful or that they can learn from. I have a relative doing exactly this. No idea how to build a simple RESTful API, yet spends all his time on Hackerrank posting on linkedIn how he's in the top 'X' percent.

When they get hired and put on a "real world" project they are absolutely lost :(.

I also tried Triplebyte after seeing ads on Reddit. Passed a few tests and nothing really ever came of it other than an email.

What is it with this repeatedly opening up profiles though? I thought the saying was that it was ok to make mistakes, but.... make new ones.

That did not make enough money so the founders pivoted like 5 times to try other things and each time they made Triplebyte more and more anti candidate.

Keep on going Ammon, we all know where these companies end.

I responded to them with a link to this thread in case they wanted to publicly comment or correct anything, but I'm not holding my breath.

EDIT: I can confirm that that fix was in fact retroactive. We'll have a response to OP momentarily. (EDIT2: Ammon, our CEO, has a top-level reply to OP.)

Can anyone recommend alternatives for the future? It's clear they're untrustworthy so I'm hesitant to even consider them again even though I passed years ago (2019 I think).

I knew better.

The obvious way is wagging your fist at all those bad guys doing bad stuff.

But doing bad stuff is actually a pretty good way too. If you can manage the backlash. Works great.

I mean, it worked here. Right? The last time I thought about triplebyte was the last time they did something bad.

Kids do it all the time to their parents. Very hard to ignore.

Something to think about.