As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016"
In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious.
Sort of plays right into the whole argument being made over on another thread (https://news.ycombinator.com/item?id=11105198) that companies shouldn't rely too heavily on proprietary tools to solve their data challenges.
Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
There is no mystery algorithm. It's just cheap college labor working obscene hours to pump out "data analysis" scripts. The sad thing is that it actually works. The places that hire palantir are so mired in bureaucracy and process that whatever palantir builds seems to be better. Of course, just like SAP and Accenture you're now kinda locked in with support contracts.
Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
Maybe it tops out early, but I'm given to understand junior salaries are also very competitive. (Maybe this is less true by comparison in the States -- it's fairly true here in the UK.)
We at CloudScrape.com are happy to welcome you onboard our platform - you get 20 hours of scraping runtime for free and the ability to navigate much more complex websites, opening up tons more opportunities then you've had previously. And we're here to stay :)
Kind regards,
Henrik Hofmeister, CTO, Co-Founder @ CloudScrape.com
Yup. I signed up with an account to play with it once. Never used it after the initial week. I wonder how many others of that 125k were just drive by test drives.
This is really bad news for us, and the tone of the announcement grated. Two weeks notice is very bad form, I'm having to come back from vacation to fix this.
When we evaluated options a year ago, kimono was the best fit. We'd happily have paid, and did ask. I don't understand what happened... Did they run out of funding?
Here is a statement that can give you some insight
"Kimono Labs explained that in the two years since launching, it wasn’t able to have the impact that it wanted, and that the acquisition by Palantir will give the team “unmatched support, resources, and the ability to work on things we could not tackle alone as a small startup.”"
from http://venturebeat.com/2016/02/15/palantir-acquires-kimono-l...
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but I found ParseHub to be a more powerful and flexible web scraper.
If you have been burned by free services like this shutting down, give https://scrapehero.com a try - we love our customers and they love us back. We dont plan to abandon them with a 2 week notice.
We've had enterprises that depended on Kimono come to us; which isn't surprising anymore :) Hope others will give Grepsr a try as well! - https://www.grepsr.com/
Kind of bummed on the two weeks notice, but I'm sure there's some pressure and tradeoff the team had to make, which I guess softens the blow a bit (for me at least).
The worst part about Palantir is their ability to masquerade as this hip startup, as though working for them is as innocent as working for Imgur or Twitter or whatever. They recruit the hell out of MIT students, and it sucks seeing my friends interview for this incredibly shitty company that has openly said it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?).
I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
I know some engineers who work or worked there. Aside from the normal engineers (tools, backend, etc.), they are essentially high-priced tech consultants sent to companies and orgs that have no internal software talent at all, to help them analyze their (atrociously stored/managed) data. None of them work on anything shady. They essentially build web apps and backend apps to analyze and visualize data. Nothing vodoo military about it, although the military is a customer.
The pay actually isn't actually particularly high for SV standards. I think Facebook outpays them, for example - and Facebook is less selective than Palantir. But their reputation is that they're full of top engineers and you'll learn a ton there, and that's why the people I know work/worked there.
The downsides I've heard about are no work/life balance and fratty culture. Those are real downside, not the FUD about ethics.
> it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?)
Well, more generally, anyone who has a nonstandard set of ethical beliefs, and is basing their company around that set of beliefs, can't really go public without their belief-set being thrown under the bus of their shareholders' belief-set.
For example, I've often considered starting a company in the game industry that actually hires experienced adults and treats them well, rather than hiring fresh college students and burning them out. I imagine, though, that if my hypothetical company became publicly-traded, I'd be forced to relinquish this policy in the name of short-sighted market competitiveness.
You could go the Google and Facebook route of having voting-class and non-voting class shares. Sergey and Larry, and Zuck still retain control of their companies.
Interning for Palantir is a pretty big resume boost for college students, and they have one of the highest paying internships in the valley. There's some serious benefits to going to Palantir.
If you have the ability to intern at Palantir you have the ability to intern at any number of more responsible companies. Palantir has a bigger booth at the career fair, but I'd bet working at Apple or Uber is just as much of a resume boost at much less of a societal cost. (source: am a college student)
Interning at Apple or Uber has much more stiff competition though. The barrier to entry for Palantir is remarkably low considering their pay, it's not surprising that college students apply for Palantir. (Source: am college senior)
Apple and Uber are not as much of a resume boost (in Silicon Valley, to a SV recruiter). Palantir is sort of on the level of quant finance / Dropbox 2 years ago.
Apple and Uber are hiring like crazy and so it's quite a lot easier to get in there as an intern than at Palantir (which is more selective).
That's surprising, as they're known for being incredibly shady and have a pretty bad rep. They might pay a lot, but it's because it's less desirable to work for the Evil Empire.
Maybe they are useful for students to signal their lack of ethics to employers? “Look, they once worked at Palantir, so they’ll have no problems doing whatever shady stuff we tell them to!”
Interning as a prison ward in Auschwitz is a pretty big resume boost for anyone who served in a prison system, and Nazis have one of the highest paying internships abroad. There's some serious benefits of becoming a prison ward in Auschwitz.
That's a rather large downgrade to the previous accusations. From evil company to a company with no benefit to mankind. I think you can say that about a lot of peoples jobs - what's your point?
They have one of the worst work/balances in the valley, fratty/kool aid culture, claim to be "making a difference" but really what they actually work on would be a major downside for most normal people. For example, they make targeting software and are very likely responsible for assassinations by the US and when we disappear people. They're also working on big brother tech if that's your thing.
> When intruders from the hacker group Anonymous gained access to thousands of emails stored on the servers of the security firm HB Gary Federal, the emails revealed that Palantir had worked with HB Gary Federal to develop proposals for attacking WikiLeaks’ infrastructure, blackmailing its supporters and identifying donors.
The sex accusations ended up being baseless, and Stanford reversed their ban on Lonsdale entering the campus (and Stanford has a lower evidentiary standard than the courts)
Also, your quote doesn't appear in that article at all any more.
As for the sex accusations, who knows what really happened. Though the article doesn't exactly paint him- edit- either of them- in a flattering light. It does show poor judgement for the dude to get involved with a student he is mentoring. Then again, he also stumped on the campaign trail for Rand Paul of all people and also pals around with Mitt Romney.
Their YouTube channel has 100's of videos that show what they do. The one that blew my mind: GovCon7: Introduction to Palantir: https://youtu.be/f86VKjFSMJE
On one hand, Palantir have (at best) questionable connections and I wouldn't be a rush to work for them. On the other, with the level of pay/benefits they provide, I'm not surprised they can find people without similar qualms (or people who overlook or aren't aware of them).
I won't share specifics (as they aren't mine to share), but I have a friend working for Palantir overseas. Once essential benefits (e.g. accommodation) are factored in, his salary is north of six figures (GBP). For someone only a few years past university, I imagine that sort of offer is difficult to refuse.
It's cool to get a 6 fig GBP salary but tbh, assuming what I've heard about the working hours there is true, I could still just do a combo of freelancing and contracting and make way, way more cash.
Some business owners just don't want shareholders inserting their opinion where it isn't wanted. If they don't need the funding, there's no reason to go public.
My wife is a student and looking for internship she will work for a company that helps people write about how they ran over than grandma if she gets a legal. Morality and ethics have a price always.
https://webrobots.io - we give free Chrome extension to run scraper robots on your own computer. For paid customers we host robust cloud of scraping instances with all advanced features one can expect. To be fair - you need to understand javascript to write robots on our platform, we are not point-and-click doodling tool. Upside of writing robots in code - nearly unlimited possibilities with dynamic content sites, in-flight data cleaning, etc.
damn. I was gonna say this means it's time to actually launch my version of this I built 3 of 4 years ago, but it looks like apifier is almost the exact same thing.....
ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns).
Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
Parsehub is easily the best hosted craping service on the market. We investigated everybody from kimono to import.io and if you want it done right they are the only option. Furthermore the guarantee of an open source version should they ever decide to close is extremely reassuring.
Awesome. Thanks for sharing your service. My current project has been faltering because of my ignorance of these types of tools. Yours looks like it's going to make a huge difference for me!
As someone who used Kimono a few times before the closure...this announcement does not surprise me at all.
The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.
Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity:
An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.
What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup).
Which is even reusable.
The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc."
Heh. As if that ever did, or ever could work.
> "non-programmers can do it. No code required, etc. etc."
translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems."
You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.
I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:
Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
I agree. I actually referred people to Kimono. I'm so sorry I did. This is unacceptable behavior in this day and age. I mean seriously? Servers are cheap. Keep the thing up for existing users. Don't add new features. Don't invest any more time in the thing. But don't pull the rug from under people's feet. That's just bad form. If this is who Palantir acquires or acquihires, then they have some really bad judgement too. It reflects poorly on Palantir to not care about commitment and responsibility.
It reminds me of a PC World magazine that I had. It took me over a year and a half to get around to reading it. It had an article about the 50 best websites or something like that. By the time I got around to reading it, at least 1/3 of the sites were discontinued.
You can use portia[1]. You can try out a hosted version at scrapinghub[2] or you can host your own. It's a web based spider creator like kimonolabs but the spiders it creates are just scrapy spiders.
If you need to do something that the UI can't handle it's not a problem, you can extend it with a little python.
"access to the world’s most important data problems"
I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
Disappointing to see just 2 weeks notice for their existing users.
* shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com Try it out and see if it works for your needs.
> Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.
Palantir is one of those organizations that makes me throw up a little in the back of my throat whenever I hear about them. They are sort of like the Nickelback of companies.
For morality reasons I would never be comfortable working there. The workplace could be heaven on earth and pay really well but there are simply things that are more important.
You can host scrapers you need to migrate on morph.io ( https://morph.io ) a free platform for hosting and running scrapers, storing your data and providing access to it in a range of formats (JSON, CSV, SQLite). It’s free, open source, supports a bunch of languages and is run by an Australian non-profit, where I happen to work ;-).
Disappointing. Was my goto service as it was a joy to use and always seemed to work. Wish they would follow the Parse lead and release something open source. Desktop doesn't really cut it for API use...
I'm always disappointed when I read of Palantir acquihires.
Technically talented employees receive a pittance and are sold into the Palantir machine, a morally unscrupulous high-tech consulting shop that takes in talented engineers and churns out one-off software black boxes designed to lock in customers and extract a long-term tax.
I had a hard time using this product. I was trying to use another developer's APIs but it didn't seem possible. The one positive thing about the announcement, from the FAQ:
Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Their "desktop tool" is complete rubbish. I have to manually click on each source to crawl it and then manually click a few buttons to save the response.
Two months is a short time but I had sort of expected this given how bad their service of performing last few days. That is why I had started moving to other services.
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but ParseHub is a more powerful and flexible web scraper. https://www.parsehub.com/
Sad to see Kimono close down. I enjoyed using it for a small project, and was waiting to see a paid version with private APIs, but it was always "coming soon"...
I don't quite get why startups are not trying to generate revenue from day one.
These people should be totally ashamed for the tool they have given. It is just nodejs wrapped inside a desktop app. They could have very well made it open sourced and offered to community.
Kimono had a beautiful chrome extension which allowed me to easily extract things quickly without needing to write any selectors / scraping. Much love to this team; best of luck!
Anyone offering a GUI solution that can export to an RSS feed like Kimono did? I know RSS is old but I have no clue how to display the other formats in wordpress. Thanks!
Its a two edged sword, sometimes you're doing great - but the fact they're shutting down means they're not doing great but have a team of experts in parsing that will benefit Palantir since they're likely into analysing surface and deep web data and just data in general :)
basically its like hiring a bunch of regex gods because you have regex problems on large scale
To all those badmouthing Palantir. I think they are just a hogwash. They are just pretending to be something that Fox Maulder would investigate I dont think they are. They are just helping the government spend more money on problems that are not worth solving and create new problems for citizens.
The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016"
In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious.
On the other hand, congrats to Kimono team!