This competition (the "Large Scale Visual Recognition Challenge" aka "ImageNet") isn't just some random competition. This is the competition that gave rise to the recent explosion in interest in Neural Networks.
In recent years Google, Microsoft and Baidu have been one-upping each other to the point now where they are getting very close to better-than-human performance (ie, humans disagree with other humans more often than their systems disagree with the average human rating)[0].
Andrew Ng went to Baidu to start their team. I don't believe he is still involved in this challenge.
Baidu has been getting close and close to Google's performance. Every year so far Google has topped it at the right time, but Baidu has later passed the Google benchmark that year.
There were reasons to think that this could be the year they finally beat them. [1] is a story from January about the system that Baidu had built then.
> There were reasons to think that this could be the year they finally beat them.
Then they shouldn't have cheated. This is the same company that didn't do anything to stop it's network being used to wreck GitHub for a week (and yes, it's possible to avoid the GFW injecting JS onto your site - buy space in a non-China DC).
It does, but I think the past performance is probably legitimate: the hopscotch nature of the Imagenet gains (in which it seemed like every month a different deep-learning team would turn in a new record) means that we can be confident that the overall performance curve of all teams is genuine unless they are all colluding, and this particular method of cheating previously can be ruled out just by looking at the submission scatterplot and cumulative submissions in the official statement ( http://www.image-net.org/challenges/LSVRC/announcement-June-... ) - we can be sure that competitors and previous records did not overfit like this because there just aren't a whole lot of previous submissions to the test set.
So now the record is probably more like 4.5% than the 4% or whatever Baidu's final paper was claiming. To keep this in perspective, that's a few months or a year of progress, the Imagenet record is still human-equivalent, and deep learning is amazing.
> This brings quite reasonable questions about their past performance.
I don't think this second claim is necessarily derivable from your first. Yes, they were caught cheating... By making multiple submissions of their own systems in order to faster derive which is the most promising. IMO, the performance they have achieved is still theirs to claim, because in the end they did create a system that reaches that performance.
Ideally, your final test data set should be completely separate from training data. Because even limited exposure quickly renders that test data meaningless.
PS: In the end it's much like taking the same test with word for word identical questions a second time. Yes, it's a test of something, just not the intended subject matter.
It is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field.
Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting.
A good short description on several issues that plague the field is "Clever Methods of Overfitting".
> getting very close to better-than-human performance
I think they've all surpassed the human-level baseline by now. There's some stuff in there, like telling different dog breeds apart, that humans are not very good at.
Andrew Ng made several posts about this result on Google+, most of which have now been removed. The reshares are still available though (like https://plus.google.com/+HeikkiArponen/posts/TaTm9jpCxqx). He doesn't appear to have worked on the project though -- the "Deep Image" paper just credits him for "many insightful conversations."
Maybe they actually don't care about the captcha aspect - they just found a good mechanism to easily acquire data from humans that no one will complain about because it's so entrenched.
Cheating is pretty entrenched in Chinese culture. Sometimes I even wonder if they consider it's cheating at all or if it's just cleverness and they should be rewarded for it. I believe they feel that way about infringing copyrights too: why should they follow someone else's rules that puts them at a disadvantage when they can instead demonstrate cleverness and break those rules?
There was an interesting story recently about how a disproportionate amount of Chinese students are expelled from US universities for cheating.
The problem is so entrenched that some students rioted when they were prevented from cheating, because they felt that this put them at an unfair disadvantage compared to other schools where cheating was tolerated:
If you look at it another way, it's not so bizarre. The person is retrieving their information from another's head. They're using a social storage mechanism. This works fine in the classroom, it works OK in the workplace. It works well in most situations. The only place it breaks down is during testing.
Because standard testing is only supposed to verify that you're storing the data locally. Standard testing is designed in a way that only helps "local storage" people, and alienates these users who get their information from their environment.
Open-book, open-note, and open-internet testing removes this barrier and levels the playing field for these "network storage" learners.
> The person is retrieving their information from another's head. They're using a social storage mechanism. This works fine in the classroom, it works OK in the workplace. It works well in most situations. The only place it breaks down is during testing.
Many tests verify a skill rather than knowledge. If you're cheating then you're masking your lack of skill by using another person's skill as a crutch. If you're later given the relevant qualifications and end up in a situation where other people expect to be able to rely on you having the said skill then you're likely to fail these people. Bad.
In today's world, remembering specific pieces of information locally is less of a concern, I agree, but for basics it still applies: you may be able to learn it from colleagues or internet when you need it, but there is still a limit on your "input bandwidth".
You may be right, but huge problems arise with the open-everything-testing solution with regards to access to resources, especially with standardized testing. SATs (and standardized tests in general) are meant to be a blind test that objectively evaluates the abilities of any individual student anywhere the same way; if we allow external resources, this will undoubtedly be a huge disadvantage for students in lower income areas who don't have easy access to the Internet or a sufficient number of reference books and materials for all the students. Providing internet access at testing sites would help somewhat, but if you have grown up in an environment where the internet was not as ubiquitous as some of us are used to, you may not have the Google-fu or knowledge of Internet research methodology necessary to compete with those in wealthier areas. This is an issue that will hopefully be diminished over time, especially in developed nations, but it will be a very long time before Internet access is sufficiently widespread in what are now developing nations to prevent a huge barrier to entry for students in these places.
Maybe students who are disadvantaged on their internet access could test on other subjects, such as math or physics. It seems unfair to penalize the other people who have access to information and can use it.
As a child I was not allowed to use calculators in math - for the reason that "what if your computer breaks?" - well, it never broke. It was a stupid rule, especially after 4th grade.
Computer access will always be available, so why ban it during testing?
"what if your computer breaks?" is a poor reason. The real reason is to get an estimate of results in your head so you can cross-check the computer for mistakes. How much ice cream to feed my city of 30,000? If the computer says 7.336E9 kg, that's a problem.
That would be OK if the rules were the same for all teams. In any situation, classroom or not, if you're breaking the rules you are cheating. This doesn't mean that "network learning" is wrong, just that doing this without common approval of all participants is not socially acceptable.
Proctors rarely get a common agreement. They make the rules, then you're forced to either accept them or completely forego the examination - the equivalent of failure, it negates the time you've spent training.
I was once in an exam hall and middle-aged female primary school teachers were in the next row writing a trivial (non-degree) exam on primary school mathematics education or something [^]. They were brazenly passing notes to each other even though the exam was open-book. Cheaters gonna cheat.
[^] With teachers like that, it isn't surprising that South Africa is so poorly ranked at maths.
Why is it even bizarre? Chinese people cheat for the same reasons anyone would cheat. Being chinese and born in America, I actually found it really surprising the first time I met software engineers who refuse to download even one copy of pirated software.
Culturally we just feel significantly less guilty about it. Doesn't mean we aren't aware that it's wrong. Chinese people are fully aware of what's right and wrong and we choose to deliberately cheat. Whether that's a cultural tendency or genetic one is another story.
> Chinese people are fully aware of what's right and wrong and we choose to deliberately cheat. Whether that's cultural tendency or genetic one is another story.
What?! Genetic?! There is very little, if any evidence to support the idea that genetics has anything to do with the Chinese propensity to cheat...
There's no evidence at all. Even when attempting to do research on this, the political backlash could destroy any scientist's career. No official evidence will ever be collected because of this.
I only speak from personal experience. I know many chinese people both born here and born abroad. I am also chinese and born in the united states. I am telling you, honestly, from a purely anecdotal standpoint: I think there's a chance it's genetic.
Edit: Just to keep things from getting out of hand, and more balanced I want to state this fact: Statistically, it is far more likely for a serial killer to be a white caucasian male then it is for a serial killer to be of any other race. Do I think this is a cultural thing? No. I'm leaning towards genetics. But that's a purely anecdotal opinion as there's no evidence pointing in either direction.
> Even when attempting to do research on this, the political backlash could destroy any scientist's career. No official evidence will ever be collected because of this.
This is really offtopic, but I think you're wrong. There are two ways to do such research - the proper one and the racist one.
The proper one is to link specific genes to specific behavior. This presumes that you screen individuals in the study for those genes. Then you are not being racist, because existence of those genes is rarely 100% coinciding with culture or skin color. Research like that is quite common, if there is a reason to think there is indeed a link (for example there is a genetic study of above average intelligence in Ashkenazy Jews, they were extremely homogenous group and yet they don't all share the genes that has been shown to have the link).
The racist way is to match skin color or culture to behavior. This is scientifically useless, because you don't show any actual genetic link, and it only serves for stirring racial hate (you could pick any indirect attribute from many, like eye color or facial hair, so why pick skin color?). Such studies are rightfully being rejected by real scientists.
Why would those things be genetic? If there were a "serial killer gene" or a "cheater gene", those traits should manifest themselves in all kinds of obvious differences in behavior - which we do not see. It's hard to imagine a protein causing such complex differences in behavior while affecting nothing else.
You don't have to go to the absurd "cheater gene" degree. Note this is not at all scientific, but rather trying to point out how easy some small genetic things might push society and culture in particular directions.
Suppose instead that a few traits like:
- Social Intelligence
- Impulse Control
- Desire for Retribution
have normal distribution with slightly different peaks in different populations.
A population with high social intelligence and low desire for revenge might tolerate more cheating(since people only cheat when they can get away with it) than one with low impulse control and high desire for revenge, where cheating might spark a shootout.
You probably don't have to move those distributions much to start seeing pretty large changes in things like mass shooting demographics, or who cheats.
What's the other explanation then? Culture? Why don't we see higher rates of serial killers in other races born in the United States?
It's very possible for many differences in behavior between races to be genetic in origin, in fact it's the more logical hypothesis versus the alternative which states genetics doesn't influence behavioral differences between races.
Think about it. If genetics influences physical traits from height, skin color, facial features, and even athleticism, what black magic in this world makes it so that genetics doesn't even touch behavior or intelligence?
>It's hard to imagine a protein causing such complex differences in behavior while affecting nothing else.
It's impossible to logically deduce a conclusion from the bottom up. We simply currently don't have enough knowledge to know how proteins scaffold the entire human neural network. With highly limited knowledge, we can only look at the problem from the top down. That being: genetics is known to influence physical traits, therefore it is logical to conclude that it also influences mental traits.
> If genetics influences physical traits from height, skin color, facial features, and even athleticism, what black magic in this world makes it so that genetics doesn't even touch behavior or intelligence?
Because we have no evidence that such is the case on a culturally grouping level.
There is no genetic concept of "Chinese". It literally doesn't exist.
You're making this about the possibility of genetics impacting behavior, when the real issue is you thinking cultural boundaries exist in genetics. They don't.
There's no concept of humanity on the atomic level. It literally doesn't exist. One configuration or mishmash of atoms we call rocks are no different then the mishmash we call humans. Try, without using any high level concepts or groupings, to define what configuration of atoms signifies a rock and what configuration signifies a human.
If you go low enough on any topic the boundaries between categories become vague and the definitions become extremely complex. It's very hard to define what a human is in terms of atoms. The same goes for race, it's very hard to define, at the genetic level what is chinese, and what is not, but the category and boundary exists at all levels, and we can't ignore it.
I've heard of your argument before. They say that the delta in genetic differences between two people of different races is the same as the delta of two people, of the same race, therefore race doesn't exist. This argument is flawed. I believe the "genetic" definition of race is immensely more complex than simply the delta of genetic differences. Here's a more accurate definition: People of the same race have a higher probability of sharing certain genetic traits.
So let me redefine my argument in way you can understand. The people who we label as "chinese" who share similar physical/genetic traits, I believe will be more likely to also share a behavioral genetic trait that makes them more likely to cheat.
I'm sorry, you misunderstand -- the people we label as "Chinese" do not share similar physical/genetic traits.
Common misconception that they do, but there is very little genetic consistency across cultural boundaries, and when such a thing does exist, it's quite noteworthy.
first paragraph from above page:
"The relationship between race and genetics is relevant to the controversy concerning race. In everyday life many societies classify populations into groups based on phenotypical traits and impressions of probable geographic ancestry and socio-economic status - these are the groups we tend to call "races". Because the patterns of variation of human genetic traits are clinal, with a gradual change in trait frequency between population clusters, it is possible to statistically correlate clusters of physical traits with individual geographic ancestry. The frequencies of alleles tend to form clusters where populations live closely together and interact over periods of time. This is due to endogamy within kin groups and lineages or national, cultural or linguistic boundaries. This causes genetic clusters to correlate statistically with population groups when a number of alleles are evaluated. Different clines align around the different centers, resulting in more complex variations than those observed comparing continental groups."
In short it's saying genetic traits can be statistically correlated with population groups (race) but variations of traits that are different within population groups can actually be more complex than those observed when compared with people outside of their race.
This is literally exactly my argument. Supported by wikipedia at the very least.
I do not accept the given definition of race from this page, as it presumes the term "race" is in any way scientific or rigorously defined when in actuality it is not.
What we "tend to call" race is not defined, despite this wiki page's attempt to do so.
This wiki page is the reflection of the general opinions of the scientific community. You can redefine any word to have any definition that fits your universe, but when communicating with other people, we must go with general consensus.
A word not having a rigorous definition means it cannot be discussed scientifically. Hence the actual problem of studying the existence of life, e.g. is a virus alive?
>A word not having a rigorous definition means it cannot be discussed scientifically.
Life is discussed scientifically in many contexts yet it is not unequivocally or rigorously defined. In fact there's an entire field based on the study of life. It's called biology, aka the study of life. If a scientific field can stem from a word that does not have a rigorous or unequivocal definition, then it can be discussed scientifically.
I'm getting pretty tired too. You choose not to accept the facts even when a scientific description proving my point is thrown in your face. Ideas need evidence for support, you have presented me with ideas, but no evidence.
The folks in the field are in agreement with me, (see the old wikipedia link I sent you). You got nothing, only empty claims.
I'm getting tired of this conversation, so I'll just leave you with the idea that words carry different definitions in different contexts. There is no scientific context by which "race" is currently known. You can choose to accept that, or you can continue to deny that, it doesn't really matter to any of the folks who work in this field.
> Chinese people are fully aware of what's right and wrong and we choose to deliberately cheat.
I personally got bit by this in my work.
This is exactly why I'm never ever going to work with a Chinese again in my life time. I should also tell about this to people I care about.
It's the unfortunate truth, imo, that chinese people are more likely to cheat. This does not preclude other races from cheating though, and it does not mean ALL chinese people cheat.
I'm not sure if it's a cultural thing as you say or what, but I have heard of universities just assuming all numbers from Chinese national applicants have been doctored.
Someone who's actually Chinese, and went to school in China, would know best, but my feeling is that it's more simple - Chinese culture just places a really high value on results, as opposed to broader goals like learning, advancing a field, etc. If the grade/win is all that matters, then cheating is just a way to get there.
Reading my post again, it was way too black-and-white. I was typing it out on a phone, so it's hard to express anything in detail. Of course cheating is decently common in all cultures, my point was that I didn't feel that cheating was respected as a sign of cleverness in Chinese culture, more that if there's just huge pressure on you to get high marks, win, etc., and little focus on other side goals, then cheating is sort of a natural side effect.
For example, in high school and university, most of my friends had parents who wanted them to do well grades-wise, but also wanted them to focus on other things like their social life, hobbies, and developing a passion in a specific area. Low grades wouldn't draw any more concern than not socializing much, not having hobbies, etc., and a "well rounded" life with mediocre grades was general seen as "my kid's doing well."
Among my Chinese friends, it was more common for their parents to focus on grades above all else. The parents weren't that interested in the side goals, they really just applied huge pressure to their kids to get the highest grades. The kid could be doing all sorts of awesome things, but if their grades were mediocre, their parents were generally pretty disappointed.
I figure it's more likely something like this with the Baidu team - major pressure to win, above all else, that would drive the cheating, vs. cheating being seen as some positive sign of cleverness.
> There was an interesting story recently about how a disproportionate amount of Chinese students are expelled from US universities for cheating.
That's not what that article claims. You cannot conflate poor grades and cheating. The article intentionally misleads by just listing % for all reasons that Chinese student are expelled and then hints (without outright saying) the reason must be cheating, it couldn't be for example poor English skills? Or a ton of other problems international students might have (e.g. cultural adaption)?
I don't know if Chinese students cheat more (I don't have that data, and the article doesn't supply it). But I do know that Americans spend a lot of time writing anti-Chinese stuff and it is getting pretty pathetic.
>Demand for an overseas education has spawned a cottage industry of businesses in China that help students prepare their applications. The industry is poorly regulated and fraud is rampant. According to Zinch China, an education consulting company, 90 percent of Chinese applicants submit fake recommendations, 70 percent have other people write their essays, 50 percent have forged high school transcripts, and 10 percent list academic awards and other achievements they did not receive. As a result, many students arrive in the U.S. and find that their English isn’t good enough to follow lectures or write papers.
>In the survey, we found that the main reason was expelled classified into the following nine kinds: poor academic performance (57.56%), academic dishonesty
(22.98%), attendance problems (9.67%), misconduct (3.87%), international student identity (2.96%), in violation of the law
(1.93%), psychological problems (0.57%), financial problems (0.23%) and Shen learn material legacy issues (0.23%). Some were expelled from reason
There is a causal relationship between the links. But the statistics, we tend to count the cause and expelled most direct contact.
Its pretty entrenched in US culture too. Example one, downloading "free" stuff off the Internet that used to have to pay for in a store. People make all kinds of rationalizations why this is not stealing.
Example two is just about any US high school or college. The US Air Force Academy graduated a 20% smaller class than normal his year because so many were expelled in cheating scandals. A major sociology paper was just retracted in Science, on of the worlds leading journals, because a grad student faked the data.
> People make all kinds of rationalizations why this is not stealing.
Arguments that it is stealing are also nothing more than rationalizations. That's what both sides wield: it's a war of rationalizations.
(I tend to be more convinced by the "it's not stealing" rationalizations myself, simply because stealing an object makes it unavailable. If someone stole my bicycle, I'd be upset. If they cloned it in a Star-Trek-style "replicator" and took the copy, I couldn't care less. If you lend a book or DVD or SD card or whatever to someone, upon it being returned, you have no way to test the hypothesis "a copy of this was retained by the borrower". Not testable is not real, for all practical purposes.)
Without derailing this into yet another "copyright infringement isn't stealing" thread, as far as the creator is concerned, they don't get compensated under other circumstances which are not commonly seen as infringement or even morally wrong. Example: A roommate watching a movie or playing a game with you.
Spinning it as purely a compensation issue is being a bit reductive, IMO. And from there it leads directly into objectively unanswerable intent questions.
Just wait; I'm going to obtain an exclusive distribution license on DNA from the oldest bacterium in the world. All life will then have to compensate me every time any cell divides.
Also, since DNA controls what you look like, including your silhouette, you will have to pay me every time you allow a light source to cast a shadow of yourself onto a flat surface.
Also, when you take a bath, you displace a volume of water corresponding to your shape. That is an unlicensed copy.
Your footprints in the dirt or snow, ditto, not to mention the fingerprints you leave on everything you touch.
---
Man, think about it. Millions of years of evolution (based on continuous unrestrained copying and alteration on a massive scale) lead to Man. What does man do? Put three guitar chords together with some puerile, hackneyed lyrics, and insist on compensation from everyone that hears the crap.
Absent piracy, the money in the pirate's pocket would not magically have been in the content producer's pocket, which is the assumption underlying your ostensibly factual statement.
> The theft isn't of an object, it is theft of compensation due to a creator. Duplication isn't the issue.
Based on my limited understanding of copyright law, it does in fact concern itself with actual duplication, and is rather mum about compensation.
You can infringe by, say, altering something that is freely distributed in a way that is not permitted by its license (for instance changing the copyright notice to say that you created that work, rather than the author). The author could be adamant that permission to do that is not granted at any price (so that it's not an issue of compensation, but purely about unlicensed duplicates with unauthorized alterations).
But I do have a way to test for it. I can break into their house and (properly) steal their computer, then analyze its contents. I'm just not supposed to. That hardly makes their copy "not real, for all practical purposes."
Breaking into a house - we have a name for that - it's called "breaking and entering" and "trespass".
However, going along with your analogy, say you're in my house - from my perspective, I would care about what you did.
If you took something that was mine, and denied it to me, I'd be quite annoyed. (I've had a expensive road bike stolen, so yes, I know what it feels like).
However, say you took photographs of everything I owned, to copy my packing style and organisational techniques? Or took them so that you could re-buy everything I have for yourself off Amazon? I probably wouldn't be annoyed, so much as think you odd...
> That hardly makes their copy "not real, for all practical purposes."
Your example of how it's "practically" impacting your life is that if you happen to commit a felony against the person and then analyze their computer explicitly looking for it, you'd happen to find out?
I think that's the very definition of "not a practical difference in my life", given the distinctly low likelihood that series of events would ever happen.
The parent claimed that it was impossible to test for the existence of a copy and that therefore its existence or lack thereof is not relevant. I claimed that it is possible to test for the existence of the copy. I would further claim that there absolutely is a practical difference as far as a renter is concerned, whether or not they test for the existence: The person who copied the rented DVD is very unlikely to rent it again when the want to watch it later. Which I presume they intend to do given that they went to the (admittedly minimal) trouble of copying the movie in the first place. I suppose that in the scenario where it is a lent item, the argument of lost revenue is slightly less strong.
It's disingenuous to equate fraud -- cheating in a zero sum game -- with violating a purely manufactured right like copyright.
Unlike fraud, the ethics of violating copyright are highly situational and quite complex. A recent example of mine:
I purchased a very expensive publication; my purchase included a PDF copy that was (unbeknownst to me) DRM-locked with very onerous DRM.
- I fired up IDA Pro and broke the DRM so that I could read the PDF without their invasive DRM; I have no intention of distributing the results. This is actually illegal, and in violation of the license under which I was granted any rights to the PDF. Was this unethical?
- If I couldn't break DRM on my own, would it be wrong to download a copy from the internet?
- Given that I own the physical book, if my purchase hadn't included the PDF, would it be wrong to download a PDF of it?
- If I wanted to know what was in the publication before I paid a great deal of money for it, would it be wrong to download it?
- If I didn't have the money to pay for the PDF (as was often the case when I was a teenager and where I learned to crack software), would it be wrong to download for free -- but not distribute -- the PDF? How wrong?
It might be cheating, it might be copyright infringement. Depending on the circumstances it may or may not be moral. But 1000 unauthorised copies do not mean 1000 lost sales. The situation is more nuanced than a simple 1-1 correspondence.
To put it the other way - all my life I've had free music. From the radio. In the car. In stores. Elevators. Birthday cards. Gas stations. Retail workplace. On the bus. Now even on Youtube and thousands of other streaming video sites.
And now you suddenly want me to pay for it? Why, what changed?
You've never had free music (unless you stole it). Radio and YouTube are free due to advertisements. All of your other examples are either public domain or a business paying to play music for you.
The funds were allocated to recording companies, not directly to artists. Which means the artists rarely saw any of that money - the percentage of that payment to which they were entitled rarely rounded up to 1 cent.
When an ad comes onto one radio station, you switch to another without ads. Redundant array of independent radio stations. Advertising is damage. So advertisers coalesce their ad timing, so the damage happens on all independent channels simultaneously, so you turn off the radio for several minutes, maybe switching to Pandora or another stream.
When Youtube plays ads, you mute it and goto another tab, and keep watching a different video. When the ad is done (such nice ads for telling you their duration), you simply return to the tab and watch your content.
There's always another service that's looking to hook you before they advertise. So you can just switch between them.
It's not stealing if another person isn't deprived of an object. That is the definition of theft. Not a certain series of bytes travelling into my NIC. IP zealots need to make all kinds of rationalizations for why this is "stealing".
What is stealing, however, is the theft of rights perpetrated by the content industry by extending the terms of copyright so they can "own" ideas in perpetuity.
When I (and many others) say "piracy is not theft", it's not to "rationalize" piracy. It's to fight incorrect terminology, because such loose, imprecise use of language muddies the discussion and hinders productive discourse. Piracy is not theft, nor is it murder, nor is it rape. It is a different crime, with different costs and repercussions.
Not that you are incorrect or that this is not an interesting aside, but I find these conclusions a bit dreadful. "American dies after eating 400 hotdogs in eating competition" = "unhealthy fastfood is pretty entrenched in American culture".
The article uses another hyperbole. "But [about Baidu cheating] artificial intelligence researchers have a more basic concern: that their work will once again fall short of expectations, leading to yet another fallow period for their field.". There are now Machine Learning PhD. thesis on the frontpage of HackerNews. This field is not going anywhere barren soon. Researchers are not worried about their own work in response to Baidu being caught out for cheating. The Rosenblatt quote was after a question by the NYT journalist, he did not give that forecast in a paper. I also think he did not specify a timeline of less than a year. Most if not all forecasts have come true by now. His claims got vindicated.
> "American dies after eating 400 hotdogs in eating competition" = "unhealthy fastfood is pretty entrenched in American culture".
Not for nothing, but I'd actually agree with that conclusion, based on your example.
If a culture promotes fast food to the point where competitions related to the rapid consumption of the fast food causes a death, I'd consider it a symptom of the larger "fast food culture" problem for that society.
Competitive eating (including stuffing 110 hot dogs down your throat[1]) is popular in Japan. Is "unhealthy fastfood pretty entrenched in Japanese culture"?
Besides this not being true. Can we please try to be a little political correct? Imagine someone writing "Crime is pretty entrenched in Black American culture" or "The feeling of superiority is entrenched in Jewish culture" - that would not be so nice, right?
Can you provide some support for your assertion that this is false -- that there are no cultural differences between Chinese cultures and Western cultures in terms of attitudes of what is "right" in terms of cheating, plagiarism, or copycat/shanzhai goods? Note that this is not just a matter of whether a culture has cheaters or not, but to what degree it is tolerated in either practice or in the stated beliefs of a culture.
Culture is not just cuisine and costumes -- it is also how one interacts with the world and others in one's society. There's a certain arrogance in assuming that all people in the world must obviously approach things in the same way that you do.
I can't help but feel that your evidence that it is false is that it is politically incorrect. Prove me wrong?
(Bear in mind that one can not have both "There's a lot of diversity in the world, and that's a good thing, and we should seek more diversity" and "All cultures are precisely the same". Pick at most one.)
Being chinese myself I would have to say he's more right than wrong. I would even go so far to say that our tendency cheat could have genetic origins as well. It's anecdotal, but I'm born and raised in the United States, I'm culturally influenced by american culture, yet I totally identify with and understand this propensity among the Chinese to cheat. As a result, I think it's deeper than just culture.
Why is this such a foreign concept? If genetics influences physical features, why doesn't it influence behavior? Do you choose to not take my comment seriously due to logic or politics?
I'd say it has a lot to do with the collectivism of the culture - there, sharing and "standing on the shoulders of giants" is emphasised more than independent thinking.
I think it has more to do with the guilt vs. shame culture metric anthropologists and psychologists talk about and have documented at length. Guilt gnaws at you even when nobody else knows you've done wrong. Westerners score relatively highly for it. People in East Asian societies tend to have a less developed sense of guilt. Instead, they feel bad for having done wrong to the extent they are shamed and shunned by the surrounding society (loss of face). For things like cheating on tests it's easy to see how it's more difficult to maintain standards by shaming mechanisms.
> Sometimes I even wonder if they consider it's cheating at all or if it's just cleverness and they should be rewarded for it.
There is a quote that comes from sports but I think is more apt for business, "If you're not cheating, you're not trying" and ties well with "It's easier to ask for forgiveness than permission". This is a competition so they cheated and lost, but their gains from the work itself will/should pay dividends in the real world.
The culture doesn't provide an association between cheating and punishment but only academic success and reward.
Anecdotally speaking, a lot of international Chinese students cheat blatantly - i.e. comparing answers with the person beside them at the end of the exam when people are turning in (and often get caught doing so).
Generalizations like these piss me off to no extent. Because students have been expelled, therefore, it must be true that Baidu tried to cheat and there can be no other possible explanation? The first conclusion you jump to is "because Chinese culture"? Really?
The naked racism of this comment is saddening. I'm sure we all know pockets in our societies where cheating runs rampant. Thankfully, people aren't regularly trying to act as though those pockets represent the whole.
So if someone were to say that Spanish culture is X, where X was negative, you wouldn't consider that racist either? Perhaps 'racist' isn't exactly the right word, but that seems like splitting hairs.
It's not really cleverness if you break the rules so obviously that you get caught and disqualified.
Clever would be finding a loophole in the rules, a way to run your tests without it getting counted as task submission, make it look like a bug or something like that.
Without the exclusion acts the American West coast would probably have become majority asian a century ago. It's doubtful the territory would have remained American.
In a globalized world, institutions need to tighten up their safeguards against cheating, fraud and dishonesty. In much of the world, cultural attitudes toward cheating are a lot more relaxed than some of us presume.
There was a story about Indian students' rampant cheating at US colleges, that was presumably flag-killed off HN despite the horror stories that were emerging from academics. I've read about Australian universities basically selling degrees to foreigners who can't speak English and get admitted fraudulently. There is a currrent story about fraudulent admissions to US colleges by Chinese students as a result of massive SAT cheating. I'm not picking on Asians: Switzerland seems to have its fair share of scoundrels, as the FIFA scandal reminds us.
We need to acknowledge that gloablization sometimes brings unwanted side-effects and deal with these head-on.
In my opinion, tightening up the safeguards isn't the right approach. Rather, institutions, particularly educational institutions, need to fulfill their obligation to inculcate the right values in people. Cheating and petty corruption culture is an existential threat and the solution isn't to catch the cheaters, its acculturate them to follow the rules.
Educational institutions, however, have totally abandoned this obligation. Shocking cheating behavior will merit just a note in someone's record, if administrators are even willing to take it that far. Cheating isn't publicized, shamed, and punished in the way necessary to have any impact on the cheaters' values. Instead, everyone gets to save face.
When budgets depend on covering up cheating, cheating will be covered up. Educational institutions are failing to uphold their responsibility to society (partly) because society has largely abdicated its responsibility to educational institutions by slashing funding and grants. In the absence of public funding, the student (or the donor), not society at large, is the customer, and the customer is always right.
I totally agree with your opinion here, and especially that "cheating and petty corruption" - and the breakdown of public trust that follows - is an actual existential threat, something that may ultimately lead to a collapse of technological civilization. The current trend of trying to figure out trustless solutions for everything actually worries me.
The current trend of trying to figure out trustless solutions for everything actually worries me.
If you're worried about breakdown in the face of loss of trust, then these sorts of solutions seem like very important things to be looking at. Do we have any good ideas about how to create trust at the institutional level?
I agree with your concerns. I think they are representative of a broader theme - the acceleration of technical change seems like it will eventually (if it hasn't already) bring us to a point where the rate of cultural change is too slow to catch up to properly adapt to what is now possible.
In my university, a small ring of students were flagged by our Computer Science department's automated plagiarism software. Upon investigation, it was determined who the culprits were. All involved students ---including the ones who shared their code--- were awarded a grade of XF, "failure due to academic dishonesty". Since the course was a gateway to other courses, most of them were effectively expelled.
In another course, the professor and assistants would view videotapes of the lecture to catch students who were taking quizzes for other students. Those students were dealt with as well. The XF grade stays on the transcript forever.
It may be one data point, but there is at least one university serious about cheating.
I would like to generalize that to all American universities, but I don't have enough data.
I don't think this can be solved by punishing the cheaters, or "educating" them about the consequences. The problem is that expectations and pressure on individuals have gone out of hand. If you feel that your very existence depends on getting outstanding results, of course you will consider cheating.
An issue is, academic dishonesty leads to people making life-and-death decisions without even the base knowledge that credential implies.
They'll be the physicians treating you and your family. Or the programmer conducting code reviews on the critical aviation software.
Given individuals are (unfortunately) in this position in many professions, we need academic rigor (until we get AI to make better decisions than the average driver/doctor/programmer)
That will never happen. "Dealing with these" means accepting that your entire educational system needs an overhaul. Far easier and more crowd-pleasing to dig in your heels and say that foreigners are cheating scumbags because they don't want to play by your outdated rules.
Working collaboratively and using resources like the internet is okay, since that's what real people actually do. Labelling this behavior as "cheating" in an academic context is outdated.
This is a comical level of rationalization from someone who is obviously still in school (and if not, I don't even know where to begin...). The whole point of taking a test in school is to see how well YOU know the material, not how well you can look stuff up or ask other people for help. If you have so little respect for the material or education, just drop out now, because you're wasting your parents' money and you clearly think you're smart enough to make a living without the things that school is offering you. What the hell is even the point of going to school if you're just going to "collaborate" your way through it? Just skip the pretense and go get a job someplace if you don't need the knowledge.
You are theoretically there to learn your field, not to learn how to work "collaboratively", which is a vague enough description of cheating that it could be used to justify anything. Hell, if I rob a bank am I not just "sharing"? This whole concept of "theft" in finance is so outdated!
Yes, that's what I'm saying. The whole point of taking a test in school is misguided.
Also, it's obvious you are mentally disabled. Otherwise you wouldn't disagree with me, because I'm objectively right. You should go back inside your padded cell and take your medication. /s
The definition of "objective" isn't "because I said so". Or more likely it's a case of "caskance sucks at things like taking tests and not cheating, so he has erected an alternate reality where he shouldn't have to do those things and gets to be right". Mentally disabled, indeed.
Nope. Read it all. If your going to accuse someone of being mentally deficient, you should work harder to not be obviously an idiot yourself. You failed at that.
Who wrote that headline. It makes it sound like they were barred for no reason and the community is mad. A more accurate headline would be "Baidu Team barred from A.I. Competition for cheating". Its like they are trying not to get censored or something.
One important thing to recognize here is that this isn't just unfair play (in the sense of trying to win a competition by submitting multiple entries). It's a loss for all players. The point of having distinct test and training sets, and not training on the test set, is to ensure that your system can generalize (IE, work on things it wasn't trained on). If you don't do that, you're not making progress, just memorizing examples. Example memorization is fine on its own, but it's not really a true improvement.
Just some anectdotal evidence, but at work I had to implement a clone of our Google Maps application in Baidu Maps for our customers in China. The only reason I was able to use their API was because they copied Google's interfaces, in some cases, word for word.
Evidence in the context that Baidu "cheats". I agree, it's not a bad thing. But from my experience it seems they copied Google's interface (which is functional and pretty) but had a complete misunderstanding of how to implement it. A lot of features that worked well with Google's implementation felt awkward or forced with Baidu's.
Google uses the APIs from Java 6 and back for the "boring" bits. Android Java is the de facto client Java today because Google abandoned the UI APIs in Java 6 and created their own. Sun's, then Oracle's argument is that Google is "harming" Java by breaking the standard.
I think there is subtile difference. Java was, is and shall be a standard. To say you had a Java implementation meant that you supported the Java APIs. It was a legal definition. Google had Java-like runtime that they promoted as a Java runtime. This led to the question of API protections and reasonableness.
Using Google's API for your own competitive product API is different. Google has not set a standard for map APIs. At least not officially. It might be THE standard for maps, but that is neither here nor there. At this point you could call it cheating. They took intellectual property and appropriated it for a clone. Plus side, we now have an implied standard. Downside, theft of intellectual property.
1. Google Play Services APIs: These are optional, proprietary APIs. You don't have to use them, and you have to treat them as optional if you want to operate across Google, Amazon, and other AOSP-derived Androids. What's the issue here? They're like any other proprietary API.
2. "Breaking" the "Java standard:" Google "appropriated," via a permissive open source license, an open source implementation of some Java base classes, and the (not protectable) syntax of the Java language. They added support for their own remote API feature and a bunch of APIs, mainly to make a usable UI system. The result runs on a runtime Google devised, using a bytecode Google devised. Where is the intellectual property theft?
It is odd that there is such a significant cultural difference. On the other hand, the difference in culture may be more subtle than we realize.
For this test, they should have had an automated way to enforce that rule, if possible.
Look at the actual levels of piracy in the US versus China. In the US, you will get some people who say they are worried it might be wrong or that they will get caught. But most people will say something like 'watching this free stream doesn't hurt anyone'.
In the US, we have no problem with high end shoes that look similar to one another, or low-end similar to high-end. But there is a subtle distinction -- there must be some way to claim that this is not copying the other. But maybe it more often comes down to taking a different attitude towards the same thing. E.g. "cheap Chinese knock-offs" versus "inexpensive sensible alternative to overpriced designer brand".
Its also perfectly fine in general for companies to copy a successful business model. But we insist that there must be some distinction. However the difference between these companies may only be slightly greater or mainly surface-level, and so when you get down to the fine analysis, I thinj the real difference between the cultures is smaller than people want to admit.
Why is the competition designed in such a way that it is an advantage for a competitor to "run test versions of their programs twice weekly ahead of a final submission"?
How does it work? You submit your classifier to some server and it is run against what? The data set that determines your final score - hopefully not.
> Why is the competition designed in such a way that it is an advantage
See my comment & links at the bottom of this page. For some reason HN always puts my comments near the bottom, even when they're getting many upvotes. (I'm guessing that HN has a comment-editing penalty from editing my comments too many times, but I'm not sure, or maybe it's from times I must have insulted the mods here.)
Edit:
I constantly get "submitting too fast", so I'll reply here.
Thanks for the info kefka. Yeah, I'll happily agree with that linked comment, that the childish crude moderation techniques used here have been hurting a once great community. It was the exact same scenario for me as for that guy. Seems I got rank-banned immediately after confronting dang about the trends of excessive downvoting and increasing side-project criticizing on HN.
Is it correctly understood that in "Large Scale Visual Recognition Challenge" the competitor's biweekly runs ahead of the final submission is against the set that actually determines the final result of the contest?!
That's what I was wondering. If so, submitting many test runs makes it trivial to provide a submission that works perfectly against the test data specifically.
...for some definitions of "trivial" and "perfect". At this level, I suspect even a small advantage would result in winning the contest, which is the point here.
The technique is actually pretty fascinating. This is something that's been well understood by the cryptography community for decades, but is somehow just recently being fully appreciated by the ML community. See here:
Submitting guesses to a system that gives you back scores for your guesses, will quickly leak out enough information that you can reverse engineer a huge number of hidden numbers/labels in surprisingly few iterations, e.g. 700 iterations to covertly extract 10,000+ real numbers with high precision. This surprisingly rapid convergence is a bit reminiscent of the birthday paradox.
Further, this not only lets you win the against the "test" dataset, as apposed to the final "validation" set, but this allows you to significantly increase the data available to you to train your model on, since now you can train your model against both the "test" and "training" datasets.
Layman summary -
ML breaks datasets into 3 partitions "test", "train", and "validation". In cases where they're evenly split, this technique can double the training data you have access to, which is a massive advantage in ML competitions where scores differ by tiny amounts.
Moral judgement -
My opinion, this moral argument is misdirecting the attention from where it needs to be. Yes, it's bad what occurred here. But at this point, in 2015, and with tools readily available to crack this problem effortlessly, it's inexcusable for contests to allow so many scoring reports against their validation sets anymore. It's no longer a question of whether contestants will do it, but how many of them will. We'd might as well just let people self-report their scores on an honor system, if we're going to be this overly trusting.
Try creating a contest system like this in the cryptography field any time in the past 3 decades and you'd be insulted and laughed out. Allowing so many scoring reports against the validation set is fundamentally flawed. The only solution is to globally limit calls to the scoring api.
Another proposed solution -
Allowing everyone to see everyone else's guesses & resulting scores against the "test" set, so that everyone is on equal ground for reverse engineering the "test" set, and then globally limiting the number of scoring attempts so that the test set isn't reverse engineered too significantly.
Overfitting the "validation" set actually is not a problem either way, because none of these contests are dumb enough to let anyone score against the validation set at all until the contest submission deadline is over.
Academia is basically "self-reporting on the honor system". It works generally but there are lots of holes. Ultimately, "trust but verify" is necessary to avoid getting caught in a wave of hype, or at least having someone in your own "circle of trust" say it works. This system leads naturally to elitism and a bunch of other problems which are seen in academia, but it seems better than the current alternatives to me given the current rabid focus on exact percentage score instead of quality/utility of an idea.
The "right way" to do it is test once only per model/paper. If you are interested there are a huge number of sneaky ways overfitting can happen in ML [1]. Also interesting that you too see crypto and ML as related - I see them as opposites of the same coin. One tries to pull signal out of noise, the other tries to bury the signal in noise... but special noise.
Why not keep the validation data secret, give the teams both the training and test data, and make sure they know that their pre-validation submissions are running against data they already have---they're really just testing the submission process?
HOWEVER, the goal of these contests should be to promote the most accurate and powerful image recognition algorithms that will transform the world as we know it. Limiting access to training data makes it more difficult to test changes to an algorithm. These rules do not make sense to me, and I would advocate against them.
Two things - firstly you need to understand the concept of overfitting : http://en.wikipedia.org/wiki/Overfitting . If teams were allowed to train on the full dataset, it would be possible to get a 100% score, yet still not have a model that was useful on any images that were not in the training set. Furthermore, if you allowed infinite submissions, teams could just train a million models with slightly different hyperparameters, and submit the one that did best (which may be what Baidu was trying to do here). This is a problem because now there is the possibility that you are overfitting the test data - there would be no way to tell if the accuracy generalised to other images without coming up with more labelled data, i.e. making a new test set.
Second of all, cross validation : http://en.wikipedia.org/wiki/Cross-validation_%28statistics%.... You don't HAVE to submit to the test set to get a feel of how well your model is performing. On datasets this large, cross validation should be effective, if more time consuming method (unless your model is extremely unstable).
Luckily you don't work in the field. You can get perfect accuracy on a target set while having approximately zero accuracy on any other set, if allowed to run infinite tests. This isn't cheating because it helped them be better. It's cheating because it helped them get results that look very good but won't generalize well.
In a contest like this, I would expect (and I could be wrong) that there is a final validation set not accessible until "game day". Thus, overfitting to a training data set of any kind would be detrimental and degrade model performance.
They can use a subset of the training data as a validation set.
Given enough attempts at the test set it's very easy to overfit your model to that test set, meaning that although the accuracy looks higher, it would generalize worse on a different subset of the data.
I suspect this reasoning, or something like it, is the same faulty rationale that Baidu used to justify their cheating at the competition. As others have already said better than I could, if the goal is indeed to
>"promote the most accurate and powerful image recognition algorithms that will transform the world as we know it"
as you say, then running hundreds of attempts at this one data set to do better at this specific competition won't foster that goal.
Funny for a moment I thought some one submitted a headline with an Indian word in it. Asthir in Hindi/Bengali means unstable, not stationary or worked up (remotely connected to Astir?)
My thought is that the "a-" prefix in this sense probably comes from the Germanic use "to show a state, condition, or manner"; http://en.wiktionary.org/wiki/a-#Etymology_2 . It's no longer used to make new words, but the "a-" form remains in many words, including abloom, aflame, and abuzz.
The first known use of the word "astir" is from 1765, says Merriam-Webster at http://www.merriam-webster.com/dictionary/astir . This is what I would expect if "stir" was a word, "a-" was a possible mechanism to create new words, and eventually people started to use "astir.
> "up and about," 1823, from phrase on the stir, or from Scottish asteer; from stir. Old English had astyrian, which yielded Middle English ben astired "be stirred up, excited, aroused."
> Old English styrian "to stir, move; rouse, agitate, incite, urge" (transitive and intransitive), from Proto-Germanic * sturjan (cognates: Middle Dutch stoeren, Dutch storen "to disturb," Old High German storan "to scatter, destroy," German stören "to disturb"), from PIE * (s)twer- (1) "to turn, whirl" (see storm (n.)).
Hindi is another descendant of Proto Indo-European, so that may be where there's a connection to "asthir". However, do bear in mind that the surface similarity is false - astyrian would be linguistically closer to the Hindi than astir.
Thanks for the informative post! However I think asthir may be a false friend in this case. The a- prefix is used to negate sthir in asthir. Sthir in Indian languages means stable, stationary, motionless etc. Kind of like the a- prefix in English sometimes (social, asocial etc.)
But it is not uncommon for word meanings to change in such a way that they take on a meaning exactly opposite of what they used to mean (see for example the meaning of the word nice: http://www.etymonline.com/index.php?term=nice&allowed_in_fra...). I wonder if there is a name for this phenomenon.
In science and technology cheating is often self-correcting. Your result is not reproducible or your product defective if you cheat too much. Everyone will know.
I'm not an expert, so apologies if my question is stupid.
Why can't the competition have the same test data each week across all participants? So that no matter how many accounts you create, you will train with the same images everyone else gets to train with.
I began watching Andrew Ng's course just this morning. I had also watched some part of the courses by Pedro Domingo at the Washington University a few months ago.
Having never been employed but having coded for 8, I wish I could get myself educated in this field.
A major company spending so much, even risk their reputation, on a competition in a field where computer already achieved better than human performance. I start to feel that maybe the industry should start looking more on other new application of deep neural network (e.g. reinforce learning by deepmind). There is still a long way to go before we can achieve a thinking machine.
It wouldn't, because final scores are only evaluated against the "validation" set.
As for turning current contests into an exercise in overfitting the "test" set - we already reached that point long ago. Test vs validation scores often diverge wildly in these contests.
Edit - Replying to arnsholt:
Completely true. The huge problem I see, is that all the classic NLP tagging corpuses are created from the very narrow domain of news articles, and a few good corpuses now appearing for biology texts, and that's about it. Want to do, e.g. NER for product reviews or chat logs? - Incredibly bad results. There's a huge corpus problem in NLP today.
I'm not familiar with computer vision, but in NLP taggers are hovering around human-level performance, and parsers are quickly approaching that level. But if you take a state-of-the-art system and test it on a slightly different corpus (even something as simple as text from the same newspaper, but a year later!) performance drops by a lot.
Why not weight results by the number of training attempts, and publish the number of attempts along with the results? You wouldn't have to disqualify anyone, but over-submitting would embarrass the cheater and hurt their score.
What's more important, winning a competition or improving AI for the entire world? Do the ends justify the means? I Don't think we should be too hard on Baidu, considering they are attempting to improve their algorithm for the interest of humanity (or evil AI that will bring an end to mankind, depending on how you look at it).
> I Don't think we should be too hard on Baidu, considering they are attempting to improve their algorithm for the interest of humanity
They are not improving the algorithm for the interest of humanity. They submitted and obtained test results much more frequently than allowed. With that information, they can tune parameters to more closely fit the test data.
Basically it is like scoring higher on an exam where some of the questions have been leaked. It does not suggest you now have a better understanding of the subject.
You misunderstand. They're using repeated attempts on the test set to improve their network. That is equivalent to training on the test set and that is unequivocably cheating.
Their actions were motivated to win the competition, not improve AI in general. There are many image sets the Baidu team could have used to test on. They used the competition's image set an unfair number of times to tweak their computation to score better on those specific images (not to improve it overall).
This competition (the "Large Scale Visual Recognition Challenge" aka "ImageNet") isn't just some random competition. This is the competition that gave rise to the recent explosion in interest in Neural Networks.
In recent years Google, Microsoft and Baidu have been one-upping each other to the point now where they are getting very close to better-than-human performance (ie, humans disagree with other humans more often than their systems disagree with the average human rating)[0].
Andrew Ng went to Baidu to start their team. I don't believe he is still involved in this challenge.
Baidu has been getting close and close to Google's performance. Every year so far Google has topped it at the right time, but Baidu has later passed the Google benchmark that year.
There were reasons to think that this could be the year they finally beat them. [1] is a story from January about the system that Baidu had built then.
Now this.
[0] http://karpathy.github.io/2014/09/02/what-i-learned-from-com...
[1] https://gigaom.com/2015/01/14/baidu-has-built-a-supercompute...