Dumb questions. Good sys admin interview questions shouldn't ask about specifics like "what is the command that does x" or anything that can be easily looked up in a manual. Rather they should quiz depth of knowledge of the Linux (if that's what its for) ecosystem.
I had some good questions posed to me during a Google DevOps interview, though I have trouble remembering what specifically they were about, I do remember that they were of a much more "holistic" and general bent. Instead of asking me to provide irrelevant details (i.e what are the switches for ls that list time oldest to newest) that are easily forgotten, they would pose a problem that seemed to have an easy and obvious solution and then make the problem more and more esoteric and difficult to solve.
These were the sysadmin trouble shooting problems and I found them quite interesting because they directly related to my experience and opinion that sysadmin trouble shooting is unfortunately sometimes quite complex and involves less an understanding of command syntax and more of a problem solving mindset.
Another point to take away is that if the interview is about linux, it should test fundamental linux things linux things such as testing the candidate's understanding of linux process management and signals. Perhaps filesystems as well.
My take away point is not that I think the FB interview was overly easy, but rather that it asks useless questions. This interview isn't looking for the sysadmin mindset at all.
This wasn't a FB interview - this was a phone screen before you got to the interview. Every linux sysadmin under the sun can answer 90% of those questions from "muscle memory." The purpose of this phone screen was to ensure that the incredibly expensive time of the interviewers was only used to meet potential candidates.
There are certain trivia or data that the entire community of linux sysadmins know, just by having worked with the operating system - and, if you miss more than about 20% of those questions - one can say with pretty much certainty that you are not a linux sysadmin.
I thought the questions were roughly the ones I would give to a technical recruiter to screen.
(PS: ls -lart - muscle memory, I type it 100+ times a day)
6. 3 Timestamps nix keeps on every file: created, modified, last access*
ctime is not when the file was created, it's when the inode (the file metadata) was last changed, as compared to the mtime which is when the file's data was last changed. Note that changing the file's data also updates the ctime.
Just to clarify things a bit more, I'll quote from the stat(2) man page [1]:
time_t st_atime; /* time of last access */
time_t st_mtime; /* time of last modification */
time_t st_ctime; /* time of last status change */
The field st_atime is changed by file accesses, for example, by
execve(2), mknod(2), pipe(2), utime(2) and read(2) (of more than zero
bytes). Other routines, like mmap(2), may or may not update
st_atime.
The field st_mtime is changed by file modifications, for example, by
mknod(2), truncate(2), utime(2) and write(2) (of more than zero
bytes). Moreover, st_mtime of a directory is changed by the creation
or deletion of files in that directory. The st_mtime field is not
changed for changes in owner, group, hard link count, or mode.
The field st_ctime is changed by writing or by setting inode
information (i.e., owner, group, link count, mode, etc.).
lies, lies, damn lies. you've never witnessed faster googling than when a system is down. googling is usually much faster than trying to recall the proper switches for all utilities. I find Google to be far better sometimes than even man pages. I've only been a sysadmin for 5 or so years so maybe my views will change
To me, it seems the questions were pretty basic and geared towards judging if someone has had experience actually administering linux servers. How many people (who actually competently deal with linux servers) need to use google to know what the output of top is? or how to kill a process? If you don't already know what MTU is about, how would you even consider an incorrect MTU as being an issue when you're debugging an issue in production?
Likewise, I wouldn't hire a Senior Javascript developer who can't describe in her own words what a closure is and what it's useful for, or talk about execution context (what 'this' refers to) and how to control that. You just can't have seriously written any javascript and not have encountered those things enough that they've been burned into your mind (though maybe an intern or a Junior developer that'd be okay)
it's probably true that sysadmins with tons of experience can do everything by heart ... but that is a separate issue of whether someone can be functional and operational and still have to check google once in a while.
Well, I have basically all of these correct... been doing this for very many years now so I'd hope so ;). I do hate these kind of interview questions (even first round interviews) though. They're not really useful at all in practice. It's like an old manager of mine said and I still think it's a good statement: There's knowledge you keep in your head and knowledge you keep on the web.
Essentially, more often than not if I'm doing something shell related I'll need to turn to man pages or doc files for a specific method/flag to use on a command unless it is the 'muscle memory' type commands like ls related flags, etc. There's only so much that is important to have memorized, and only so much you can memorize. The rest, look to the docs my friend!
I hope that these questions were from the author's memory and not worded exactly as typed. Because the questions used for first round filtering of interview candidates should be clear cut and not open to interpretation.
Namely, question 14 is odd,
What Signal does a child send to a parent when it is finished.
The child doesn't send signals in this event, the kernel sends signals to the parent when the child's state changes, and one of those states is the child exiting. Although, the default action of SIGCHLD is ignore, and can a signal that is ignored be considered to be sent at all?
a) not all signals can be caught, blocked or ignored c.f. SIGKILL, SIGSTOP
b) in acutality when it comes to the kernel code if a process/thread/task has a signal ignored (the blocked semantics are slightly different) it is not actually delivered x.ref. prepare_signal() in kernel/signal.c in Linux.
I'm sure if you just said SIGCHLD you demonstrated the knowledge they were looking for. If you gave your extended answer, they would probably be additionally impressed.
It really depends on who is administering the first pass questionnaire and how they have been prepared. When I was doing hiring, a dedicated recruiting team was tasked with doing this first round, which we called "prescreen questions". I spent a lot of time briefing the people talking to candidates for my team on how to interpret the answers, and I encouraged them to keep detailed notes so we could audit what candidates were saying and needed to adjust something. I personally reviewed every prescreen result to make sure we didn't pass on someone who could provide more than the straight forward answer. However, we found that prescreens were most effective when the answers were straight forward and there was only one way to interpret the question (that is, for prescreens, non-essay answers are preferred).
I got the feeling that the uptime question was also inviting an extended answer, because the load figures (last minute, last 5 minutes, last 15 minutes) in uptime seem obvious, but are actually computed as exponentially-filtered averages (http://stackoverflow.com/questions/5892104/how-were-the-weig...). In an interview situation, it would show above-average depth-of-knowledge to mention this.
Yeah, honestly even asking for clarification which shows that you really understand the concepts at play is probably as good or even better than providing the intended answer.
fsck checks the consistency of the file system and the kernel? Asking those two together seems really weird. If that's how it was presented the best guess I could make is sync (forces changed blocks to disk) and fsck.
My first guess was running package manager checksum verify, like "rpm -V kernel" , for rpm based distro. But i'm not sure if it was the question was aiming for.
The question, as written, has no answer. fsck only checks the filesystem. There is no way to check the consistency of the running kernel but a kernel panic is a telltale sign of INconsistency :)
You can check the consistency of the kernel "file" and compare a MD5 / SHA-1 of known good kernel. Assumption here that the kernel is loaded into healthy hardware. Depends on the spirit of the question. I do agree, running kernel looking at itself... chicken::egg.
I was very confused on this question as well. Seeing both kernel and filesystem there together with the word consistent made me think of a sort of general health. So my answer would have been `iostat`. That would show me the running kernel, cpu utilization, and disk utilization.
I presume 'not much' or 'you get an error', but I don't have an instance I'm prepared to sacrifice right now, just in case something hilariously destructive happens.
fsck doesn't run on a directory, it runs on a block device. You can't run it on /proc or /sys because those aren't backed by a block device. /proc only looks like it has a file system because the kernel "pulls" the "files" out of thin air when you try to access them.
There's more to come. I've interviewed with Facebook for a similar position.
After a phone screen, they setup another call that was probably more intense. They asked me to be near a computer with a net connection so I assumed I'd be writing code or troubleshooting something.
I ended up turning it down because I received an offer from another .com that seemed more promising but I believe if I passed the next call, I would have been flown in.
The last question isn't answerable; it's kind of like the question "have you stopped beating your wife?", the question presupposes a fallacy.
Actually a child almost never sends a signal to the parent process on termination; the signal is sent by the kernel from the kernel context.
A child could manually send a SGIC(H)LD by using signal(2), but then any wait(2) would block; this is why you should always use a non-blocking flag to waitpid(2) or its ilk in case some idiot program sends your program a SIGC(H)LD.
That's a good first level interview I guess. I'm a .NET programmer and I knew all but one (but I am also a full stack dev, so it's probably normal that I would know these things.)
Good illustration of why in some
ways it's actually easier to be
a solo founder of a startup than
an employee of one!
I know next to nothing about
Linux -- it's yet another multi-user,
demand paged, virtual memory operating system
with an 'embedded' operating system
and a hierarchical file system, written
mostly in C. Right? That's about all
or a little more than I know about it.
There have been a lot of those, back
to Multics. So, Linux is yet another one.
Okay.
But I'm developing on Windows. Even on
Windows I don't know that much detail. For
such questions on Windows, I wouldn't
know a good source (wish I did), and
when I need answers to such questions have
to Google/Bing a lot, hit Stack Overflow,
etc., one silly little question at a time.
Bummer, but the frequency with which
I have to do
such scavenger hunting in muddy swamps
has been declining.
No doubt when I bring up Windows Server
I'll have a lot more such detailed mud
wrestling to do; similarly for serious
work with SQL Server. Bummer -- again
I'll be screaming at the lack of good
documentation for such questions.
Still, so far my software development is
going ahead well without such detailed
knowledge. Actually I've long suspected
that once my company gets serious with
number of users arriving per hour and revenue
getting up to the useful level, I'll get
answers to such questions by calling for
paid, expert support at Microsoft. It's
their software, not mine! And while most
such questions are simple conceptually,
digging out the answers often is not.
Indeed, one reason I selected Windows instead
of Linux is the hope, maybe false, that
one way and another, say, from Microsoft
or Stack Overflow, books, Microsoft's
MSDN Web site and the similar site for SQL Server,
I'll have better means of answers to such
questions. Similarly, when I need a high
end LAN switch, I'll try to pick Cisco
heavily due to a hope for good sources,
possibly from Cisco, for answers to such
questions. LAN switching is their darned
business, not mine; they can do their
business, and I'll be a paying customer,
and I'll do my business.
For hiring, there's a recent remark from
Richard Branson that he hires for personality
and not skills since skills can be taught,
he believes quickly, while, implicitly, it
appeared he was saying that personality could
not be taught quickly. Actually, for such
detailed questions for Linux/Windows Server,
he is significantly correct. So, I'm hoping
to hire bright people with good personalities
who are at least okay computer users (even if
they have never written a line of code in their
lives) but are good at typing, reading, writing,
speaking, 'critical thinking', getting some okay
understanding from some horribly badly written
material (i.e., nearly all computer industry
documentation), and analyzing ambiguous situations.
Right: Sounds like I might want to hire some
good humanities majors! Nerd me!
Then have such
employees start with simple things, say, plugging
together a server, working up a simple, quick
and dirty parts inventory system (maybe just a
flat file with some keywords maintained with just
a text editor maybe with a few macros -- maybe
even a, yuck, spreadsheet), pulling CAT 5 cables
and using a crimping tool for the ends, installing
Windows Server, calling Microsoft for technical
details, documenting what they learned about how
to install, configure, and administer Windows Server,
etc.
Then, teach them programming a few lines at
a time, e.g., Visual Basic .NET (apparently about
the same as C# for getting to the CLR and .NET
Framework but, to my eyes, easier to read on
the page than C# with its syntax from C
and, thus, deliberately idiosyncratic).
For basic computer
and network architecture, I'll give them a 30 minute
overview lecture they can capture on video and use
for other new hires. The hope is that such people
will be able to grow in 'skills' as fast as the
company grows. I.e., I'll show them what I
already know, and as we need to learn new things
it's their job to dig them out and document
them for others. Did I mention I want them to
know how to write (and not learned in the
computer industry)?
Then for Facebook hiring, I would ask: Whatever
the heck happened to the last person in that position?
And, why the heck does Facebook have to go recruiting
for such a person? You mean, Facebook doesn't have
a backup for such a person? Or, Facebook has
400 billion executing instances of Linux, may be
compiling some unique versions of Linux, and, still,
has only one Linux system administrator? Not good.
Broadly, if Facebook needs such 'skills',
then they should grow the capabilities internally. Or,
I can do my startup without knowing all that stuff;
if Facebook would want me to know that stuff, then
they should pay me to learn it.
This stuff about having to walk in with a lot of
detailed knowledge, learned on my dime and time,
just as needed by Facebook for just an 'employee'
position in a big company, that is, an organization
that in a few years might just decide to layoff
20,000 employees, is getting to be a bad, old story.
A startup, even for grass mowing, literally, looks
more promising.
"This stuff about having to walk in with a lot of detailed knowledge, learned on my dime and time, just as needed by Facebook for just an 'employee' position in a big company, that is, an organization that in a few years might just decide to layoff 20,000 employees, is getting to be a bad, old story. A startup, even for grass mowing, literally, looks more promising."
Spot on! Another way of looking at it is that tech companies don't want to spend $ training their employees (including a slightly longer ramp up time for an employee to begin contributing). If they did, the hiring process would be: you had a 4.0 in college. This shows you are good at learning new things. Lets check for personality fit and some other things that a college transcript does not capture. Do a one hour interview. Hire or No Hire based on this.
As a startup founder, you should be happy about this state of affairs. The attitude of large companies makes it easier for you to hire :)
I'm getting happy: I'm 70 miles north of
Wall Street and that means really close to
Vassar and not far from Princeton, Yale,
Harvard, and more. So find some bright,
humanities major coeds who otherwise
would have to start in retail or go to
law school or for an MBA and get them
started.
So, they have great SAT
scores, know how to study and learn,
know what high quality work is,
are good at working with people,
are really good at reading comprehension,
can knock out an A+/A+ term paper between
10 PM and 7 AM with just a quart of
coffee, can give a good presentation,
etc.
So, teach them about Volts, Amps,
Watts, Ohm's law, DC, AC, resistors,
capacitors, inductors, basic electronic
signals, the basic conceptual principles
of how a processor core works, a little
on caching and main memory,
data representation 101,
how a
disk drive works, the basics of
an NIC, Ethernet, a LAN switch,
and TCP/IP, and IP router,
a little
on memory protection, an 'embedded'
operating system, virtual memory,
virtual machines, a file system
with locking and security,
console sessions and command lines,
text files and a good text editor,
the Internet from DNS, SMTP, POP3,
to base 64, MIME, HTTP, HTML, and
CSS, the basics of programming languages,
e.g., Visual Basic .NET and its
memory management, classes and instances,
the .NET Framework, relational database,
SQL, SQL Server, how to use Microsoft's
MSDN, about Stack Overflow, HN, and
how to Google/Bing to get answers to
questions, how to use our technical
support account at Microsoft and
Cisco, all in about one afternoon,
with some exercises, and then have
a graduation party.
Then get to
work on some basic stuff, say,
plugging together another 20 servers, repairing
a server with a busted hard disk,
etc.
Clean, indoor work, no heavy lifting,
and better than grass mowing or retailing.
Easily, just the way Facebook does it except
maybe better, with one consideration -- they
will have to work their way into such work.
Let's see: For such scaling, it has long been
the case that a first-cut quantitative approach
has been theoretical and
empirical network queuing analysis. So, how
many such Facebook scaling architects are
able to do such analysis, either theoretically
or empirically? I'll bet, not many. Net,
I doubt that the Facebook scaling architects
have a very good approach to the problem.
What are they doing? They have a good
grasp of the basic issues of the components,
that is, the server hard/software, NICs,
LAN switches, TCP/IP stack details,
Linux thread switching performance, etc.
Then they concentrate on the bottlenecks,
solve those empirically one at a time,
and otherwise just f'get about the rest.
Right? I mean, what the heck else are
they going to do, dig into the Poisson
process hypotheses and the renewal theorem?
I doubt it.
So, sure, a humanities major won't come
knowing all about the time/date stamps
on Linux files or about file locking.
That may be stuff they have to learn.
So, for more, need people who can learn
stuff, find materials for the learning,
work with people for the learning,
use some critical thinking about the
material and how to use it.
Then for implementing the results of the
analysis, have to involve other people,
in software, hardware, cable pulling,
HVAC, electrical power, ..., lots of
other people. So, now have to work
with people -- with technical papers,
planning documents, foil decks, presentations,
meetings small to large, one on one discussions,
etc. So, have work with egos, power
struggles, turf battles, office romances,
human nature, etc., i.e., the stuff
nerds ignored (I'm an expert nerd)
in high school, college, and beyond.
The Heine-Borel theorem, fine;
human nature, f'get about it.
So, after I saw Branson's remark, I happened
to visit the Web site of my college
and, there, saw some of the undergraduate
students they were proud of. I doubt
that any of those students knew anything
about Linux internals, but one of the
students in just her extracurricular
activities has organized nearly everything
on campus short of the birds in the trees.
Give her a couple of years, starting as
I outlined, yes with her able to learn
about time stamps on files, etc.,
and when it's time to scale
the Web site, along with moving it to
a colo or our own building, I'd bet she'd
knock the socks off nearly all nerds,
including me. And, if really need some
good work, say, in queuing, bet she'd
blow away the self-taught Linux hackers
who started with C++ in middle school;
for one, she'd be smart enough and good
enough to pick a good consultant in,
say, computer and network performance
at a computer science department and
make good use of some consulting.
Again, I bet that so far the Linux
performance details remain nearly all
fairly simple conceptually. Yes, good
thread switching likely remains a
challenge, but that's quite specialized
work for which draw the larger lessons
from specialists.
I'll provide an example: I was working
in artificial intelligence (AI). We developed
an AI language and did joint work with
MET LIFE and GM and gave a paper at
an AAAI IAAI conference at Stanford.
My wife wanted to learn some of this
material. So I gave her a lecture of
about 15 minutes on how to connect to
the server and use it and then another
15 minutes on our AI language. She then
took two weeks and wrote a nice, first
AI program. Right: BAM, with next to
no background in programming. I looked
at her program and said, "It's a program.
It works. It does what you wanted.
There's nothing fundamentally wrong with
it. But 'AI' has a 'belief', a 'theme',
maybe just a 'hope', and you didn't really
illustrate that ...." So, building on
some of what she'd studied as an
undergraduate, I explained the 'dreams'
of the MIT 'school' of AI. She smiled.
Two weeks later she had a really nice
first AI program. "Nice"? Our project
had as programmers some of the brightest
computer science students in the country,
who were taking computing and AI very
seriously, and they had written first,
sample programs. Her program was several
notches better, the best first AI program
our group ever saw. She did that work
with all of about 45 minutes of instruction
from me and four weeks of time.
Of course, she was 'cheating': She
was brilliant, not just a little but
the real thing. Yes, high school
Valedictorian and in college PBK,
'Summa Cum Laude', Woodrow Wilson,
NSF, etc. She got her Ph.D. in
essentially mathematical sociology
from one of the world's best
research universities and where two
of her professors became President
of the American Sociological Association.
Did I mention she was brilliant?
This detailed computer stuff can be
picked up, fairly quickly, especially
if it is well presented. Good students
can be darned good at picking up such
stuff, including from some horribly
badly written documentation, e.g., as
in the computer industry.
I'm guessing that maybe Branson had
a point. That is, it may be better to
get good smarts, ability to learn,
especially from poorly organized materials,
really good at the '3Rs',
and really good working with groups of
people and just pick up over, say,
two years, what need to know about
a growing server farm based on Windows
Server and SQL Server and my software
(a particular, just one, growing
server farm, not all possible growing
server farms)
instead of hiring for Windows Server,
etc. right away. Maybe.
> for one, she'd be smart enough and good enough to pick a good consultant in, say, computer and network performance at a computer science department and make good use of some consulting
And this is what Facebook and other companies do; they hire those consultants.
On the other hand, these are all pretty superficial and elementary questions. I have never been a Linux admin and I still knew about 2/3 of them, or at least could have fumbled through them (e.g. 3 timestamps).
Debug a possible situation or even something that happened in the past. Take for example these question from ServerFault: "DNAT:ed packets disappearing"* [1] and "Openvpn, forwarding slow since reboot" [2].
*Sorry for the self promotion, but I don't have any better examples right now.
ehh..What is the worst that can happen? People study these questions and by the act of studying these questions, they acquire more information on how things work? If the interviewer is good and is not asking for factoids, it is very easy to figure out how much a person knows about something.
The questions are very simple. Anyone who depends on a blog post to know the answers will fail a slightly different set of questions, not to mention the next interview.
While I don't really have a problem with being "rude" to a corporation, especially some evil behemoth like Facebook, it's worth noting that sharing these questions probably violates an NDA that was signed/agreed to before the interview.
I had some good questions posed to me during a Google DevOps interview, though I have trouble remembering what specifically they were about, I do remember that they were of a much more "holistic" and general bent. Instead of asking me to provide irrelevant details (i.e what are the switches for ls that list time oldest to newest) that are easily forgotten, they would pose a problem that seemed to have an easy and obvious solution and then make the problem more and more esoteric and difficult to solve.
These were the sysadmin trouble shooting problems and I found them quite interesting because they directly related to my experience and opinion that sysadmin trouble shooting is unfortunately sometimes quite complex and involves less an understanding of command syntax and more of a problem solving mindset.
Another point to take away is that if the interview is about linux, it should test fundamental linux things linux things such as testing the candidate's understanding of linux process management and signals. Perhaps filesystems as well.
My take away point is not that I think the FB interview was overly easy, but rather that it asks useless questions. This interview isn't looking for the sysadmin mindset at all.