Friday
May302008
Is "Scaling Engineer" a new job title?
Friday, May 30, 2008 at 7:59AM
Justin.tv is looking to hire a Scaling Engineer to help scale their video cluster, IRC server, web app, monitoring and search services. I've never seen this job title before. A quick search that showed only a few previous instances of it being used. Has anyone else seen Scaling Engineer as a job title before?
It's a great idea. Scaling is certainly a worthy specialty of it's own. Why there's a difficult lingo, obscure tools, endlessly subtle concepts, a massive body of knowledge to master, and many competing religious factions. All a good start. Next I see a chain of Scalability Universities. Maybe use all those Starbucks that are closing down. Contact me for franchise opportunities :-)
It's a great idea. Scaling is certainly a worthy specialty of it's own. Why there's a difficult lingo, obscure tools, endlessly subtle concepts, a massive body of knowledge to master, and many competing religious factions. All a good start. Next I see a chain of Scalability Universities. Maybe use all those Starbucks that are closing down. Contact me for franchise opportunities :-)
Reader Comments (11)
We have grown a generation of software engineers and programmers that consider the low end open source database that one prototypes on as candidates for transitioning to production scale. The hard lessons learned in the Mainframe and Super Mini days are lost.
Moreover, the scaling solutions offered by the MySql community are worse than death (clusters, replication).
We're regular readers of highscalability over at Justin.tv, so I'm flattered to see our new job posting linked to from here.
We made up "Scalability Engineer" as a job title while brainstorming about how to describe the next person we wanted to hire. I hope it does become an established category, because maybe then it would be easier to find one!
Well, I have an open position at Netflix for a senior software engineer that I titled: Scalable Personalization Systems - since thats what my team works on. http://jobs.netflix.com/DetailFlix.asp?flix2161
Its actually quite hard to find engineers who have experience of large scale web site issues. In our case we have a lot of complex multi-threaded Java code to pick the best movies for each user on each page. My team is re-factoring the code into several middle tier services.
This site is a great resource by the way, thanks...
Adrian
> We made up "Scalability Engineer" as a job title while brainstorming about how to describe the next person we wanted to hire.
It's brilliant. I did a double take when I read it and thought doh, why haven't I seen that before? Then I thought I was just oblivious, but no, it doesn't seem to be a recognized specialty in the market and one gathers in schools either.
> I hope it does become an established category
Do you think there can be a defined set of skills that Scalability Engineers should have or do you think it's too niche specific?
> Well, I have an open position at Netflix for a senior software engineer that I titled: Scalable Personalization Systems - since
So I guess we are seeing it more as a specific set of skills along side more domain specific knowledge.
> several middle tier services.
I can't help myself :-) So are you accessing the service via REST or a binary interface? And is the service architecture not use threads or is more of Actor model? I get questions on how to architect backend services so I'm just curious what you guys have settled on. It's important I get my movies :-)
I've not heard of a "Scalability Engineer", but I've heard of "Performance Engineers" - though I believe with very few exceptions, its not an official position either. Not the same thing, but some overlapping responsibilities and skills.
Anyway..a quest to learn how to build truly scalable architectures is what led me to your site. What I'm curious to find out is how the YouTube, Google, eBay, Amazon, and yes, Twitter (etc) teams build their sites/applications/services layers (API).
How much of it was crash and burn trial and error? How much was picked up in school? How much was just evolved over time? Has the underlying design changed significantly as the services utilization grew? How much of the scalability is baked into the application design? How about the deployment topology? Hardware? Etc..
This is a great interview question to ask some of the developers/architects at these companies the next time you sit down with one!
Thanks!
Z
No, but "Software Architect" is an overused term that should only be applied to people who can actually architect a system to meet BOTH the functional AND non-functional requirements, be flexible enough to meet future requirements, and be scalable enough to grow to meet future growth needs. Unfortunately, "Software Architect" is coming to be interchangeable with "Technical Lead" or "Senior Developer".
The web tier and the middle tier are both Tomcat based, for strictly internal service calls a client jar file hides the interface so we can do what we like for the calls, and have used several different patterns, some stateless and some session based, mostly via load balancers. Also I'm thinking of looking into Hadoop as a more batch oriented computation engine.
The code my team works on is aimed at generating lists of movies you will want to watch, individually picking from 100,000 movies for 8 million different customers at fairly high transaction rates.
In your opinion, what are those lessons?
My experience is that we're doing what we can in a way that best meets business needs, balancing cost and engineering effort. Oracle is huge, expensive, and for the scenario I was in (millions of tiny write transactions) a remarkable pain to scale. Postgres and MySQL are free, and it's not like anyone *wanted* to do the skanky things you have to do to make them scale--data duplication, lack of referential integrity, the list goes on. But it's turned out to be the best choice for a wide variety of big-data problems.
How much of what applied to a 20MB storage device, the size of a refrigerator and costing more than a house, now applies to 1TB disks that I can put in my back pocket? What lessons aren't being learned? I'd love to see this discussion happen, because if we're missing good stuff from history, we should learn it.
That title goes more towards people who are related to IT sector such as software programmers, Computer engineers, Software analysts and so on.
-----
http://underwaterseaplants.awardspace.com">sea plants
http://underwaterseaplants.awardspace.com/seagrapes.htm">Sea grapes...http://underwaterseaplants.awardspace.com/plantroots.htm">plant roots
The code my team works on is aimed at generating lists of movies you will want to watch, individually picking from 100,000 movies for 8 million different customers at fairly high transaction rates.