Who does this job for a living? What should I study?
I want to study the assembler language, specifically for windows operating systems. I already reverse engineer malware for fun and debug things. I love programming in assembler and taking things apart.
What should I study so that I can program in low level languages and reverse engineer other peoples software, maybe for antivirus companies?
I'd be interested in any version of assembler, 8, 16, 32, 64 bit, even microcontrollers. Who does this? What job?
Asked By: JoeSmith - 12/11/2012
You are in the best place and the best time for this kind of interest. You can get cheap microcontroller systems (for example, TI's LaunchPad will cost you $4.30 and includes two 16-bit micros and a board that includes easy ability to add daughter cards (many of which you can already buy for similar amounts of money.) There are MANY other excellent options from other companies, as well, if you step up to about $10 or so. And MANY MANY more if you move up towards $30. You have access to FPGA boards where you can learn VHDL or verilog and design and test your own CPU, even (I've designed my own, in fact.) The cost for these start at around $50 for a good system, though you can certainly pay more. But these are incredible values and if you think you like assembly code, imagine how you'd like to be able to write your own cpu? It's not hard. One book you might get (it is for neophyte beginners) is "Bebop BYTES Back: An Unconventional Guide to Computers," by Clive Maxfield. You can learn most of what is necessary there. Then get Douglas Smith's older "HDL Chip Design Using VHDL or Verilog" which gives a fantastically easy to follow guide for both VHDL and Verilog in what is called a "parallel format" showing how something is done in both languages and working you through from why these languages exist and what they do all the way through multipliers, ALUs, and dividers. Etc.
You have dozens of tools for writing for the IBM PC in assembly, from Microsoft's freely available Visual Studio using C/C++ and its built-in in-line assembler, as well as it's separate ML assembler that comes installed (at least in older versions, such as the 2005 edition -- I haven't checked the newer one.) You can write for DOS box environments, as well. Most development toolchains, even the commercial (for real money) ones, include a free assembler tool in the package -- commonly also without any code or data size limitations.
Assembler coding jobs break down into:
1) Cases where only very skilled designer/coders can properly analyze a complex hardware environment and use it to its fullest potential and where applying that skillset (expensive) provides a meaningful commercial advantage that translates to additional profits. The x86 chips are quite complex. There are multiple cores, within each core the registration station supports multiple functional units, there is branch prediction, L1, L2, and even L3 caches at varying clock and physical distances from the core, there is a frontside bus with each attached chip having a limited number of cached transactions in progress, the chipsets include inbound and outbound queues with their own features and limits, there are external buses and peripherals... well, you get the idea. And this doesn't include the extremely fancy capabilities of modern GPUs found on graphics cards. If you are writing an application (such as, for example, a SPICE program for circuit analysis), you may wish to hand-code some aspects to gain best access. A compiler can't know the application or all of these hardware details nearly as well as a human may.
2) Commercial instrumentation, which although often written largely in C, will almost ALWAYS require SOME assembly code, as well. Or at the very least, some detailed knowledge of compiler code generation. And assembly experience is vital here. An example here is that I recently finished a product using the SiLabs C8051F061 which is the ONLY cpu out there that includes a 16-bit SAR ADC that runs at 1MHz. Unfortunately, the cpu core is an 8051 which has ONE external memory pointer register. Processing this fast data in a timely way required the use of assembly coding. No compiler was able to achieve the data transfer and mathematical optimizations that I could perform by hand in assembly code. Constraints like this force the situation.
3) Extremely low power/low cost/small size, but higher volume, commercial devices where doing things in assembly mean less power, cheaper cpu, etc.
There are other cases, also. But that gives a flavor.
Just DO IT!!! Get going. And learn some electronics along the way, too. And develop your math skills, too. Particularly those related to discrete (integer) math.
Answered By: Jonathan - 12/12/2012