Notes

The reason why we need to add the "jmp esp" step in this case is ASLR (address space layout randomization).

ASLR is a security mitigation designed to make it more difficult to find the payload (and data in general) in the memory. It randomizes the start address of a few sections of the program on each run, such as stack and heap. The stack start address will change each time the program is run and thus so will the payload address. That way we can't tell EIP where to point to, since we don't know the exact address.

However, in that case the payload was (not by luck, but) lucky located in the address ESP was pointing to. So we add the "jmp esp" step (which is the same to "$eip = $esp"), making ESP point to the NOP sled. This technique is used when we can't tell where the payload is, but a register is conveniently pointing to it.

The interesting part is there are sections of the program which do not have its memory address randomized. The TEXT section, for instance, has the same address on every run, meaning that we can use it to find our jmp esp gadget and make EIP point to it. In the post's case, a few libs also did not have ASLR so luckly we could use it for gadgets, too

---

Your exploit payload ends up on the stack because you're overflowing a buffer on the stack, and this is how you gain control of the return address as well.
ESP points directly to the start of your payload (after execution of the ret in the function you're attacking) because you put the payload right after the 4 bytes that overwrite the return address on the stack. ret pops 4 (or 8) bytes into EIP, leaving ESP pointing to the payload that directly follows.
But you don't know what value ESP will have at that point, because of stack ASLR and because a different depth of call stack leading up to this point could change the address. So you can't hard-code a correct return address.
But if there are bytes that decode as jmp esp or call esp anywhere at a fixed (non-ASLRed) address in the process's memory, you can hard-code that address as the return address in your exploit. Execution will go there, then to your payload.
This is in fact often the case: Some DLLs don't have ASLR enabled for their code, and the main executable's code may not be ASLRed either.
Code ASLR for all code defeats a jmp esp attack, unless the attacker can cause the target process to leak addresses.
Note that for 64-bit code, you're unlikely to be able to use jmp rsp for string-based buffer overflows, because code addresses will contain some leading 0 bytes.
Thus, jmp esp gives you a much more reliable exploit than repeatedly guessing a return address (with a very large NOP sled).
Repeated guessing will crash the target process every time you're wrong, but a jmp esp can give you a high chance of success on the first try. This will avoid leaving crash logs. It could also defeat an intrusion-detection system that looks for crashing server processes and blocks connections from your IP address, or similar.
Note that the 2-byte instruction you're looking for can appear as part of another instruction when the program executes normally, or as static data (especially read-only data is often in executable pages). So you just need to search for the 2-byte sequence, not for jmp esp in disassembly of the program. Compilers will never use jmp esp, so you won't find one that way.
More generally, any function that ends with a buffer pointer in any register (e.g. from a memcpy or especially strcpy) can allow a ret2reg attack, by looking for a jmp eax instruction.
This can work in 64-bit mode, where addresses have some high zero bytes; if strcpy's trailing zero writes that high address byte for you, the end of your exploit string could be the non-zero address bytes that overwrite the return address on the stack.
In this case, the executable payload would go before the return address, at the spot in the buffer where the function leaves a register pointing. (Typically the beginning of the buffer if there are any useful pointers to the buffer in registers at all).