To catch any monster (except Humanoids, Elites and Bosses, these can't be tamed) you'll need to have in your bag and quick/action slot bar some traps. These vary in their capture chance and some are specially made to capture sea pets, air pets, etc.
Each time you use a trap, be it a success or not, 1 trap will be consumed.
The capture chances rise as the monster HP drops.
If you capture the pet, you have now in your bag a Soul Orb .
To get your pet you must have it Appraised. You'll need a Pet Appraisal book, which can be bought at the pet merchant. Then just double left click the Soul ball and you have your pet ready at the Pet menu. When you appraise a pet its talent level varies. It means that you cannot have a pet that is a supreme star 2 100%. their ranks are Garbage, Clumsy, Regular, Smart, Genius Star 1, Genius Star 2, Genius Star 3, Supreme Star 1, Supreme Star 2, Supreme Star 3 and Ultimate. You can have a small chance to capture a supreme star 1,2,3 but for Ultimate pet is nearly impossible.
Note : 1 Soul Orb occupies 1 bag slot, 10 Appraisal Books (same type) occupy 1 bag slot, 20 trap (same type) occupy 1 bag slot.