Your brother is confusing forgiveness with legal pardon. To pardon is to cancel a debt that is rightly owed. To forgive, in contrast, is to try as best you can to offer goodness toward your brother. Both are merciful, but they are not the same. You can forgive and not offer legal pardon (cancel the debt). You can forgive (offer goodness) and at the same time present him with the I.O.U. And if you forgive him first, you are likely to present that slip to him with graciousness and gentleness rather than with anger.