It's a pity you didn't ask us to spend the time with you.

If only .................................................................................................................................................................................... .

If only you had asked us to spend the time with you.