I've not got any hard data to support this, but I would say for small quantities of toast, yes a toaster is generally more efficient. The timings below are based on my experience with grill and toaster.
Your results would obviously depend on the individual products involved. If you have a rubbish toaster and a brilliant grill the timings and energy usage would change.

The grill element in most ovens is large and typically takes a minute or two to reach full temperature, you then need to grill each side for around 30-60 seconds, so the total time the grill is around 2-3 minutes.

By contrast a toaster's elements are fine filaments that reach operating temperature almost immediately. The elements also heat both sides of the toast concurrently so the toaster operates for 30-60 seconds max.

A bit of light googling indicates modern toasters use around 1.1 kW (see bottom of table), so in a minute would use around 66 kJ. Whereas grills seem to be around 1-1.8 kW, so in 2 minutes would use 120-216 kJ, or in 3 minutes 180-324 kJ.

Of course if your grill heats up a lot quicker the advantage of a toaster is diminished, and once up to operating temperature it is probably more efficient, so if you were doing toasting continuously for some time you may find the grill wins in the long run, maybe.