This has been a subject of numerous debates and character analysis of Rama.
This one act does seem to cast a stain on otherwise spotless and virtuous character of Rama. Vaali himself fumes and rants at the apparent injustice in being killed while be engaged in a duel with another.
There's extensive analysis of the text and it's not very clear if Rama indeed killed Vaali by hiding. There is some small room for doubt that all might have not been fair in the duel between Rama and Vaali but nothing is conclusive.
Here's the analysis of all the events leading up to the fight with Sugriva, slaying of Vaali, conversation with Rama and finally Vaali's acceptance and repentance of his mistake.
-----
First challenge
Rama's intention on being unable to distinguish between them is not clear here. It might be that he needed identification to shoot down Vaali while hiding. It should not be assumed that they were fighting next to each other. Valmiki says that they threw boulders and trees at each other and fought all over the forest. But still, why would Rama need the identification while he can challenge Vaali directly?
Since this attempt was unsuccessful it's difficult to gauge Rama's intention and we need to look at the second challenge by Sugriva in order to assess Rama's intentions
They all on quickly going to Kishkindha, the city of Vali, hedged themselves in the thickets of forest trees and waited. [4-12-14]
For which reason Raghava could not differentiate between Sugreeva or Vali, for the same reason did not make up his mind to release his arrow, otherwise it would have been a deadening arrow, as confusion is the only reason for Rama. [4-12-20]
Meanwhile Sugreeva is unable to spot out his benefactor Raghava as Vali whacked him down, and thereby he immediately fled to Mt. Rishyamuka. [4-12-21]
"Thereby, oh, great monkey, I have not released this great speeded and enemy annihilating arrow of mine as I was confused in the similarity of your personalities. [4-12-32]
"I was ambiguous by your similarities, as such I have not released this ghastly and life-taking arrow, otherwise the basic object of ours will be ruined, isn't it. [4-12-33]
-----
Tara's advice to Vaali before he leaves for the second challenge by Sugriva.
"Those that are born in Ikshvaku dynasty, sons of the king of Ayodhya, valiant and unconquerable ones in war, called Rama and Lakshmana are there in this country. These two unassailable ones have chanced there at Sugreeva's place to fulfil the longing desire of Sugreeva. [4-15-17, 18a]
"He is an acclaimed one for his warfare, and like the fire flaring up at the end of era he shatters the strength of enemies, and he is Rama your brother's helpmate, they say so. [4-15-18b, 19a]
And he is said to the habitable tree for the polite, the ultimate course for the woebegone, a hospice for the agonised, and for grace Rama is the only abode. [4-15-19b, 20a]
By that reason, with that unconquerable on in conflicts, an imponderable one and a great-soul Rama, your enmity is inapposite. [4-15-21b, 22a]
Tara is cleary informing Vaali that Rama is unconquerable in conflicts, inspite of she being aware that Vali has that necklace which is supposed to make him invincible. Tara’s statements clearly indicate that “no necklace” and “nobody” can stop Rama in a war. So, the theory that Rama was hiding behind the tree otherwise Vali would have taken away half of Rama’s energy is untenable
-----
Vaali is aware that Sugriva has Rama's support but thinks that he may not cause him harm as he thinks he hasn't done any mistake
"Also, you need not bother about Raghava's doing some harm to me, as I wonder how a diligent man and knower of virtue can commit sin. [4-16-5]
-----
Similar to the first challenge, the following reference gives doubt if Rama was in hiding. But it's inconclusive as it's not said if they were in hiding or if they came out.
He that Sugreeva whose physique became effulgent by that creeper-garland is again emboldened by Rama's words, and he again advanced to Kishkindha with Rama, and reached it again. [4-12-42]
They all (including Sugriva, Hanuman, Nala and other vanaras along with Rama and Lakshmana) on quickly going to Kishkindha, the city of Vali, hedged themselves in the thickets of forest trees and waited. [4-14-1]
-----
Following verses in Kishkinda Kanda clearly shows that Rama didn't hit him from behind as his arrow pierces Vaali in his chest.
Rama raises his bow only after Sugriva was drained of all his energy and looks forlorn and doesn't know what to do. Not when their fight was in progress.
When Rama blasted his bowstring it made a tremendous noise and there was no way Vaali would not have been aware that someone is attacking him.
Raghava has then seen the lord of monkeys Sugreeva who is repeatedly eyeing all sides for help and who is even deteriorating in his enterprise. [4-16-31]
On seeing the lord of monkeys Sugreeva in a forlornness, then the refulgent and fearless Rama scanned for an arrow aiming to eliminate Vali. [4-16-32]
Then on tautening a venomous serpent like arrow in the bow, Rama started to draw out bowstring, whereby that bow attained a similitude with the Time-disc of the Terminator. [4-16-33]
At the blast of bowstring the lordly birds and animals are panicked, like those that will be startled by the approach of ear ending, and they all fled. [4-16-34]
The arrow released by Raghava that has the boom of thunderbolt's thunderclap and the flashes of a lightning fell on the chest of Vali. [4-16-35]
Hit by the fleetness of that arrow then that highly magnificent and intrepid lord of monkeys Vali fell onto the plane of earth. [4-16-36]
Like the flag that will be raised in honour of Indra during the month of ashvin on a full-moon day, but thrown onto earth along with its flagstaff after the festival, Vali with depleted energy and dissipated vitality (Sattvo vichetana) slowly fell onto ground, and with tears blocking throat he moaned piteously. [4-16-37]
That best one among men Rama released a blazing and enemy subjugating arrow which in simile is like an Epoch-ender at the end of era, and that best arrow decorated in gold and silver looked like the glance from the Third-Eye of Rudra, emitting fire with smoke. [4-16-38]
-----
Vaali, after being hit by Rama's arrow asked the following questions relating to his morality, propriety etc.,
"When you have not appeared before me when I confronted Sugreeva my concept was, 'it will be inapt of Rama to hurt me while I am combating with another combatant, besides, when I will be unvigilant in that fight...' [4-17-21]
1. By killing one who is facing away, what worth is achieved by you?2. You have not punished the wrongdoer
3. Killed one who is combating with another and an unvigilant one
4. In your country or city I did no misdeed
5. Non-guilty being is hurt
6. Fruits, tuber eating being is killed
7. No dispute of land, gold or silver
8. Your primary aspiration is to kill without probing into good or bad 9. How do you face criticism by scholars?
10. Unnecessary killers are hell-goers
11. Un-wearable is my skin uneatable is my flesh
12. Five kinds of five-nailed animals are usable by humans
13 I would have brought back Mythili in one day
-----
In response to Vali's claim that Rama has no jurisdiction to enforce laws there, since they're not in Ayodhya, Rama says that the Ikshkvaku dynasty has authority over the entire earth
"This earth with its mountains, woods, and forests, even along with the authority to condone or condemn the animals, birds, and humans on it belongs to Ikshvaku-s. [4-18-6]
-----
Rama says that in punishing evildoers, he is carrying out his brother Bharata's decree to enforce dharma (righteousness):
"He who is virtue-souled, truth-abiding, plain-speaking, and the knower of the import of probity, pleasures, and prosperity, and the one who is concerned in controlling or condoning his subjects, that Bharata is the ruler of earth. [4-18-7]
"In whom both scrupulousness and benignancy are there, truthfulness is firmed up, and valour as evinced by scriptures is evident, and he who is the knower of time and place is the king, namely Bharata. [4-18-8]
"Holding his virtuous decree desiring to keep up the continuum of righteousness, we and some other kings are wayfaring this earth in its entirety. [4-18-9]
"While that Bharata, the kingly-tiger and a patron of virtue, is ruling the earth in its entirety, who is there to conduct himself in an unacceptable way to morality on it? [4-18-10]
"Abiding in our own pre-eminent righteousness, and even abiding by the order of Bharata we punish him who deviated from the path of morality, according to custom. [4-18-11]
-----
In response to Vali's claim that he did nothing wrong, Rama describes Vali's misconduct, engaging in actions quite inappropriate for a king:
"As for you, you brought virtue to a state of decadence, rendered yourself reprovable by your own decadent behaviour, for carnality alone has become your primary doctrine, and thus you have not abided by the conduct meetly to a king. [4-18-12]
"The probity practised by principled people is very subtle and highly imponderable, and the soul that abides in the hearts of all beings alone can differentiate between just and unjust. [4-18-15]
"Like a blind by birth counselling with similarly blind, you being a vacillant, on your counselling only with frantic minded vacillant monkeys, what can you really fathom about right and wrong? [4-18-16]
"Realise this reason by which I have eliminated you... you misbehaved with your brother's wife, forsaking the perpetual tradition. [4-18-18]
"While the great-souled Sugreeva is still alive, you with your habit of sinful acts have lustily misbehaved with Sugreeva's wife Ruma, who should be counted as your daughter-in-law. [4-18-19]
"Thereby, oh, vanara, this punishment is imposed on you, for your dissolute sinning in abusing your brother's wife, thereby for your transgression of tradition and virtue. [4-18-20]
-----
These verses seem to indicate Rama might have killed Vaali unfairly. But it could also be Rama's response to Vaali's wrong accusation, that either ways Rama's action was correct
"I have neither angst nor ire in this matter of my eliminating you, or, your reviling me, oh, best monkey, but listen to the other point I wish to make clear. People will be capturing several animals, either covertly or overtly, with snares, springes and even with numerous contrivances. [4-18-37, 38a]
"Meat eating people will undeniably kill animals, either they are speedily sprinting or standing steadily, fully dismayed or undismayed, vigilant or unvigilant, and even if they are facing away, in that there is no sacrilege. [4-18-38b, 39]
"In this world even the kingly sages well-versed in virtue will go on hunting, and hunting is no face to face game, as such, oh, vanara, therefore I felled you in combat with my arrow because you are a tree-branch animal, whether you are not combating with me or combating against me. [4-18-40]
Here Rama explains that he felled Vaali in a combat and Vaali doesn't refute it.
------
On seeing Rama and the great mighty Lakshmana as well, he who fell to ground flatly hurt by arrow, and whose energy and lives are trifling and vigour trivialised thereby, that Vali spoke these sarcastic words in his proper pride to Rama, who is taking pride in this conflict as a victor, which words at the same time have meaning, politeness, and self-righteousness. [4-17-14, 15]
"You are a renowned prince with pleasing looks.. but, which kind of death I am getting now, that too when I was in the commotion of conflict with another, alas, that ignoble death is owing to you, and what merit is achieved by you in this undertaking of yours to kill someone who is facing away from you... [4-17-16]
Valmiki clearly says that Vaali had lost his senses and he might well be rambling when he accuses Rama here. Later when he regains his senses, he accepts his mistake and says that he spoke wrongly.
"Oh, lord, oh, tremendously braving one, oh, Rama, the lord of people, I blamed you when your arrow sweltered and rendered me imbecile, thus I blamed you unthinkingly for which I may please be pardoned, I appease you for the same. [4-18-66]
It's not clear what's wrong - his earlier tirade against Rama or his accusation that Rama killed him when he wasn't looking
-----
Vali realizes that Rama acted properly and repents for his mistake:
That lord of vanara-s then replied Rama with adjoined palms, "oh, best one among men, what all you have said is that way proper, undoubtedly. [4-18-45]
"Indeed an ignoble cannot disprove a nobleman, Raghava, and with regards to the undesirable and improper words I have unwittingly spoken earlier, in that mater too it will be truly unapt of you to make me blameworthy, as I spoke them in anguish and ignorance. [4-18-46, 47a]
"You alone are the knower of recourses and their real nature, namely probity, prosperity, pleasure seeking, and emancipation; dharma, artha, kaama, moksha ; and you take delight in the welfare of subjects, and your faultless intellect is clear in accomplishing ends by judging the causes and effects. [4-18-47b, c]
"Oh, Rama, the knower of probity, I am the one who digressed from the rightness and a forerunner among such transgressors, such as I am, give absolution even to me with words abounded with rightness." Vali is thus saying to Rama. [4-18-48]
-----
This is how the vanaras report the slaying of Vaali to Tara. It's important to note that they say that Rama shattered the trees and boulders thrown at Rama. Which means that they were engaged in a direct fight.
"Return, oh, lady with a living son, and safeguard Angada, for the Terminator in the form of Rama is taking away Vali on killing. [4-19-11]
"Shattering trees and massive boulders that Vali hurled at him with thunderbolt like arrows Rama felled Vali as if by thunderbolt. [4-19-12]
-----
Finally, in Uttara Kandam, Narada and Agastya praise Rama's victory over Vaali. It's doubtful if they will do so if Rama won the battle unfairly.
Similarly, in Yuddha Kandam, Hanuman lectures Ravana about the impropriety of slaying someone when they weren't looking. Again, it's doubtful if Hanuman would have praised Rama for his valour over Vaali if it had been unfair
------
All these arguments have been extensively discussed and debated, but according to Valmiki's Ramayana it's inconclusive if Rama killed Vaali by not engaging in a face to face fight.