My group was in this cottage gathering clues when someone kicked open the door while wielding a battle axe. I as a cleric reacted to this by using my action in order to make the now open door slam shut on his face pushing him outside using thaumaturgy:

You instantaneously cause an unlocked door or window to fly open or slam shut.

So what happens to the guy who had kicked the door?

The guy was standing right at the entrance breathing heavily and I won initiative.

The rules don't say, thus it is up to the DM

Thaumaturgy only has this to say about the door closing effect:

You instantaneously cause an unlocked door or window to fly open or slam shut.

Now, 5e spells are designed to do only the things that they say in the spell description. However, in this case, that doesn't really help settle the dilemma. Does the door shut, regardless of obstacles because the spell says that the door shuts? Or will obstacles prevent the door from shutting because there is nothing to indicate that the door slamming shut is in any way immune to the effects of normal physics with respect to things blocking it?

The thing is, the rules don't say. And either ruling, when applied uniformly, can lead to nonsensical results. For example, the door flinging enormous boulders out of the way to close.

And that is another thing 5e spells are designed for. Many spells are written in language that purposefully doesn't try to cover every detail or circumstance instead choosing to leave it up to the DM to adjudicate the results in those cases.

This is one of those cases. Ask your DM.

Reasonable ruling: it acts like a normal door slammed by a normal person

The way I would rule it at my table is that the door is not imbued with any kind of supernatural strength, but instead is simply closed as if I had slammed it myself.

So, what would happen to the guy in the door? It would depend on the exact circumstances of where they where, but if they were in enough to block the door, likely the only thing it would do to them is stub their toe or bruise their nose (not enough to do HP damage) just as a normal slammed door would do and the door would remain ajar.

This is my ruling that I would probably make a my table. It makes sense to me because we all know how doors normally work and thus it makes it easy to keep my rulings consistent and reasonable.

Your DM should consider what option is best fit for their table that will make sense to them and maximise fun.

Not much happens to the guy in the door

As spells only do what they say they do, what happens when you try to slam the door shut with someone standing in them? The door does not shut. Will it be painful for them? Maybe. Will it do any damage? No, the spell would say so. Will it smack into their face? Depends on where they stand – I would presume that after kicking the door open, they are well on the way through that door. Would it have any mechanical effect if it did? No, see above. (I assume the door opens into the room; in any case the it may also be destroyed by the aforementioned kicking in which case there is not much to use the spell on.)

Obviously, this is a rather strict, RAW reading. Your DM may rule otherwise, allowing for fun consequences.

You can try anything, the DM will narrate the result

Recommended ruling: opposed ability check for a mini-shove.

The guy was standing right at the entrance breathing heavily and I won
initiative.

This case looks like a way to use the cantrip's ability to slam the door shut in a tactical way without trying to turn that spell into a weapon/damage wielding spell, which none of its features indicates that it is, and the opponent gets a chance to resist the effects since he's already in the doorway.

In order to see if the door slams in the opponent's face and closes (which would push him back a foot or so outside the door, use the contest rules.

Contests

Sometimes one character’s or monster’s efforts are directly
opposed to another’s. This can occur when both of them are trying to
do the same thing and only one can succeed, such as attempting to
snatch up a magic ring that has fallen on the floor. This situation
also applies when one of them is trying to prevent the other one from
accomplishing a goal—for example, when a monster tries to force open a
door that an adventurer is holding closed.

The situation you describe is very similar to this. A contest is an opposed ability check. This does not seem to be a case where an automatic win should be granted to slam the door, unless the opponent has not entered the doorway. There should also be a chance that the door bounces off of the guy with the ax, and does not knock him back.

How to adjudicate the contest: oppose the cleric's spell casting DC versus the opponent's Athletics(Strength) ability check and see if the cleric succeeds, or if the opponent is strong enough, or quick enough, to resist the door slamming in his face.

Discuss this with your DM. This is a case where rulings over rules, a 5e design paradigm, can work to make the game fun. Rewarding innovative use of game features: any DM ought to be on the lookout for that chance. It is also a case where the DM might offer advantage to the guy with the ax, or the cleric, depending on how the rest of the situation is set up.

Rules As Fun-embrace it.